From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 72F72C7EE23 for ; Fri, 26 May 2023 12:32:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242955AbjEZMcN (ORCPT ); Fri, 26 May 2023 08:32:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33456 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237056AbjEZMcM (ORCPT ); Fri, 26 May 2023 08:32:12 -0400 Received: from smtp-out2.suse.de (smtp-out2.suse.de [IPv6:2001:67c:2178:6::1d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 84D0AE73; Fri, 26 May 2023 05:31:32 -0700 (PDT) Received: from relay2.suse.de (relay2.suse.de [149.44.160.134]) by smtp-out2.suse.de (Postfix) with ESMTP id 986451F8AE; Fri, 26 May 2023 12:29:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1685104199; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=WhfCogRNdmmsvJU5CjDqdyyExEkgp9ppc07tbwxSsOg=; b=UBlT9ISwUDHYPeMb+a+ztXThl86vi9IDG6M9ms/XS0tVW0/E2hbhprn7RRLeyq1C45+D/b UHhvn1UnkoPvxRIU7hyiv3JpraRb3fXflpZJXSH2dOTVPbnALq5BAMR9pbIBcE99YuCZJK S6IfNcRb2vDTizhuODQNL5WRubLUMN8= Received: from suse.cz (pmladek.tcp.ovpn2.prg.suse.de [10.100.208.146]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by relay2.suse.de (Postfix) with ESMTPS id 3EB022D38D; Fri, 26 May 2023 12:29:56 +0000 (UTC) Date: Fri, 26 May 2023 14:29:53 +0200 From: Petr Mladek To: Doug Anderson Cc: Andrew Morton , Matthias Kaehlcke , kgdb-bugreport@lists.sourceforge.net, Stephane Eranian , mpe@ellerman.id.au, Tzung-Bi Shih , Daniel Thompson , Mark Rutland , linuxppc-dev@lists.ozlabs.org, Sumit Garg , npiggin@gmail.com, davem@davemloft.net, Marc Zyngier , Stephen Boyd , sparclinux@vger.kernel.org, christophe.leroy@csgroup.eu, Catalin Marinas , ravi.v.shankar@intel.com, Randy Dunlap , Pingfan Liu , Guenter Roeck , Lecopzer Chen , Ian Rogers , ito-yuichi@fujitsu.com, ricardo.neri@intel.com, linux-arm-kernel@lists.infradead.org, linux-perf-users@vger.kernel.org, Will Deacon , Chen-Yu Tsai , linux-kernel@vger.kernel.org, Masayoshi Mizuma , Andi Kleen , Colin Cross Subject: Re: [PATCH v5 14/18] watchdog/hardlockup: detect hard lockups using secondary (buddy) CPUs Message-ID: References: <20230519101840.v5.18.Ia44852044cdcb074f387e80df6b45e892965d4a1@changeid> <20230519101840.v5.14.I6bf789d21d0c3d75d382e7e51a804a7a51315f2c@changeid> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: linux-perf-users@vger.kernel.org On Thu 2023-05-25 13:08:04, Doug Anderson wrote: > Hi, > > On Thu, May 25, 2023 at 9:27 AM Petr Mladek wrote: > > > > On Fri 2023-05-19 10:18:38, Douglas Anderson wrote: > > > Implement a hardlockup detector that doesn't doesn't need any extra > > > arch-specific support code to detect lockups. Instead of using > > > something arch-specific we will use the buddy system, where each CPU > > > watches out for another one. Specifically, each CPU will use its > > > softlockup hrtimer to check that the next CPU is processing hrtimer > > > interrupts by verifying that a counter is increasing. > > > > > > --- a/kernel/watchdog.c > > > +++ b/kernel/watchdog.c > > > @@ -85,7 +85,7 @@ __setup("nmi_watchdog=", hardlockup_panic_setup); > > > > > > #endif /* CONFIG_HARDLOCKUP_DETECTOR */ > > > > > > -#if defined(CONFIG_HARDLOCKUP_DETECTOR_PERF) > > > +#if defined(CONFIG_HARDLOCKUP_DETECTOR_COUNTS_HRTIMER) > > > > > > static DEFINE_PER_CPU(atomic_t, hrtimer_interrupts); > > > static DEFINE_PER_CPU(int, hrtimer_interrupts_saved); > > > @@ -106,6 +106,14 @@ notrace void arch_touch_nmi_watchdog(void) > > > } > > > EXPORT_SYMBOL(arch_touch_nmi_watchdog); > > > > > > +void watchdog_hardlockup_touch_cpu(unsigned int cpu) > > > +{ > > > + per_cpu(watchdog_hardlockup_touched, cpu) = true; > > > + > > > + /* Match with smp_rmb() in watchdog_hardlockup_check() */ > > > + smp_wmb(); > > > > It is great that you described where the related barrier is. > > > > Another important information is what exactly is synchronized. > > And I am actually not sure what we are synchronizing here. > > > > My understanding is that a write barrier should synchronize > > related writes, for example: > > > > X = ...; > > /* Make sure that X is modified before Y */ > > smp_wmb(); > > Y = ...; > > > > And the related read barrier should synchronize the related reads, > > for example: > > > > if (test(Y)) { > > /* > > * Make sure that we use the updated X when > > * we saw the updated Y. > > */ > > smp_rmb(); > > do_something(X); > > } > > > > IMHO, we do not need any barrier here because we have only > > one variable "watchdog_hardlockup_touched" here. > > watchdog_hardlockup_check() will either see the updated value > > or not. But it does not synchronize it against any other > > variables or values. > > Fair. These barriers were present in the original buddy lockup > detector that we've been carrying in ChromeOS but that doesn't > necessarily mean that they were there for a good reason. > > Reasoning about weakly ordered memory always makes my brain hurt and I > never feel confident at the end that I got the right answer and, of > course, this is coupled by the fact that if I have a logic error in my > reasoning that it might cause a rare / subtle bug. :( Sure. Lockless code is complicated. > When possible I > try to write code that uses full blown locks to make it easier to > reason about (even if less efficient), Makes sense. There should be a good reason to use lockless code because it is complicated to do it right and maintain. > but that's not necessarily > possible here. While we obviously don't just want to sprinkle barriers > all over the code, IMO it's not a terrible sin to put a barrier in a > case where it makes it easier to reason about the order of things. I understand this. Well, it is always important to describe the the reason why the barrier was added there. Even when it is wrong, it gives a clue what was the motivation. Otherwise, it is hard to do any changes on the code later. I guess that it might be more problematic for you because you probably are not the original author. > In any case, I guess in this case I would worry about some sort of > ordering race when enabling / disabling the buddy lockup detector. At > the end of the buddy's watchdog_hardlockup_enable() / > watchdog_hardlockup_disable() we adjust the "watchdog_cpus" which > changes buddy assignments. Without a barrier, I _think_ it would be > possible for a new CPU to notice the change in buddies without > noticing the touch. Does that match your understanding? Now when > reasoning about CPUs going online/offline we need to consider this > extra case and we have to decide if there's any chance it could lead > to a false lockup detection. With the memory barriers here, it's a > little easier to think about. This makes sense. I did not think about the hotplug scenario. Well, I suggest to move the barriers into the buddy code and describe it there. It does not make sense to use the barriers for the perf hardlockup. I mean something like: diff --git a/kernel/watchdog_buddy.c b/kernel/watchdog_buddy.c index fee45af2e5bd..ebe71dcb55e6 100644 --- a/kernel/watchdog_buddy.c +++ b/kernel/watchdog_buddy.c @@ -52,6 +52,13 @@ void watchdog_hardlockup_enable(unsigned int cpu) if (next_cpu < nr_cpu_ids) watchdog_hardlockup_touch_cpu(next_cpu); + /* + * Makes sure that watchdog is touched on this CPU before + * other CPUs could see it in watchdog_cpus. The counter + * part is in watchdog_buddy_check_hardlockup(). + */ + smp_wmb(); + cpumask_set_cpu(cpu, &watchdog_cpus); } @@ -69,6 +76,13 @@ void watchdog_hardlockup_disable(unsigned int cpu) if (next_cpu < nr_cpu_ids) watchdog_hardlockup_touch_cpu(next_cpu); + /* + * Makes sure that watchdog is touched on the next CPU before + * this CPU disappear in watchdog_cpus. The counter part is in + * watchdog_buddy_check_hardlockup(). + */ + smp_wmb(); + cpumask_clear_cpu(cpu, &watchdog_cpus); } @@ -89,5 +103,12 @@ void watchdog_buddy_check_hardlockup(unsigned long hrtimer_interrupts) if (next_cpu >= nr_cpu_ids) return; + /* + * Make sure that the watchdog was touched on next CPU when + * watchdog_next_cpu() returned another one because of + * a change in watchdog_hardlockup_enable()/disable(). + */ + smp_rmb(); + watchdog_hardlockup_check(next_cpu, NULL); } > Did the above convince you about keeping the barriers? If so, do you > have any suggested comment that would make it clearer? > > > > > +} > > > + > > > static bool is_hardlockup(unsigned int cpu) > > > { > > > int hrint = atomic_read(&per_cpu(hrtimer_interrupts, cpu)); > > > @@ -443,11 +454,15 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer) > > > struct pt_regs *regs = get_irq_regs(); > > > int duration; > > > int softlockup_all_cpu_backtrace = sysctl_softlockup_all_cpu_backtrace; > > > + unsigned long hrtimer_interrupts; > > > > > > if (!watchdog_enabled) > > > return HRTIMER_NORESTART; > > > > > > - watchdog_hardlockup_kick(); > > > + hrtimer_interrupts = watchdog_hardlockup_kick(); > > > + > > > + /* test for hardlockups */ > > > > I would omit the comment. It is not valid when perf detector is used. > > And checking the buddy is clear from the function name. > > > > > + watchdog_buddy_check_hardlockup(hrtimer_interrupts); > > > > I would personally move this into watchdog_hardlockup_kick(). > > watchdog_timer_fn() is already complex enough. And checking > > the buddy when kicking a CPU makes sense. > > Sure, I'll add that to my list of things to follow-up with. > > > > Also I would not pass "hrtimer_interrupts". I guess that it is > > just an optimization. It is an extra churn in the code. IMHO, > > is is not wort it. This code does not need to be super optimized. > > The main reason I did it is that "hrtimer_interrupts" is static to > watchdog.c now. If I don't pass it in then I have to make it > non-static and add it to the header. That also means anyone looking at > the variable and figuring out how it is read/written needs to go > search for other people that reference it. I feel like it's cleaner to > just pass it in. If you feel strongly that I should change this then > let me know, but otherwise I'll plan to leave this how I have it. I do not have strong opinion. For me, the more important change is to move watchdog_buddy_check_hardlockup() into watchdog_hardlockup_kick(). watchdog_timer_fn() is already too complex. > > > > /* kick the softlockup detector */ > > > if (completion_done(this_cpu_ptr(&softlockup_completion))) { > > > --- a/lib/Kconfig.debug > > > +++ b/lib/Kconfig.debug > > > @@ -1035,10 +1035,55 @@ config BOOTPARAM_SOFTLOCKUP_PANIC > > > > > > Say N if unsure. > > > > > > -config HARDLOCKUP_DETECTOR_PERF > > > +# Both the "perf" and "buddy" hardlockup detectors count hrtimer > > > +# interrupts. This config enables functions managing this common code. > > > +config HARDLOCKUP_DETECTOR_COUNTS_HRTIMER > > > bool > > > select SOFTLOCKUP_DETECTOR > > > > > > +config HARDLOCKUP_DETECTOR_PERF > > > + bool > > > + depends on HAVE_HARDLOCKUP_DETECTOR_PERF > > > + select HARDLOCKUP_DETECTOR_COUNTS_HRTIMER > > > + > > > +config HARDLOCKUP_DETECTOR_BUDDY > > > + bool > > > + depends on SMP > > > + select HARDLOCKUP_DETECTOR_COUNTS_HRTIMER > > > + > > > +# For hardlockup detectors you can have one directly provided by the arch > > > +# or use a "non-arch" one. If you're using a "non-arch" one that is > > > +# further divided the perf hardlockup detector (which, confusingly, needs > > > +# arch-provided perf support) and the buddy hardlockup detector (which just > > > +# needs SMP). In either case, using the "non-arch" code conflicts with > > > +# the NMI watchdog code (which is sometimes used directly and sometimes used > > > +# by the arch-provided hardlockup detector). > > > +config HAVE_HARDLOCKUP_DETECTOR_NON_ARCH > > > + bool > > > + depends on (HAVE_HARDLOCKUP_DETECTOR_PERF || SMP) && !HAVE_NMI_WATCHDOG > > > + default y > > > + > > > +config HARDLOCKUP_DETECTOR_PREFER_BUDDY > > > + bool "Prefer the buddy CPU hardlockup detector" > > > + depends on HAVE_HARDLOCKUP_DETECTOR_NON_ARCH && HAVE_HARDLOCKUP_DETECTOR_PERF && SMP > > > > Huh, I have big troubles to scratch my head around this check: > > > > HAVE_HARDLOCKUP_DETECTOR_NON_ARCH depends on HAVE_HARDLOCKUP_DETECTOR_PERF and SMP > > > > and this depends on HAVE_HARDLOCKUP_DETECTOR_NON_ARCH and again > > on HAVE_HARDLOCKUP_DETECTOR_PERF and SMP. > > The goal is to have "HARDLOCKUP_DETECTOR_PREFER_BUDDY" to show up as > an option if there is an option _other_ than the buddy. If there's not > more than one hardlockup detector to pick from then there's no reason > to ask the person configuring the kernel which one they'd prefer. At > the moment, if you have an "arch" lockup detector then you're stuck > with it, so you only get a choice if a "perf" detector is available > and you've got SMP. > > Ah, so I guess this could be simplified to: > > depends on HAVE_HARDLOCKUP_DETECTOR_PERF && SMP Yes, this is much better. > OK, I'll add that to the list. > > > > > + help > > > + Say Y here to prefer the buddy hardlockup detector over the perf one. > > > + > > > + With the buddy detector, each CPU uses its softlockup hrtimer > > > + to check that the next CPU is processing hrtimer interrupts by > > > + verifying that a counter is increasing. > > > + > > > + This hardlockup detector is useful on systems that don't have > > > + an arch-specific hardlockup detector or if resources needed > > > + for the hardlockup detector are better used for other things. > > > + > > > +# This will select the appropriate non-arch hardlockdup detector > > > +config HARDLOCKUP_DETECTOR_NON_ARCH > > > + bool > > > + depends on HAVE_HARDLOCKUP_DETECTOR_NON_ARCH > > > + select HARDLOCKUP_DETECTOR_BUDDY if !HAVE_HARDLOCKUP_DETECTOR_PERF || HARDLOCKUP_DETECTOR_PREFER_BUDDY > > > + select HARDLOCKUP_DETECTOR_PERF if HAVE_HARDLOCKUP_DETECTOR_PERF && !HARDLOCKUP_DETECTOR_PREFER_BUDDY > > > + > > > # > > > # Enables a timestamp based low pass filter to compensate for perf based > > > # hard lockup detection which runs too fast due to turbo modes. > > > @@ -1053,9 +1098,10 @@ config HARDLOCKUP_CHECK_TIMESTAMP > > > config HARDLOCKUP_DETECTOR > > > bool "Detect Hard Lockups" > > > depends on DEBUG_KERNEL && !S390 > > > > Is there any reason why S390 could not or do not want to use the buddy > > hardlockup detector. > > This isn't a new dependency, but it's a good question. Looking at the > git history, I see commit dea20a3fbdd0 ("[PATCH] Disable > DETECT_SOFTLOCKUP for s390"). ...and it looks like the softlockup > detector still says it's broken on s390. That would mean that the > buddy detector is broken too. It seems that s390 wanted to disable the watchdog completely, see the commit dea20a3fbdd08e5 ("[PATCH] Disable DETECT_SOFTLOCKUP for s390") because they got too many false positives. > > > > - depends on HAVE_HARDLOCKUP_DETECTOR_PERF || HAVE_HARDLOCKUP_DETECTOR_ARCH > > > + depends on HAVE_HARDLOCKUP_DETECTOR_NON_ARCH || HAVE_HARDLOCKUP_DETECTOR_ARCH > > > select LOCKUP_DETECTOR > > > - select HARDLOCKUP_DETECTOR_PERF if HAVE_HARDLOCKUP_DETECTOR_PERF > > > + select HARDLOCKUP_DETECTOR_NON_ARCH if HAVE_HARDLOCKUP_DETECTOR_NON_ARCH > > > > Anyway, the configuration of the hard lockup detectors is insane and > > this patchset makes it even worse, especially the new > > HARDLOCKUP_DETECTOR_NON_ARCH stuff. > > > > It seems that sparc, powerpc and s390 are somehow special. Do you > > still have in mind how they are distinguished using the Kconfig > > variables? > > > > For example, I am pretty confused by the meaning of HAVE_NMI_WATCHDOG. > > > > And sparc has its own variant of > > watchdog_hardlockup_enable()/disable(). It means that it is > > arch-specific. Does it work with the 13th patch which made > > watchdog_hardlockup_enable()/disable() to be watchdog-hardlockup-type > > specific? Is is somehow related to HAVE_NMI_WATCHDOG? > > Does this replace the entire watchdog only only the enable part? > > > > I think that we need to make this more straightforward. But I first > > need to understand the existing maze of config variables. > > I agree that it's confusing. I'm obviously biased, but IMO it's less > confusing after my patchset than before. ;-) The state of the world > before my patchset set a pretty low bar. > > As far as I understand it, at an architecture-level you can choose any > _ONE_ of the following: > > a) Implement bits needed for the the "perf" hardlockup detector. x86 > has done this, some configs of powerpc do this, and arm64 now after my > patch series. This is HAVE_HARDLOCKUP_DETECTOR_PERF. > > b) Implement your own totally separate hardlockup detector that > doesn't use any of the common "perf" code but still looks the same to > userspace (same sysctls, etc). Only powerpc does this (in some > configs). As per conversations in previous versions of my patch > series, apparently powerpc's version is quite fancy and maybe someday > people can move some of these features to the common code. This is > HAVE_HARDLOCKUP_DETECTOR_ARCH. > > c) Don't implement the full features of a hardlockup detector but > still have the basics. In the very least, I think it doesn't support > the sysctls "hardlockup_panic" and "hardlockup_all_cpu_backtrace". It > doesn't support the kernel command line parameter "nmi_watchdog=". I > don't know for sure if there are any other differences. Only sparc64 > does this. This is HAVE_NMI_WATCHDOG. Confusingly, > HAVE_HARDLOCKUP_DETECTOR_ARCH selects HAVE_NMI_WATCHDOG. > > d) Don't implement _any_ hardlockup detector of any sort. After my > patchset you can still end up with "buddy" if you have SMP. > > One thing that would probably help would be to bring sparc64 to a full > "arch" hardlockup implementation and then get rid of the special case. > That seems a bit outside my scope, though if someone wanted to post > patches for that I'd be willing to give them a review. It would be nice but it might be problematic if we do not have access to the hardware. > I guess other than that, the best we could try to do is to rename some > configs and/or add some subconfigs to describe certain features? Maybe > HAVE_NMI_WATCHDOG => HAVE_HARDLOCKUP_DETECTOR_ARCH_BASIC_FEATURES > would help? I'd love to come up with a better name for > HAVE_HARDLOCKUP_DETECTOR_NON_ARCH but I couldn't come up with one. > Maybe the unwieldy "HAVE_HARDLOCKUP_DETECTOR_THAT_COUNTS_HRTIMER"? Renaming the config variables seems to be the best solution at the moment. IMHO, it would be nice to have something like: + CONFIG_HARDLOCKUP_DETECTOR for code shared by all hardlockup detectors + CONFIG_HARDLOCKUP_DETECTOR_PERF for code using kernel/watchdog_perf.c + CONFIG_HARDLOCKUP_DETECTOR_BUDDY for code using kernel/watchdog_buddy.c + HAVE_HARDLOCKUP_DETECTOR_PERF set by architectures that support using kernel/watchdog_perf.c + HAVE_HARDLOCKUP_DETECTOR_ARCH set by architectures that have alternative implementation of the hardlockup detector + CONFIG_HARDLOCKUP_DETECTOR_PREFER_BUDDY Allow to prefer the buddy detector when _PERF or _ARCH is available as well. + HAVE_HARDLOCKUP_PANIC + HAVE_HARDLOCKUP_ALL_CPU_BACKTRACE set when the earchitecture support these features and used for the sysfs interface > If you have concrete suggestions for what would be cleaner, let me > know and I can queue up a patch. ...or I'm happy to review a patch. I am not sure how complicated it would be to rename the config variables to somehing sane. I am sorry I do not have time to prepare the patches at the moment. I'll let Andrew to decide if he would require this cleanup to accept the patchset. Best Regards, Petr From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id CCEE3C77B7C for ; Fri, 26 May 2023 12:31:02 +0000 (UTC) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4QSPR86s6rz3fDw for ; Fri, 26 May 2023 22:31:00 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=suse.com header.i=@suse.com header.a=rsa-sha256 header.s=susede1 header.b=UBlT9ISw; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=suse.com (client-ip=2001:67c:2178:6::1d; helo=smtp-out2.suse.de; envelope-from=pmladek@suse.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=suse.com header.i=@suse.com header.a=rsa-sha256 header.s=susede1 header.b=UBlT9ISw; dkim-atps=neutral Received: from smtp-out2.suse.de (smtp-out2.suse.de [IPv6:2001:67c:2178:6::1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4QSPQ96T22z3c8v for ; Fri, 26 May 2023 22:30:08 +1000 (AEST) Received: from relay2.suse.de (relay2.suse.de [149.44.160.134]) by smtp-out2.suse.de (Postfix) with ESMTP id 986451F8AE; Fri, 26 May 2023 12:29:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1685104199; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=WhfCogRNdmmsvJU5CjDqdyyExEkgp9ppc07tbwxSsOg=; b=UBlT9ISwUDHYPeMb+a+ztXThl86vi9IDG6M9ms/XS0tVW0/E2hbhprn7RRLeyq1C45+D/b UHhvn1UnkoPvxRIU7hyiv3JpraRb3fXflpZJXSH2dOTVPbnALq5BAMR9pbIBcE99YuCZJK S6IfNcRb2vDTizhuODQNL5WRubLUMN8= Received: from suse.cz (pmladek.tcp.ovpn2.prg.suse.de [10.100.208.146]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by relay2.suse.de (Postfix) with ESMTPS id 3EB022D38D; Fri, 26 May 2023 12:29:56 +0000 (UTC) Date: Fri, 26 May 2023 14:29:53 +0200 From: Petr Mladek To: Doug Anderson Subject: Re: [PATCH v5 14/18] watchdog/hardlockup: detect hard lockups using secondary (buddy) CPUs Message-ID: References: <20230519101840.v5.18.Ia44852044cdcb074f387e80df6b45e892965d4a1@changeid> <20230519101840.v5.14.I6bf789d21d0c3d75d382e7e51a804a7a51315f2c@changeid> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , Ian Rogers , ito-yuichi@fujitsu.com, Lecopzer Chen , kgdb-bugreport@lists.sourceforge.net, ricardo.neri@intel.com, Stephane Eranian , sparclinux@vger.kernel.org, Guenter Roeck , Will Deacon , Daniel Thompson , Andi Kleen , Marc Zyngier , Chen-Yu Tsai , Matthias Kaehlcke , Catalin Marinas , Masayoshi Mizuma , ravi.v.shankar@intel.com, Tzung-Bi Shih , Colin Cross , npiggin@gmail.com, Stephen Boyd , Pingfan Liu , linux-arm-kernel@lists.infradead.org, Sumit Garg , Randy Dunlap , linux-kernel@vger.kernel.org, linux-perf-users@vger.kernel.org, Andrew Morton , linuxppc-dev@lists.ozlabs.org, davem@davemloft.net Errors-To: linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Sender: "Linuxppc-dev" On Thu 2023-05-25 13:08:04, Doug Anderson wrote: > Hi, > > On Thu, May 25, 2023 at 9:27 AM Petr Mladek wrote: > > > > On Fri 2023-05-19 10:18:38, Douglas Anderson wrote: > > > Implement a hardlockup detector that doesn't doesn't need any extra > > > arch-specific support code to detect lockups. Instead of using > > > something arch-specific we will use the buddy system, where each CPU > > > watches out for another one. Specifically, each CPU will use its > > > softlockup hrtimer to check that the next CPU is processing hrtimer > > > interrupts by verifying that a counter is increasing. > > > > > > --- a/kernel/watchdog.c > > > +++ b/kernel/watchdog.c > > > @@ -85,7 +85,7 @@ __setup("nmi_watchdog=", hardlockup_panic_setup); > > > > > > #endif /* CONFIG_HARDLOCKUP_DETECTOR */ > > > > > > -#if defined(CONFIG_HARDLOCKUP_DETECTOR_PERF) > > > +#if defined(CONFIG_HARDLOCKUP_DETECTOR_COUNTS_HRTIMER) > > > > > > static DEFINE_PER_CPU(atomic_t, hrtimer_interrupts); > > > static DEFINE_PER_CPU(int, hrtimer_interrupts_saved); > > > @@ -106,6 +106,14 @@ notrace void arch_touch_nmi_watchdog(void) > > > } > > > EXPORT_SYMBOL(arch_touch_nmi_watchdog); > > > > > > +void watchdog_hardlockup_touch_cpu(unsigned int cpu) > > > +{ > > > + per_cpu(watchdog_hardlockup_touched, cpu) = true; > > > + > > > + /* Match with smp_rmb() in watchdog_hardlockup_check() */ > > > + smp_wmb(); > > > > It is great that you described where the related barrier is. > > > > Another important information is what exactly is synchronized. > > And I am actually not sure what we are synchronizing here. > > > > My understanding is that a write barrier should synchronize > > related writes, for example: > > > > X = ...; > > /* Make sure that X is modified before Y */ > > smp_wmb(); > > Y = ...; > > > > And the related read barrier should synchronize the related reads, > > for example: > > > > if (test(Y)) { > > /* > > * Make sure that we use the updated X when > > * we saw the updated Y. > > */ > > smp_rmb(); > > do_something(X); > > } > > > > IMHO, we do not need any barrier here because we have only > > one variable "watchdog_hardlockup_touched" here. > > watchdog_hardlockup_check() will either see the updated value > > or not. But it does not synchronize it against any other > > variables or values. > > Fair. These barriers were present in the original buddy lockup > detector that we've been carrying in ChromeOS but that doesn't > necessarily mean that they were there for a good reason. > > Reasoning about weakly ordered memory always makes my brain hurt and I > never feel confident at the end that I got the right answer and, of > course, this is coupled by the fact that if I have a logic error in my > reasoning that it might cause a rare / subtle bug. :( Sure. Lockless code is complicated. > When possible I > try to write code that uses full blown locks to make it easier to > reason about (even if less efficient), Makes sense. There should be a good reason to use lockless code because it is complicated to do it right and maintain. > but that's not necessarily > possible here. While we obviously don't just want to sprinkle barriers > all over the code, IMO it's not a terrible sin to put a barrier in a > case where it makes it easier to reason about the order of things. I understand this. Well, it is always important to describe the the reason why the barrier was added there. Even when it is wrong, it gives a clue what was the motivation. Otherwise, it is hard to do any changes on the code later. I guess that it might be more problematic for you because you probably are not the original author. > In any case, I guess in this case I would worry about some sort of > ordering race when enabling / disabling the buddy lockup detector. At > the end of the buddy's watchdog_hardlockup_enable() / > watchdog_hardlockup_disable() we adjust the "watchdog_cpus" which > changes buddy assignments. Without a barrier, I _think_ it would be > possible for a new CPU to notice the change in buddies without > noticing the touch. Does that match your understanding? Now when > reasoning about CPUs going online/offline we need to consider this > extra case and we have to decide if there's any chance it could lead > to a false lockup detection. With the memory barriers here, it's a > little easier to think about. This makes sense. I did not think about the hotplug scenario. Well, I suggest to move the barriers into the buddy code and describe it there. It does not make sense to use the barriers for the perf hardlockup. I mean something like: diff --git a/kernel/watchdog_buddy.c b/kernel/watchdog_buddy.c index fee45af2e5bd..ebe71dcb55e6 100644 --- a/kernel/watchdog_buddy.c +++ b/kernel/watchdog_buddy.c @@ -52,6 +52,13 @@ void watchdog_hardlockup_enable(unsigned int cpu) if (next_cpu < nr_cpu_ids) watchdog_hardlockup_touch_cpu(next_cpu); + /* + * Makes sure that watchdog is touched on this CPU before + * other CPUs could see it in watchdog_cpus. The counter + * part is in watchdog_buddy_check_hardlockup(). + */ + smp_wmb(); + cpumask_set_cpu(cpu, &watchdog_cpus); } @@ -69,6 +76,13 @@ void watchdog_hardlockup_disable(unsigned int cpu) if (next_cpu < nr_cpu_ids) watchdog_hardlockup_touch_cpu(next_cpu); + /* + * Makes sure that watchdog is touched on the next CPU before + * this CPU disappear in watchdog_cpus. The counter part is in + * watchdog_buddy_check_hardlockup(). + */ + smp_wmb(); + cpumask_clear_cpu(cpu, &watchdog_cpus); } @@ -89,5 +103,12 @@ void watchdog_buddy_check_hardlockup(unsigned long hrtimer_interrupts) if (next_cpu >= nr_cpu_ids) return; + /* + * Make sure that the watchdog was touched on next CPU when + * watchdog_next_cpu() returned another one because of + * a change in watchdog_hardlockup_enable()/disable(). + */ + smp_rmb(); + watchdog_hardlockup_check(next_cpu, NULL); } > Did the above convince you about keeping the barriers? If so, do you > have any suggested comment that would make it clearer? > > > > > +} > > > + > > > static bool is_hardlockup(unsigned int cpu) > > > { > > > int hrint = atomic_read(&per_cpu(hrtimer_interrupts, cpu)); > > > @@ -443,11 +454,15 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer) > > > struct pt_regs *regs = get_irq_regs(); > > > int duration; > > > int softlockup_all_cpu_backtrace = sysctl_softlockup_all_cpu_backtrace; > > > + unsigned long hrtimer_interrupts; > > > > > > if (!watchdog_enabled) > > > return HRTIMER_NORESTART; > > > > > > - watchdog_hardlockup_kick(); > > > + hrtimer_interrupts = watchdog_hardlockup_kick(); > > > + > > > + /* test for hardlockups */ > > > > I would omit the comment. It is not valid when perf detector is used. > > And checking the buddy is clear from the function name. > > > > > + watchdog_buddy_check_hardlockup(hrtimer_interrupts); > > > > I would personally move this into watchdog_hardlockup_kick(). > > watchdog_timer_fn() is already complex enough. And checking > > the buddy when kicking a CPU makes sense. > > Sure, I'll add that to my list of things to follow-up with. > > > > Also I would not pass "hrtimer_interrupts". I guess that it is > > just an optimization. It is an extra churn in the code. IMHO, > > is is not wort it. This code does not need to be super optimized. > > The main reason I did it is that "hrtimer_interrupts" is static to > watchdog.c now. If I don't pass it in then I have to make it > non-static and add it to the header. That also means anyone looking at > the variable and figuring out how it is read/written needs to go > search for other people that reference it. I feel like it's cleaner to > just pass it in. If you feel strongly that I should change this then > let me know, but otherwise I'll plan to leave this how I have it. I do not have strong opinion. For me, the more important change is to move watchdog_buddy_check_hardlockup() into watchdog_hardlockup_kick(). watchdog_timer_fn() is already too complex. > > > > /* kick the softlockup detector */ > > > if (completion_done(this_cpu_ptr(&softlockup_completion))) { > > > --- a/lib/Kconfig.debug > > > +++ b/lib/Kconfig.debug > > > @@ -1035,10 +1035,55 @@ config BOOTPARAM_SOFTLOCKUP_PANIC > > > > > > Say N if unsure. > > > > > > -config HARDLOCKUP_DETECTOR_PERF > > > +# Both the "perf" and "buddy" hardlockup detectors count hrtimer > > > +# interrupts. This config enables functions managing this common code. > > > +config HARDLOCKUP_DETECTOR_COUNTS_HRTIMER > > > bool > > > select SOFTLOCKUP_DETECTOR > > > > > > +config HARDLOCKUP_DETECTOR_PERF > > > + bool > > > + depends on HAVE_HARDLOCKUP_DETECTOR_PERF > > > + select HARDLOCKUP_DETECTOR_COUNTS_HRTIMER > > > + > > > +config HARDLOCKUP_DETECTOR_BUDDY > > > + bool > > > + depends on SMP > > > + select HARDLOCKUP_DETECTOR_COUNTS_HRTIMER > > > + > > > +# For hardlockup detectors you can have one directly provided by the arch > > > +# or use a "non-arch" one. If you're using a "non-arch" one that is > > > +# further divided the perf hardlockup detector (which, confusingly, needs > > > +# arch-provided perf support) and the buddy hardlockup detector (which just > > > +# needs SMP). In either case, using the "non-arch" code conflicts with > > > +# the NMI watchdog code (which is sometimes used directly and sometimes used > > > +# by the arch-provided hardlockup detector). > > > +config HAVE_HARDLOCKUP_DETECTOR_NON_ARCH > > > + bool > > > + depends on (HAVE_HARDLOCKUP_DETECTOR_PERF || SMP) && !HAVE_NMI_WATCHDOG > > > + default y > > > + > > > +config HARDLOCKUP_DETECTOR_PREFER_BUDDY > > > + bool "Prefer the buddy CPU hardlockup detector" > > > + depends on HAVE_HARDLOCKUP_DETECTOR_NON_ARCH && HAVE_HARDLOCKUP_DETECTOR_PERF && SMP > > > > Huh, I have big troubles to scratch my head around this check: > > > > HAVE_HARDLOCKUP_DETECTOR_NON_ARCH depends on HAVE_HARDLOCKUP_DETECTOR_PERF and SMP > > > > and this depends on HAVE_HARDLOCKUP_DETECTOR_NON_ARCH and again > > on HAVE_HARDLOCKUP_DETECTOR_PERF and SMP. > > The goal is to have "HARDLOCKUP_DETECTOR_PREFER_BUDDY" to show up as > an option if there is an option _other_ than the buddy. If there's not > more than one hardlockup detector to pick from then there's no reason > to ask the person configuring the kernel which one they'd prefer. At > the moment, if you have an "arch" lockup detector then you're stuck > with it, so you only get a choice if a "perf" detector is available > and you've got SMP. > > Ah, so I guess this could be simplified to: > > depends on HAVE_HARDLOCKUP_DETECTOR_PERF && SMP Yes, this is much better. > OK, I'll add that to the list. > > > > > + help > > > + Say Y here to prefer the buddy hardlockup detector over the perf one. > > > + > > > + With the buddy detector, each CPU uses its softlockup hrtimer > > > + to check that the next CPU is processing hrtimer interrupts by > > > + verifying that a counter is increasing. > > > + > > > + This hardlockup detector is useful on systems that don't have > > > + an arch-specific hardlockup detector or if resources needed > > > + for the hardlockup detector are better used for other things. > > > + > > > +# This will select the appropriate non-arch hardlockdup detector > > > +config HARDLOCKUP_DETECTOR_NON_ARCH > > > + bool > > > + depends on HAVE_HARDLOCKUP_DETECTOR_NON_ARCH > > > + select HARDLOCKUP_DETECTOR_BUDDY if !HAVE_HARDLOCKUP_DETECTOR_PERF || HARDLOCKUP_DETECTOR_PREFER_BUDDY > > > + select HARDLOCKUP_DETECTOR_PERF if HAVE_HARDLOCKUP_DETECTOR_PERF && !HARDLOCKUP_DETECTOR_PREFER_BUDDY > > > + > > > # > > > # Enables a timestamp based low pass filter to compensate for perf based > > > # hard lockup detection which runs too fast due to turbo modes. > > > @@ -1053,9 +1098,10 @@ config HARDLOCKUP_CHECK_TIMESTAMP > > > config HARDLOCKUP_DETECTOR > > > bool "Detect Hard Lockups" > > > depends on DEBUG_KERNEL && !S390 > > > > Is there any reason why S390 could not or do not want to use the buddy > > hardlockup detector. > > This isn't a new dependency, but it's a good question. Looking at the > git history, I see commit dea20a3fbdd0 ("[PATCH] Disable > DETECT_SOFTLOCKUP for s390"). ...and it looks like the softlockup > detector still says it's broken on s390. That would mean that the > buddy detector is broken too. It seems that s390 wanted to disable the watchdog completely, see the commit dea20a3fbdd08e5 ("[PATCH] Disable DETECT_SOFTLOCKUP for s390") because they got too many false positives. > > > > - depends on HAVE_HARDLOCKUP_DETECTOR_PERF || HAVE_HARDLOCKUP_DETECTOR_ARCH > > > + depends on HAVE_HARDLOCKUP_DETECTOR_NON_ARCH || HAVE_HARDLOCKUP_DETECTOR_ARCH > > > select LOCKUP_DETECTOR > > > - select HARDLOCKUP_DETECTOR_PERF if HAVE_HARDLOCKUP_DETECTOR_PERF > > > + select HARDLOCKUP_DETECTOR_NON_ARCH if HAVE_HARDLOCKUP_DETECTOR_NON_ARCH > > > > Anyway, the configuration of the hard lockup detectors is insane and > > this patchset makes it even worse, especially the new > > HARDLOCKUP_DETECTOR_NON_ARCH stuff. > > > > It seems that sparc, powerpc and s390 are somehow special. Do you > > still have in mind how they are distinguished using the Kconfig > > variables? > > > > For example, I am pretty confused by the meaning of HAVE_NMI_WATCHDOG. > > > > And sparc has its own variant of > > watchdog_hardlockup_enable()/disable(). It means that it is > > arch-specific. Does it work with the 13th patch which made > > watchdog_hardlockup_enable()/disable() to be watchdog-hardlockup-type > > specific? Is is somehow related to HAVE_NMI_WATCHDOG? > > Does this replace the entire watchdog only only the enable part? > > > > I think that we need to make this more straightforward. But I first > > need to understand the existing maze of config variables. > > I agree that it's confusing. I'm obviously biased, but IMO it's less > confusing after my patchset than before. ;-) The state of the world > before my patchset set a pretty low bar. > > As far as I understand it, at an architecture-level you can choose any > _ONE_ of the following: > > a) Implement bits needed for the the "perf" hardlockup detector. x86 > has done this, some configs of powerpc do this, and arm64 now after my > patch series. This is HAVE_HARDLOCKUP_DETECTOR_PERF. > > b) Implement your own totally separate hardlockup detector that > doesn't use any of the common "perf" code but still looks the same to > userspace (same sysctls, etc). Only powerpc does this (in some > configs). As per conversations in previous versions of my patch > series, apparently powerpc's version is quite fancy and maybe someday > people can move some of these features to the common code. This is > HAVE_HARDLOCKUP_DETECTOR_ARCH. > > c) Don't implement the full features of a hardlockup detector but > still have the basics. In the very least, I think it doesn't support > the sysctls "hardlockup_panic" and "hardlockup_all_cpu_backtrace". It > doesn't support the kernel command line parameter "nmi_watchdog=". I > don't know for sure if there are any other differences. Only sparc64 > does this. This is HAVE_NMI_WATCHDOG. Confusingly, > HAVE_HARDLOCKUP_DETECTOR_ARCH selects HAVE_NMI_WATCHDOG. > > d) Don't implement _any_ hardlockup detector of any sort. After my > patchset you can still end up with "buddy" if you have SMP. > > One thing that would probably help would be to bring sparc64 to a full > "arch" hardlockup implementation and then get rid of the special case. > That seems a bit outside my scope, though if someone wanted to post > patches for that I'd be willing to give them a review. It would be nice but it might be problematic if we do not have access to the hardware. > I guess other than that, the best we could try to do is to rename some > configs and/or add some subconfigs to describe certain features? Maybe > HAVE_NMI_WATCHDOG => HAVE_HARDLOCKUP_DETECTOR_ARCH_BASIC_FEATURES > would help? I'd love to come up with a better name for > HAVE_HARDLOCKUP_DETECTOR_NON_ARCH but I couldn't come up with one. > Maybe the unwieldy "HAVE_HARDLOCKUP_DETECTOR_THAT_COUNTS_HRTIMER"? Renaming the config variables seems to be the best solution at the moment. IMHO, it would be nice to have something like: + CONFIG_HARDLOCKUP_DETECTOR for code shared by all hardlockup detectors + CONFIG_HARDLOCKUP_DETECTOR_PERF for code using kernel/watchdog_perf.c + CONFIG_HARDLOCKUP_DETECTOR_BUDDY for code using kernel/watchdog_buddy.c + HAVE_HARDLOCKUP_DETECTOR_PERF set by architectures that support using kernel/watchdog_perf.c + HAVE_HARDLOCKUP_DETECTOR_ARCH set by architectures that have alternative implementation of the hardlockup detector + CONFIG_HARDLOCKUP_DETECTOR_PREFER_BUDDY Allow to prefer the buddy detector when _PERF or _ARCH is available as well. + HAVE_HARDLOCKUP_PANIC + HAVE_HARDLOCKUP_ALL_CPU_BACKTRACE set when the earchitecture support these features and used for the sysfs interface > If you have concrete suggestions for what would be cleaner, let me > know and I can queue up a patch. ...or I'm happy to review a patch. I am not sure how complicated it would be to rename the config variables to somehing sane. I am sorry I do not have time to prepare the patches at the moment. I'll let Andrew to decide if he would require this cleanup to accept the patchset. Best Regards, Petr From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4CD04C77B7C for ; Fri, 26 May 2023 12:30:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=qaGy1QvYjhcmpctRsE/0tkwYXd+b55ngkO4XtwwQfzg=; b=B53niHpOjjH5li N7uBk/fM7GSV3nhzHwCAN3DcEB7SCEtLa/b0JrhBENzkhsYctttTrLb1j6bIeom3x9/ceOgoNA6K7 WfWbM+ZRGMgIPxVi+6XqEYLu/crTSkmrG0gC6zxOIqZi0w1phTt7vB8t7M9tV1AEhLiJOrF2pouog AnqPutzxpuOxy1SRsreWASz5WCuCcfTNYUPgFz1MBeEQWIUg81SbMjVTJqrgg9ejkjJj8WPk9py+S BmpSQt63TqU0W899VPwK7CH21h/R9F8YeQaZZeiVS7L+VO4/g4cvkpeLcn1jXh2sw4IGcS1D/mask uSwI+m8KqUkgFslr58ng==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1q2Wa5-002Tyc-22; Fri, 26 May 2023 12:30:05 +0000 Received: from smtp-out2.suse.de ([2001:67c:2178:6::1d]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1q2Wa1-002Twf-0i for linux-arm-kernel@lists.infradead.org; Fri, 26 May 2023 12:30:04 +0000 Received: from relay2.suse.de (relay2.suse.de [149.44.160.134]) by smtp-out2.suse.de (Postfix) with ESMTP id 986451F8AE; Fri, 26 May 2023 12:29:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1685104199; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=WhfCogRNdmmsvJU5CjDqdyyExEkgp9ppc07tbwxSsOg=; b=UBlT9ISwUDHYPeMb+a+ztXThl86vi9IDG6M9ms/XS0tVW0/E2hbhprn7RRLeyq1C45+D/b UHhvn1UnkoPvxRIU7hyiv3JpraRb3fXflpZJXSH2dOTVPbnALq5BAMR9pbIBcE99YuCZJK S6IfNcRb2vDTizhuODQNL5WRubLUMN8= Received: from suse.cz (pmladek.tcp.ovpn2.prg.suse.de [10.100.208.146]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by relay2.suse.de (Postfix) with ESMTPS id 3EB022D38D; Fri, 26 May 2023 12:29:56 +0000 (UTC) Date: Fri, 26 May 2023 14:29:53 +0200 From: Petr Mladek To: Doug Anderson Cc: Andrew Morton , Matthias Kaehlcke , kgdb-bugreport@lists.sourceforge.net, Stephane Eranian , mpe@ellerman.id.au, Tzung-Bi Shih , Daniel Thompson , Mark Rutland , linuxppc-dev@lists.ozlabs.org, Sumit Garg , npiggin@gmail.com, davem@davemloft.net, Marc Zyngier , Stephen Boyd , sparclinux@vger.kernel.org, christophe.leroy@csgroup.eu, Catalin Marinas , ravi.v.shankar@intel.com, Randy Dunlap , Pingfan Liu , Guenter Roeck , Lecopzer Chen , Ian Rogers , ito-yuichi@fujitsu.com, ricardo.neri@intel.com, linux-arm-kernel@lists.infradead.org, linux-perf-users@vger.kernel.org, Will Deacon , Chen-Yu Tsai , linux-kernel@vger.kernel.org, Masayoshi Mizuma , Andi Kleen , Colin Cross Subject: Re: [PATCH v5 14/18] watchdog/hardlockup: detect hard lockups using secondary (buddy) CPUs Message-ID: References: <20230519101840.v5.18.Ia44852044cdcb074f387e80df6b45e892965d4a1@changeid> <20230519101840.v5.14.I6bf789d21d0c3d75d382e7e51a804a7a51315f2c@changeid> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230526_053001_553743_F5BFC33D X-CRM114-Status: GOOD ( 88.63 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org T24gVGh1IDIwMjMtMDUtMjUgMTM6MDg6MDQsIERvdWcgQW5kZXJzb24gd3JvdGU6Cj4gSGksCj4g Cj4gT24gVGh1LCBNYXkgMjUsIDIwMjMgYXQgOToyN+KAr0FNIFBldHIgTWxhZGVrIDxwbWxhZGVr QHN1c2UuY29tPiB3cm90ZToKPiA+Cj4gPiBPbiBGcmkgMjAyMy0wNS0xOSAxMDoxODozOCwgRG91 Z2xhcyBBbmRlcnNvbiB3cm90ZToKPiA+ID4gSW1wbGVtZW50IGEgaGFyZGxvY2t1cCBkZXRlY3Rv ciB0aGF0IGRvZXNuJ3QgZG9lc24ndCBuZWVkIGFueSBleHRyYQo+ID4gPiBhcmNoLXNwZWNpZmlj IHN1cHBvcnQgY29kZSB0byBkZXRlY3QgbG9ja3Vwcy4gSW5zdGVhZCBvZiB1c2luZwo+ID4gPiBz b21ldGhpbmcgYXJjaC1zcGVjaWZpYyB3ZSB3aWxsIHVzZSB0aGUgYnVkZHkgc3lzdGVtLCB3aGVy ZSBlYWNoIENQVQo+ID4gPiB3YXRjaGVzIG91dCBmb3IgYW5vdGhlciBvbmUuIFNwZWNpZmljYWxs eSwgZWFjaCBDUFUgd2lsbCB1c2UgaXRzCj4gPiA+IHNvZnRsb2NrdXAgaHJ0aW1lciB0byBjaGVj ayB0aGF0IHRoZSBuZXh0IENQVSBpcyBwcm9jZXNzaW5nIGhydGltZXIKPiA+ID4gaW50ZXJydXB0 cyBieSB2ZXJpZnlpbmcgdGhhdCBhIGNvdW50ZXIgaXMgaW5jcmVhc2luZy4KPiA+ID4KPiA+ID4g LS0tIGEva2VybmVsL3dhdGNoZG9nLmMKPiA+ID4gKysrIGIva2VybmVsL3dhdGNoZG9nLmMKPiA+ ID4gQEAgLTg1LDcgKzg1LDcgQEAgX19zZXR1cCgibm1pX3dhdGNoZG9nPSIsIGhhcmRsb2NrdXBf cGFuaWNfc2V0dXApOwo+ID4gPgo+ID4gPiAgI2VuZGlmIC8qIENPTkZJR19IQVJETE9DS1VQX0RF VEVDVE9SICovCj4gPiA+Cj4gPiA+IC0jaWYgZGVmaW5lZChDT05GSUdfSEFSRExPQ0tVUF9ERVRF Q1RPUl9QRVJGKQo+ID4gPiArI2lmIGRlZmluZWQoQ09ORklHX0hBUkRMT0NLVVBfREVURUNUT1Jf Q09VTlRTX0hSVElNRVIpCj4gPiA+Cj4gPiA+ICBzdGF0aWMgREVGSU5FX1BFUl9DUFUoYXRvbWlj X3QsIGhydGltZXJfaW50ZXJydXB0cyk7Cj4gPiA+ICBzdGF0aWMgREVGSU5FX1BFUl9DUFUoaW50 LCBocnRpbWVyX2ludGVycnVwdHNfc2F2ZWQpOwo+ID4gPiBAQCAtMTA2LDYgKzEwNiwxNCBAQCBu b3RyYWNlIHZvaWQgYXJjaF90b3VjaF9ubWlfd2F0Y2hkb2codm9pZCkKPiA+ID4gIH0KPiA+ID4g IEVYUE9SVF9TWU1CT0woYXJjaF90b3VjaF9ubWlfd2F0Y2hkb2cpOwo+ID4gPgo+ID4gPiArdm9p ZCB3YXRjaGRvZ19oYXJkbG9ja3VwX3RvdWNoX2NwdSh1bnNpZ25lZCBpbnQgY3B1KQo+ID4gPiAr ewo+ID4gPiArICAgICBwZXJfY3B1KHdhdGNoZG9nX2hhcmRsb2NrdXBfdG91Y2hlZCwgY3B1KSA9 IHRydWU7Cj4gPiA+ICsKPiA+ID4gKyAgICAgLyogTWF0Y2ggd2l0aCBzbXBfcm1iKCkgaW4gd2F0 Y2hkb2dfaGFyZGxvY2t1cF9jaGVjaygpICovCj4gPiA+ICsgICAgIHNtcF93bWIoKTsKPiA+Cj4g PiBJdCBpcyBncmVhdCB0aGF0IHlvdSBkZXNjcmliZWQgd2hlcmUgdGhlIHJlbGF0ZWQgYmFycmll ciBpcy4KPiA+Cj4gPiBBbm90aGVyIGltcG9ydGFudCBpbmZvcm1hdGlvbiBpcyB3aGF0IGV4YWN0 bHkgaXMgc3luY2hyb25pemVkLgo+ID4gQW5kIEkgYW0gYWN0dWFsbHkgbm90IHN1cmUgd2hhdCB3 ZSBhcmUgc3luY2hyb25pemluZyBoZXJlLgo+ID4KPiA+IE15IHVuZGVyc3RhbmRpbmcgaXMgdGhh dCBhIHdyaXRlIGJhcnJpZXIgc2hvdWxkIHN5bmNocm9uaXplCj4gPiByZWxhdGVkIHdyaXRlcywg Zm9yIGV4YW1wbGU6Cj4gPgo+ID4gICAgICAgICBYID0gLi4uOwo+ID4gICAgICAgICAvKiBNYWtl IHN1cmUgdGhhdCBYIGlzIG1vZGlmaWVkIGJlZm9yZSBZICovCj4gPiAgICAgICAgIHNtcF93bWIo KTsKPiA+ICAgICAgICAgWSA9IC4uLjsKPiA+Cj4gPiBBbmQgdGhlIHJlbGF0ZWQgcmVhZCBiYXJy aWVyIHNob3VsZCBzeW5jaHJvbml6ZSB0aGUgcmVsYXRlZCByZWFkcywKPiA+IGZvciBleGFtcGxl Ogo+ID4KPiA+ICAgICAgICAgaWYgKHRlc3QoWSkpIHsKPiA+ICAgICAgICAgICAgICAgICAvKgo+ ID4gICAgICAgICAgICAgICAgICAqIE1ha2Ugc3VyZSB0aGF0IHdlIHVzZSB0aGUgdXBkYXRlZCBY IHdoZW4KPiA+ICAgICAgICAgICAgICAgICAgKiB3ZSBzYXcgdGhlIHVwZGF0ZWQgWS4KPiA+ICAg ICAgICAgICAgICAgICAgKi8KPiA+ICAgICAgICAgICAgICAgICAgc21wX3JtYigpOwo+ID4gICAg ICAgICAgICAgICAgICBkb19zb21ldGhpbmcoWCk7Cj4gPiAgICAgICAgICB9Cj4gPgo+ID4gSU1I Tywgd2UgZG8gbm90IG5lZWQgYW55IGJhcnJpZXIgaGVyZSBiZWNhdXNlIHdlIGhhdmUgb25seQo+ ID4gb25lIHZhcmlhYmxlICJ3YXRjaGRvZ19oYXJkbG9ja3VwX3RvdWNoZWQiIGhlcmUuCj4gPiB3 YXRjaGRvZ19oYXJkbG9ja3VwX2NoZWNrKCkgd2lsbCBlaXRoZXIgc2VlIHRoZSB1cGRhdGVkIHZh bHVlCj4gPiBvciBub3QuIEJ1dCBpdCBkb2VzIG5vdCBzeW5jaHJvbml6ZSBpdCBhZ2FpbnN0IGFu eSBvdGhlcgo+ID4gdmFyaWFibGVzIG9yIHZhbHVlcy4KPiAKPiBGYWlyLiBUaGVzZSBiYXJyaWVy cyB3ZXJlIHByZXNlbnQgaW4gdGhlIG9yaWdpbmFsIGJ1ZGR5IGxvY2t1cAo+IGRldGVjdG9yIHRo YXQgd2UndmUgYmVlbiBjYXJyeWluZyBpbiBDaHJvbWVPUyBidXQgdGhhdCBkb2Vzbid0Cj4gbmVj ZXNzYXJpbHkgbWVhbiB0aGF0IHRoZXkgd2VyZSB0aGVyZSBmb3IgYSBnb29kIHJlYXNvbi4KPiAK PiBSZWFzb25pbmcgYWJvdXQgd2Vha2x5IG9yZGVyZWQgbWVtb3J5IGFsd2F5cyBtYWtlcyBteSBi cmFpbiBodXJ0IGFuZCBJCj4gbmV2ZXIgZmVlbCBjb25maWRlbnQgYXQgdGhlIGVuZCB0aGF0IEkg Z290IHRoZSByaWdodCBhbnN3ZXIgYW5kLCBvZgo+IGNvdXJzZSwgdGhpcyBpcyBjb3VwbGVkIGJ5 IHRoZSBmYWN0IHRoYXQgaWYgSSBoYXZlIGEgbG9naWMgZXJyb3IgaW4gbXkKPiByZWFzb25pbmcg dGhhdCBpdCBtaWdodCBjYXVzZSBhIHJhcmUgLyBzdWJ0bGUgYnVnLiA6KAoKU3VyZS4gTG9ja2xl c3MgY29kZSBpcyBjb21wbGljYXRlZC4KCj4gV2hlbiBwb3NzaWJsZSBJCj4gdHJ5IHRvIHdyaXRl IGNvZGUgdGhhdCB1c2VzIGZ1bGwgYmxvd24gbG9ja3MgdG8gbWFrZSBpdCBlYXNpZXIgdG8KPiBy ZWFzb24gYWJvdXQgKGV2ZW4gaWYgbGVzcyBlZmZpY2llbnQpLAoKTWFrZXMgc2Vuc2UuIFRoZXJl IHNob3VsZCBiZSBhIGdvb2QgcmVhc29uIHRvIHVzZSBsb2NrbGVzcyBjb2RlCmJlY2F1c2UgaXQg aXMgY29tcGxpY2F0ZWQgdG8gZG8gaXQgcmlnaHQgYW5kIG1haW50YWluLgoKPiBidXQgdGhhdCdz IG5vdCBuZWNlc3NhcmlseQo+IHBvc3NpYmxlIGhlcmUuIFdoaWxlIHdlIG9idmlvdXNseSBkb24n dCBqdXN0IHdhbnQgdG8gc3ByaW5rbGUgYmFycmllcnMKPiBhbGwgb3ZlciB0aGUgY29kZSwgSU1P IGl0J3Mgbm90IGEgdGVycmlibGUgc2luIHRvIHB1dCBhIGJhcnJpZXIgaW4gYQo+IGNhc2Ugd2hl cmUgaXQgbWFrZXMgaXQgZWFzaWVyIHRvIHJlYXNvbiBhYm91dCB0aGUgb3JkZXIgb2YgdGhpbmdz LgoKSSB1bmRlcnN0YW5kIHRoaXMuIFdlbGwsIGl0IGlzIGFsd2F5cyBpbXBvcnRhbnQgdG8gZGVz Y3JpYmUgdGhlCnRoZSByZWFzb24gd2h5IHRoZSBiYXJyaWVyIHdhcyBhZGRlZCB0aGVyZS4gRXZl biB3aGVuIGl0IGlzIHdyb25nLAppdCBnaXZlcyBhIGNsdWUgd2hhdCB3YXMgdGhlIG1vdGl2YXRp b24uIE90aGVyd2lzZSwgaXQgaXMgaGFyZAp0byBkbyBhbnkgY2hhbmdlcyBvbiB0aGUgY29kZSBs YXRlci4KCkkgZ3Vlc3MgdGhhdCBpdCBtaWdodCBiZSBtb3JlIHByb2JsZW1hdGljIGZvciB5b3Ug YmVjYXVzZQp5b3UgcHJvYmFibHkgYXJlIG5vdCB0aGUgb3JpZ2luYWwgYXV0aG9yLgoKPiBJbiBh bnkgY2FzZSwgSSBndWVzcyBpbiB0aGlzIGNhc2UgSSB3b3VsZCB3b3JyeSBhYm91dCBzb21lIHNv cnQgb2YKPiBvcmRlcmluZyByYWNlIHdoZW4gZW5hYmxpbmcgLyBkaXNhYmxpbmcgdGhlIGJ1ZGR5 IGxvY2t1cCBkZXRlY3Rvci4gQXQKPiB0aGUgZW5kIG9mIHRoZSBidWRkeSdzIHdhdGNoZG9nX2hh cmRsb2NrdXBfZW5hYmxlKCkgLwo+IHdhdGNoZG9nX2hhcmRsb2NrdXBfZGlzYWJsZSgpIHdlIGFk anVzdCB0aGUgIndhdGNoZG9nX2NwdXMiIHdoaWNoCj4gY2hhbmdlcyBidWRkeSBhc3NpZ25tZW50 cy4gV2l0aG91dCBhIGJhcnJpZXIsIEkgX3RoaW5rXyBpdCB3b3VsZCBiZQo+IHBvc3NpYmxlIGZv ciBhIG5ldyBDUFUgdG8gbm90aWNlIHRoZSBjaGFuZ2UgaW4gYnVkZGllcyB3aXRob3V0Cj4gbm90 aWNpbmcgdGhlIHRvdWNoLiBEb2VzIHRoYXQgbWF0Y2ggeW91ciB1bmRlcnN0YW5kaW5nPyBOb3cg d2hlbgo+IHJlYXNvbmluZyBhYm91dCBDUFVzIGdvaW5nIG9ubGluZS9vZmZsaW5lIHdlIG5lZWQg dG8gY29uc2lkZXIgdGhpcwo+IGV4dHJhIGNhc2UgYW5kIHdlIGhhdmUgdG8gZGVjaWRlIGlmIHRo ZXJlJ3MgYW55IGNoYW5jZSBpdCBjb3VsZCBsZWFkCj4gdG8gYSBmYWxzZSBsb2NrdXAgZGV0ZWN0 aW9uLiBXaXRoIHRoZSBtZW1vcnkgYmFycmllcnMgaGVyZSwgaXQncyBhCj4gbGl0dGxlIGVhc2ll ciB0byB0aGluayBhYm91dC4KClRoaXMgbWFrZXMgc2Vuc2UuIEkgZGlkIG5vdCB0aGluayBhYm91 dCB0aGUgaG90cGx1ZyBzY2VuYXJpby4KCldlbGwsIEkgc3VnZ2VzdCB0byBtb3ZlIHRoZSBiYXJy aWVycyBpbnRvIHRoZSBidWRkeSBjb2RlIGFuZCBkZXNjcmliZQppdCB0aGVyZS4gSXQgZG9lcyBu b3QgbWFrZSBzZW5zZSB0byB1c2UgdGhlIGJhcnJpZXJzIGZvciB0aGUgcGVyZgpoYXJkbG9ja3Vw LgoKSSBtZWFuIHNvbWV0aGluZyBsaWtlOgoKZGlmZiAtLWdpdCBhL2tlcm5lbC93YXRjaGRvZ19i dWRkeS5jIGIva2VybmVsL3dhdGNoZG9nX2J1ZGR5LmMKaW5kZXggZmVlNDVhZjJlNWJkLi5lYmU3 MWRjYjU1ZTYgMTAwNjQ0Ci0tLSBhL2tlcm5lbC93YXRjaGRvZ19idWRkeS5jCisrKyBiL2tlcm5l bC93YXRjaGRvZ19idWRkeS5jCkBAIC01Miw2ICs1MiwxMyBAQCB2b2lkIHdhdGNoZG9nX2hhcmRs b2NrdXBfZW5hYmxlKHVuc2lnbmVkIGludCBjcHUpCiAJaWYgKG5leHRfY3B1IDwgbnJfY3B1X2lk cykKIAkJd2F0Y2hkb2dfaGFyZGxvY2t1cF90b3VjaF9jcHUobmV4dF9jcHUpOwogCisJLyoKKwkg KiBNYWtlcyBzdXJlIHRoYXQgd2F0Y2hkb2cgaXMgdG91Y2hlZCBvbiB0aGlzIENQVSBiZWZvcmUK KwkgKiBvdGhlciBDUFVzIGNvdWxkIHNlZSBpdCBpbiB3YXRjaGRvZ19jcHVzLiBUaGUgY291bnRl cgorCSAqIHBhcnQgaXMgaW4gd2F0Y2hkb2dfYnVkZHlfY2hlY2tfaGFyZGxvY2t1cCgpLgorCSAq LworCXNtcF93bWIoKTsKKwogCWNwdW1hc2tfc2V0X2NwdShjcHUsICZ3YXRjaGRvZ19jcHVzKTsK IH0KIApAQCAtNjksNiArNzYsMTMgQEAgdm9pZCB3YXRjaGRvZ19oYXJkbG9ja3VwX2Rpc2FibGUo dW5zaWduZWQgaW50IGNwdSkKIAlpZiAobmV4dF9jcHUgPCBucl9jcHVfaWRzKQogCQl3YXRjaGRv Z19oYXJkbG9ja3VwX3RvdWNoX2NwdShuZXh0X2NwdSk7CiAKKwkvKgorCSAqIE1ha2VzIHN1cmUg dGhhdCB3YXRjaGRvZyBpcyB0b3VjaGVkIG9uIHRoZSBuZXh0IENQVSBiZWZvcmUKKwkgKiB0aGlz IENQVSBkaXNhcHBlYXIgaW4gd2F0Y2hkb2dfY3B1cy4gVGhlIGNvdW50ZXIgcGFydCBpcyBpbgor CSAqIHdhdGNoZG9nX2J1ZGR5X2NoZWNrX2hhcmRsb2NrdXAoKS4KKwkgKi8KKwlzbXBfd21iKCk7 CisKIAljcHVtYXNrX2NsZWFyX2NwdShjcHUsICZ3YXRjaGRvZ19jcHVzKTsKIH0KIApAQCAtODks NSArMTAzLDEyIEBAIHZvaWQgd2F0Y2hkb2dfYnVkZHlfY2hlY2tfaGFyZGxvY2t1cCh1bnNpZ25l ZCBsb25nIGhydGltZXJfaW50ZXJydXB0cykKIAlpZiAobmV4dF9jcHUgPj0gbnJfY3B1X2lkcykK IAkJcmV0dXJuOwogCisJLyoKKwkgKiBNYWtlIHN1cmUgdGhhdCB0aGUgd2F0Y2hkb2cgd2FzIHRv dWNoZWQgb24gbmV4dCBDUFUgd2hlbgorCSAqIHdhdGNoZG9nX25leHRfY3B1KCkgcmV0dXJuZWQg YW5vdGhlciBvbmUgYmVjYXVzZSBvZgorCSAqIGEgY2hhbmdlIGluIHdhdGNoZG9nX2hhcmRsb2Nr dXBfZW5hYmxlKCkvZGlzYWJsZSgpLgorCSAqLworCXNtcF9ybWIoKTsKKwogCXdhdGNoZG9nX2hh cmRsb2NrdXBfY2hlY2sobmV4dF9jcHUsIE5VTEwpOwogfQoKPiBEaWQgdGhlIGFib3ZlIGNvbnZp bmNlIHlvdSBhYm91dCBrZWVwaW5nIHRoZSBiYXJyaWVycz8gSWYgc28sIGRvIHlvdQo+IGhhdmUg YW55IHN1Z2dlc3RlZCBjb21tZW50IHRoYXQgd291bGQgbWFrZSBpdCBjbGVhcmVyPwo+IAo+IAo+ ID4gPiArfQo+ID4gPiArCj4gPiA+ICBzdGF0aWMgYm9vbCBpc19oYXJkbG9ja3VwKHVuc2lnbmVk IGludCBjcHUpCj4gPiA+ICB7Cj4gPiA+ICAgICAgIGludCBocmludCA9IGF0b21pY19yZWFkKCZw ZXJfY3B1KGhydGltZXJfaW50ZXJydXB0cywgY3B1KSk7Cj4gPiA+IEBAIC00NDMsMTEgKzQ1NCwx NSBAQCBzdGF0aWMgZW51bSBocnRpbWVyX3Jlc3RhcnQgd2F0Y2hkb2dfdGltZXJfZm4oc3RydWN0 IGhydGltZXIgKmhydGltZXIpCj4gPiA+ICAgICAgIHN0cnVjdCBwdF9yZWdzICpyZWdzID0gZ2V0 X2lycV9yZWdzKCk7Cj4gPiA+ICAgICAgIGludCBkdXJhdGlvbjsKPiA+ID4gICAgICAgaW50IHNv ZnRsb2NrdXBfYWxsX2NwdV9iYWNrdHJhY2UgPSBzeXNjdGxfc29mdGxvY2t1cF9hbGxfY3B1X2Jh Y2t0cmFjZTsKPiA+ID4gKyAgICAgdW5zaWduZWQgbG9uZyBocnRpbWVyX2ludGVycnVwdHM7Cj4g PiA+Cj4gPiA+ICAgICAgIGlmICghd2F0Y2hkb2dfZW5hYmxlZCkKPiA+ID4gICAgICAgICAgICAg ICByZXR1cm4gSFJUSU1FUl9OT1JFU1RBUlQ7Cj4gPiA+Cj4gPiA+IC0gICAgIHdhdGNoZG9nX2hh cmRsb2NrdXBfa2ljaygpOwo+ID4gPiArICAgICBocnRpbWVyX2ludGVycnVwdHMgPSB3YXRjaGRv Z19oYXJkbG9ja3VwX2tpY2soKTsKPiA+ID4gKwo+ID4gPiArICAgICAvKiB0ZXN0IGZvciBoYXJk bG9ja3VwcyAqLwo+ID4KPiA+IEkgd291bGQgb21pdCB0aGUgY29tbWVudC4gSXQgaXMgbm90IHZh bGlkIHdoZW4gcGVyZiBkZXRlY3RvciBpcyB1c2VkLgo+ID4gQW5kIGNoZWNraW5nIHRoZSBidWRk eSBpcyBjbGVhciBmcm9tIHRoZSBmdW5jdGlvbiBuYW1lLgo+ID4KPiA+ID4gKyAgICAgd2F0Y2hk b2dfYnVkZHlfY2hlY2tfaGFyZGxvY2t1cChocnRpbWVyX2ludGVycnVwdHMpOwo+ID4KPiA+IEkg d291bGQgcGVyc29uYWxseSBtb3ZlIHRoaXMgaW50byB3YXRjaGRvZ19oYXJkbG9ja3VwX2tpY2so KS4KPiA+IHdhdGNoZG9nX3RpbWVyX2ZuKCkgaXMgYWxyZWFkeSBjb21wbGV4IGVub3VnaC4gQW5k IGNoZWNraW5nCj4gPiB0aGUgYnVkZHkgd2hlbiBraWNraW5nIGEgQ1BVIG1ha2VzIHNlbnNlLgo+ IAo+IFN1cmUsIEknbGwgYWRkIHRoYXQgdG8gbXkgbGlzdCBvZiB0aGluZ3MgdG8gZm9sbG93LXVw IHdpdGguCj4gCj4gCj4gPiBBbHNvIEkgd291bGQgbm90IHBhc3MgImhydGltZXJfaW50ZXJydXB0 cyIuIEkgZ3Vlc3MgdGhhdCBpdCBpcwo+ID4ganVzdCBhbiBvcHRpbWl6YXRpb24uIEl0IGlzIGFu IGV4dHJhIGNodXJuIGluIHRoZSBjb2RlLiBJTUhPLAo+ID4gaXMgaXMgbm90IHdvcnQgaXQuIFRo aXMgY29kZSBkb2VzIG5vdCBuZWVkIHRvIGJlIHN1cGVyIG9wdGltaXplZC4KPiAKPiBUaGUgbWFp biByZWFzb24gSSBkaWQgaXQgaXMgdGhhdCAiaHJ0aW1lcl9pbnRlcnJ1cHRzIiBpcyBzdGF0aWMg dG8KPiB3YXRjaGRvZy5jIG5vdy4gSWYgSSBkb24ndCBwYXNzIGl0IGluIHRoZW4gSSBoYXZlIHRv IG1ha2UgaXQKPiBub24tc3RhdGljIGFuZCBhZGQgaXQgdG8gdGhlIGhlYWRlci4gVGhhdCBhbHNv IG1lYW5zIGFueW9uZSBsb29raW5nIGF0Cj4gdGhlIHZhcmlhYmxlIGFuZCBmaWd1cmluZyBvdXQg aG93IGl0IGlzIHJlYWQvd3JpdHRlbiBuZWVkcyB0byBnbwo+IHNlYXJjaCBmb3Igb3RoZXIgcGVv cGxlIHRoYXQgcmVmZXJlbmNlIGl0LiBJIGZlZWwgbGlrZSBpdCdzIGNsZWFuZXIgdG8KPiBqdXN0 IHBhc3MgaXQgaW4uIElmIHlvdSBmZWVsIHN0cm9uZ2x5IHRoYXQgSSBzaG91bGQgY2hhbmdlIHRo aXMgdGhlbgo+IGxldCBtZSBrbm93LCBidXQgb3RoZXJ3aXNlIEknbGwgcGxhbiB0byBsZWF2ZSB0 aGlzIGhvdyBJIGhhdmUgaXQuCgpJIGRvIG5vdCBoYXZlIHN0cm9uZyBvcGluaW9uLiBGb3IgbWUs IHRoZSBtb3JlIGltcG9ydGFudCBjaGFuZ2UgaXMKdG8gbW92ZSB3YXRjaGRvZ19idWRkeV9jaGVj a19oYXJkbG9ja3VwKCkgaW50bwp3YXRjaGRvZ19oYXJkbG9ja3VwX2tpY2soKS4gd2F0Y2hkb2df dGltZXJfZm4oKSBpcyBhbHJlYWR5IHRvbyBjb21wbGV4LgoKPiAKPiA+ID4gICAgICAgLyoga2lj ayB0aGUgc29mdGxvY2t1cCBkZXRlY3RvciAqLwo+ID4gPiAgICAgICBpZiAoY29tcGxldGlvbl9k b25lKHRoaXNfY3B1X3B0cigmc29mdGxvY2t1cF9jb21wbGV0aW9uKSkpIHsKPiA+ID4gLS0tIGEv bGliL0tjb25maWcuZGVidWcKPiA+ID4gKysrIGIvbGliL0tjb25maWcuZGVidWcKPiA+ID4gQEAg LTEwMzUsMTAgKzEwMzUsNTUgQEAgY29uZmlnIEJPT1RQQVJBTV9TT0ZUTE9DS1VQX1BBTklDCj4g PiA+Cj4gPiA+ICAgICAgICAgU2F5IE4gaWYgdW5zdXJlLgo+ID4gPgo+ID4gPiAtY29uZmlnIEhB UkRMT0NLVVBfREVURUNUT1JfUEVSRgo+ID4gPiArIyBCb3RoIHRoZSAicGVyZiIgYW5kICJidWRk eSIgaGFyZGxvY2t1cCBkZXRlY3RvcnMgY291bnQgaHJ0aW1lcgo+ID4gPiArIyBpbnRlcnJ1cHRz LiBUaGlzIGNvbmZpZyBlbmFibGVzIGZ1bmN0aW9ucyBtYW5hZ2luZyB0aGlzIGNvbW1vbiBjb2Rl Lgo+ID4gPiArY29uZmlnIEhBUkRMT0NLVVBfREVURUNUT1JfQ09VTlRTX0hSVElNRVIKPiA+ID4g ICAgICAgYm9vbAo+ID4gPiAgICAgICBzZWxlY3QgU09GVExPQ0tVUF9ERVRFQ1RPUgo+ID4gPgo+ ID4gPiArY29uZmlnIEhBUkRMT0NLVVBfREVURUNUT1JfUEVSRgo+ID4gPiArICAgICBib29sCj4g PiA+ICsgICAgIGRlcGVuZHMgb24gSEFWRV9IQVJETE9DS1VQX0RFVEVDVE9SX1BFUkYKPiA+ID4g KyAgICAgc2VsZWN0IEhBUkRMT0NLVVBfREVURUNUT1JfQ09VTlRTX0hSVElNRVIKPiA+ID4gKwo+ ID4gPiArY29uZmlnIEhBUkRMT0NLVVBfREVURUNUT1JfQlVERFkKPiA+ID4gKyAgICAgYm9vbAo+ ID4gPiArICAgICBkZXBlbmRzIG9uIFNNUAo+ID4gPiArICAgICBzZWxlY3QgSEFSRExPQ0tVUF9E RVRFQ1RPUl9DT1VOVFNfSFJUSU1FUgo+ID4gPiArCj4gPiA+ICsjIEZvciBoYXJkbG9ja3VwIGRl dGVjdG9ycyB5b3UgY2FuIGhhdmUgb25lIGRpcmVjdGx5IHByb3ZpZGVkIGJ5IHRoZSBhcmNoCj4g PiA+ICsjIG9yIHVzZSBhICJub24tYXJjaCIgb25lLiBJZiB5b3UncmUgdXNpbmcgYSAibm9uLWFy Y2giIG9uZSB0aGF0IGlzCj4gPiA+ICsjIGZ1cnRoZXIgZGl2aWRlZCB0aGUgcGVyZiBoYXJkbG9j a3VwIGRldGVjdG9yICh3aGljaCwgY29uZnVzaW5nbHksIG5lZWRzCj4gPiA+ICsjIGFyY2gtcHJv dmlkZWQgcGVyZiBzdXBwb3J0KSBhbmQgdGhlIGJ1ZGR5IGhhcmRsb2NrdXAgZGV0ZWN0b3IgKHdo aWNoIGp1c3QKPiA+ID4gKyMgbmVlZHMgU01QKS4gSW4gZWl0aGVyIGNhc2UsIHVzaW5nIHRoZSAi bm9uLWFyY2giIGNvZGUgY29uZmxpY3RzIHdpdGgKPiA+ID4gKyMgdGhlIE5NSSB3YXRjaGRvZyBj b2RlICh3aGljaCBpcyBzb21ldGltZXMgdXNlZCBkaXJlY3RseSBhbmQgc29tZXRpbWVzIHVzZWQK PiA+ID4gKyMgYnkgdGhlIGFyY2gtcHJvdmlkZWQgaGFyZGxvY2t1cCBkZXRlY3RvcikuCj4gPiA+ ICtjb25maWcgSEFWRV9IQVJETE9DS1VQX0RFVEVDVE9SX05PTl9BUkNICj4gPiA+ICsgICAgIGJv b2wKPiA+ID4gKyAgICAgZGVwZW5kcyBvbiAoSEFWRV9IQVJETE9DS1VQX0RFVEVDVE9SX1BFUkYg fHwgU01QKSAmJiAhSEFWRV9OTUlfV0FUQ0hET0cKPiA+ID4gKyAgICAgZGVmYXVsdCB5Cj4gPiA+ ICsKPiA+ID4gK2NvbmZpZyBIQVJETE9DS1VQX0RFVEVDVE9SX1BSRUZFUl9CVUREWQo+ID4gPiAr ICAgICBib29sICJQcmVmZXIgdGhlIGJ1ZGR5IENQVSBoYXJkbG9ja3VwIGRldGVjdG9yIgo+ID4g PiArICAgICBkZXBlbmRzIG9uIEhBVkVfSEFSRExPQ0tVUF9ERVRFQ1RPUl9OT05fQVJDSCAmJiBI QVZFX0hBUkRMT0NLVVBfREVURUNUT1JfUEVSRiAmJiBTTVAKPiA+Cj4gPiBIdWgsIEkgaGF2ZSBi aWcgdHJvdWJsZXMgdG8gc2NyYXRjaCBteSBoZWFkIGFyb3VuZCB0aGlzIGNoZWNrOgo+ID4KPiA+ ICAgICAgICBIQVZFX0hBUkRMT0NLVVBfREVURUNUT1JfTk9OX0FSQ0ggZGVwZW5kcyBvbiBIQVZF X0hBUkRMT0NLVVBfREVURUNUT1JfUEVSRiBhbmQgU01QCj4gPgo+ID4gICAgICAgIGFuZCB0aGlz IGRlcGVuZHMgb24gSEFWRV9IQVJETE9DS1VQX0RFVEVDVE9SX05PTl9BUkNIIGFuZCBhZ2Fpbgo+ ID4gICAgICAgICAgICAgICAgb24gSEFWRV9IQVJETE9DS1VQX0RFVEVDVE9SX1BFUkYgYW5kIFNN UC4KPiAKPiBUaGUgZ29hbCBpcyB0byBoYXZlICJIQVJETE9DS1VQX0RFVEVDVE9SX1BSRUZFUl9C VUREWSIgdG8gc2hvdyB1cCBhcwo+IGFuIG9wdGlvbiBpZiB0aGVyZSBpcyBhbiBvcHRpb24gX290 aGVyXyB0aGFuIHRoZSBidWRkeS4gSWYgdGhlcmUncyBub3QKPiBtb3JlIHRoYW4gb25lIGhhcmRs b2NrdXAgZGV0ZWN0b3IgdG8gcGljayBmcm9tIHRoZW4gdGhlcmUncyBubyByZWFzb24KPiB0byBh c2sgdGhlIHBlcnNvbiBjb25maWd1cmluZyB0aGUga2VybmVsIHdoaWNoIG9uZSB0aGV5J2QgcHJl ZmVyLiBBdAo+IHRoZSBtb21lbnQsIGlmIHlvdSBoYXZlIGFuICJhcmNoIiBsb2NrdXAgZGV0ZWN0 b3IgdGhlbiB5b3UncmUgc3R1Y2sKPiB3aXRoIGl0LCBzbyB5b3Ugb25seSBnZXQgYSBjaG9pY2Ug aWYgYSAicGVyZiIgZGV0ZWN0b3IgaXMgYXZhaWxhYmxlCj4gYW5kIHlvdSd2ZSBnb3QgU01QLgo+ IAo+IEFoLCBzbyBJIGd1ZXNzIHRoaXMgY291bGQgYmUgc2ltcGxpZmllZCB0bzoKPiAKPiBkZXBl bmRzIG9uIEhBVkVfSEFSRExPQ0tVUF9ERVRFQ1RPUl9QRVJGICYmIFNNUAoKWWVzLCB0aGlzIGlz IG11Y2ggYmV0dGVyLgoKPiBPSywgSSdsbCBhZGQgdGhhdCB0byB0aGUgbGlzdC4KPiAKPiAKPiA+ ID4gKyAgICAgaGVscAo+ID4gPiArICAgICAgIFNheSBZIGhlcmUgdG8gcHJlZmVyIHRoZSBidWRk eSBoYXJkbG9ja3VwIGRldGVjdG9yIG92ZXIgdGhlIHBlcmYgb25lLgo+ID4gPiArCj4gPiA+ICsg ICAgICAgV2l0aCB0aGUgYnVkZHkgZGV0ZWN0b3IsIGVhY2ggQ1BVIHVzZXMgaXRzIHNvZnRsb2Nr dXAgaHJ0aW1lcgo+ID4gPiArICAgICAgIHRvIGNoZWNrIHRoYXQgdGhlIG5leHQgQ1BVIGlzIHBy b2Nlc3NpbmcgaHJ0aW1lciBpbnRlcnJ1cHRzIGJ5Cj4gPiA+ICsgICAgICAgdmVyaWZ5aW5nIHRo YXQgYSBjb3VudGVyIGlzIGluY3JlYXNpbmcuCj4gPiA+ICsKPiA+ID4gKyAgICAgICBUaGlzIGhh cmRsb2NrdXAgZGV0ZWN0b3IgaXMgdXNlZnVsIG9uIHN5c3RlbXMgdGhhdCBkb24ndCBoYXZlCj4g PiA+ICsgICAgICAgYW4gYXJjaC1zcGVjaWZpYyBoYXJkbG9ja3VwIGRldGVjdG9yIG9yIGlmIHJl c291cmNlcyBuZWVkZWQKPiA+ID4gKyAgICAgICBmb3IgdGhlIGhhcmRsb2NrdXAgZGV0ZWN0b3Ig YXJlIGJldHRlciB1c2VkIGZvciBvdGhlciB0aGluZ3MuCj4gPiA+ICsKPiA+ID4gKyMgVGhpcyB3 aWxsIHNlbGVjdCB0aGUgYXBwcm9wcmlhdGUgbm9uLWFyY2ggaGFyZGxvY2tkdXAgZGV0ZWN0b3IK PiA+ID4gK2NvbmZpZyBIQVJETE9DS1VQX0RFVEVDVE9SX05PTl9BUkNICj4gPiA+ICsgICAgIGJv b2wKPiA+ID4gKyAgICAgZGVwZW5kcyBvbiBIQVZFX0hBUkRMT0NLVVBfREVURUNUT1JfTk9OX0FS Q0gKPiA+ID4gKyAgICAgc2VsZWN0IEhBUkRMT0NLVVBfREVURUNUT1JfQlVERFkgaWYgIUhBVkVf SEFSRExPQ0tVUF9ERVRFQ1RPUl9QRVJGIHx8IEhBUkRMT0NLVVBfREVURUNUT1JfUFJFRkVSX0JV RERZCj4gPiA+ICsgICAgIHNlbGVjdCBIQVJETE9DS1VQX0RFVEVDVE9SX1BFUkYgaWYgSEFWRV9I QVJETE9DS1VQX0RFVEVDVE9SX1BFUkYgJiYgIUhBUkRMT0NLVVBfREVURUNUT1JfUFJFRkVSX0JV RERZCj4gPiA+ICsKPiA+ID4gICMKPiA+ID4gICMgRW5hYmxlcyBhIHRpbWVzdGFtcCBiYXNlZCBs b3cgcGFzcyBmaWx0ZXIgdG8gY29tcGVuc2F0ZSBmb3IgcGVyZiBiYXNlZAo+ID4gPiAgIyBoYXJk IGxvY2t1cCBkZXRlY3Rpb24gd2hpY2ggcnVucyB0b28gZmFzdCBkdWUgdG8gdHVyYm8gbW9kZXMu Cj4gPiA+IEBAIC0xMDUzLDkgKzEwOTgsMTAgQEAgY29uZmlnIEhBUkRMT0NLVVBfQ0hFQ0tfVElN RVNUQU1QCj4gPiA+ICBjb25maWcgSEFSRExPQ0tVUF9ERVRFQ1RPUgo+ID4gPiAgICAgICBib29s ICJEZXRlY3QgSGFyZCBMb2NrdXBzIgo+ID4gPiAgICAgICBkZXBlbmRzIG9uIERFQlVHX0tFUk5F TCAmJiAhUzM5MAo+ID4KPiA+IElzIHRoZXJlIGFueSByZWFzb24gd2h5IFMzOTAgY291bGQgbm90 IG9yIGRvIG5vdCB3YW50IHRvIHVzZSB0aGUgYnVkZHkKPiA+IGhhcmRsb2NrdXAgZGV0ZWN0b3Iu Cj4gCj4gVGhpcyBpc24ndCBhIG5ldyBkZXBlbmRlbmN5LCBidXQgaXQncyBhIGdvb2QgcXVlc3Rp b24uIExvb2tpbmcgYXQgdGhlCj4gZ2l0IGhpc3RvcnksIEkgc2VlIGNvbW1pdCBkZWEyMGEzZmJk ZDAgKCJbUEFUQ0hdIERpc2FibGUKPiBERVRFQ1RfU09GVExPQ0tVUCBmb3IgczM5MCIpLiAuLi5h bmQgaXQgbG9va3MgbGlrZSB0aGUgc29mdGxvY2t1cAo+IGRldGVjdG9yIHN0aWxsIHNheXMgaXQn cyBicm9rZW4gb24gczM5MC4gVGhhdCB3b3VsZCBtZWFuIHRoYXQgdGhlCj4gYnVkZHkgZGV0ZWN0 b3IgaXMgYnJva2VuIHRvby4KCkl0IHNlZW1zIHRoYXQgczM5MCB3YW50ZWQgdG8gZGlzYWJsZSB0 aGUgd2F0Y2hkb2cgY29tcGxldGVseSwgc2VlCnRoZSBjb21taXQgIGRlYTIwYTNmYmRkMDhlNSAo IltQQVRDSF0gRGlzYWJsZSBERVRFQ1RfU09GVExPQ0tVUCBmb3IgczM5MCIpCmJlY2F1c2UgdGhl eSBnb3QgdG9vIG1hbnkgZmFsc2UgcG9zaXRpdmVzLgoKPiAKPiA+ID4gLSAgICAgZGVwZW5kcyBv biBIQVZFX0hBUkRMT0NLVVBfREVURUNUT1JfUEVSRiB8fCBIQVZFX0hBUkRMT0NLVVBfREVURUNU T1JfQVJDSAo+ID4gPiArICAgICBkZXBlbmRzIG9uIEhBVkVfSEFSRExPQ0tVUF9ERVRFQ1RPUl9O T05fQVJDSCB8fCBIQVZFX0hBUkRMT0NLVVBfREVURUNUT1JfQVJDSAo+ID4gPiAgICAgICBzZWxl Y3QgTE9DS1VQX0RFVEVDVE9SCj4gPiA+IC0gICAgIHNlbGVjdCBIQVJETE9DS1VQX0RFVEVDVE9S X1BFUkYgaWYgSEFWRV9IQVJETE9DS1VQX0RFVEVDVE9SX1BFUkYKPiA+ID4gKyAgICAgc2VsZWN0 IEhBUkRMT0NLVVBfREVURUNUT1JfTk9OX0FSQ0ggaWYgSEFWRV9IQVJETE9DS1VQX0RFVEVDVE9S X05PTl9BUkNICj4gPgo+ID4gQW55d2F5LCB0aGUgY29uZmlndXJhdGlvbiBvZiB0aGUgaGFyZCBs b2NrdXAgZGV0ZWN0b3JzIGlzIGluc2FuZSBhbmQKPiA+IHRoaXMgcGF0Y2hzZXQgbWFrZXMgaXQg ZXZlbiB3b3JzZSwgZXNwZWNpYWxseSB0aGUgbmV3Cj4gPiBIQVJETE9DS1VQX0RFVEVDVE9SX05P Tl9BUkNIIHN0dWZmLgo+ID4KPiA+IEl0IHNlZW1zIHRoYXQgc3BhcmMsIHBvd2VycGMgYW5kIHMz OTAgYXJlIHNvbWVob3cgc3BlY2lhbC4gRG8geW91Cj4gPiBzdGlsbCBoYXZlIGluIG1pbmQgaG93 IHRoZXkgYXJlIGRpc3Rpbmd1aXNoZWQgdXNpbmcgdGhlIEtjb25maWcKPiA+IHZhcmlhYmxlcz8K PiA+Cj4gPiBGb3IgZXhhbXBsZSwgSSBhbSBwcmV0dHkgY29uZnVzZWQgYnkgdGhlIG1lYW5pbmcg b2YgSEFWRV9OTUlfV0FUQ0hET0cuCj4gPgo+ID4gQW5kIHNwYXJjIGhhcyBpdHMgb3duIHZhcmlh bnQgb2YKPiA+IHdhdGNoZG9nX2hhcmRsb2NrdXBfZW5hYmxlKCkvZGlzYWJsZSgpLiBJdCBtZWFu cyB0aGF0IGl0IGlzCj4gPiBhcmNoLXNwZWNpZmljLiBEb2VzIGl0IHdvcmsgd2l0aCB0aGUgMTN0 aCBwYXRjaCB3aGljaCBtYWRlCj4gPiB3YXRjaGRvZ19oYXJkbG9ja3VwX2VuYWJsZSgpL2Rpc2Fi bGUoKSB0byBiZSB3YXRjaGRvZy1oYXJkbG9ja3VwLXR5cGUKPiA+IHNwZWNpZmljPyBJcyBpcyBz b21laG93IHJlbGF0ZWQgdG8gSEFWRV9OTUlfV0FUQ0hET0c/Cj4gPiBEb2VzIHRoaXMgcmVwbGFj ZSB0aGUgZW50aXJlIHdhdGNoZG9nIG9ubHkgb25seSB0aGUgZW5hYmxlIHBhcnQ/Cj4gPgo+ID4g SSB0aGluayB0aGF0IHdlIG5lZWQgdG8gbWFrZSB0aGlzIG1vcmUgc3RyYWlnaHRmb3J3YXJkLiBC dXQgSSBmaXJzdAo+ID4gbmVlZCB0byB1bmRlcnN0YW5kIHRoZSBleGlzdGluZyBtYXplIG9mIGNv bmZpZyB2YXJpYWJsZXMuCj4gCj4gSSBhZ3JlZSB0aGF0IGl0J3MgY29uZnVzaW5nLiBJJ20gb2J2 aW91c2x5IGJpYXNlZCwgYnV0IElNTyBpdCdzIGxlc3MKPiBjb25mdXNpbmcgYWZ0ZXIgbXkgcGF0 Y2hzZXQgdGhhbiBiZWZvcmUuIDstKSBUaGUgc3RhdGUgb2YgdGhlIHdvcmxkCj4gYmVmb3JlIG15 IHBhdGNoc2V0IHNldCBhIHByZXR0eSBsb3cgYmFyLgo+IAo+IEFzIGZhciBhcyBJIHVuZGVyc3Rh bmQgaXQsIGF0IGFuIGFyY2hpdGVjdHVyZS1sZXZlbCB5b3UgY2FuIGNob29zZSBhbnkKPiBfT05F XyBvZiB0aGUgZm9sbG93aW5nOgo+IAo+IGEpIEltcGxlbWVudCBiaXRzIG5lZWRlZCBmb3IgdGhl IHRoZSAicGVyZiIgaGFyZGxvY2t1cCBkZXRlY3Rvci4geDg2Cj4gaGFzIGRvbmUgdGhpcywgc29t ZSBjb25maWdzIG9mIHBvd2VycGMgZG8gdGhpcywgYW5kIGFybTY0IG5vdyBhZnRlciBteQo+IHBh dGNoIHNlcmllcy4gVGhpcyBpcyBIQVZFX0hBUkRMT0NLVVBfREVURUNUT1JfUEVSRi4KPiAKPiBi KSBJbXBsZW1lbnQgeW91ciBvd24gdG90YWxseSBzZXBhcmF0ZSBoYXJkbG9ja3VwIGRldGVjdG9y IHRoYXQKPiBkb2Vzbid0IHVzZSBhbnkgb2YgdGhlIGNvbW1vbiAicGVyZiIgY29kZSBidXQgc3Rp bGwgbG9va3MgdGhlIHNhbWUgdG8KPiB1c2Vyc3BhY2UgKHNhbWUgc3lzY3RscywgZXRjKS4gT25s eSBwb3dlcnBjIGRvZXMgdGhpcyAoaW4gc29tZQo+IGNvbmZpZ3MpLiBBcyBwZXIgY29udmVyc2F0 aW9ucyBpbiBwcmV2aW91cyB2ZXJzaW9ucyBvZiBteSBwYXRjaAo+IHNlcmllcywgYXBwYXJlbnRs eSBwb3dlcnBjJ3MgdmVyc2lvbiBpcyBxdWl0ZSBmYW5jeSBhbmQgbWF5YmUgc29tZWRheQo+IHBl b3BsZSBjYW4gbW92ZSBzb21lIG9mIHRoZXNlIGZlYXR1cmVzIHRvIHRoZSBjb21tb24gY29kZS4g VGhpcyBpcwo+IEhBVkVfSEFSRExPQ0tVUF9ERVRFQ1RPUl9BUkNILgo+IAo+IGMpIERvbid0IGlt cGxlbWVudCB0aGUgZnVsbCBmZWF0dXJlcyBvZiBhIGhhcmRsb2NrdXAgZGV0ZWN0b3IgYnV0Cj4g c3RpbGwgaGF2ZSB0aGUgYmFzaWNzLiBJbiB0aGUgdmVyeSBsZWFzdCwgSSB0aGluayBpdCBkb2Vz bid0IHN1cHBvcnQKPiB0aGUgc3lzY3RscyAiaGFyZGxvY2t1cF9wYW5pYyIgYW5kICJoYXJkbG9j a3VwX2FsbF9jcHVfYmFja3RyYWNlIi4gSXQKPiBkb2Vzbid0IHN1cHBvcnQgdGhlIGtlcm5lbCBj b21tYW5kIGxpbmUgcGFyYW1ldGVyICJubWlfd2F0Y2hkb2c9Ii4gSQo+IGRvbid0IGtub3cgZm9y IHN1cmUgaWYgdGhlcmUgYXJlIGFueSBvdGhlciBkaWZmZXJlbmNlcy4gT25seSBzcGFyYzY0Cj4g ZG9lcyB0aGlzLiBUaGlzIGlzIEhBVkVfTk1JX1dBVENIRE9HLiBDb25mdXNpbmdseSwKPiBIQVZF X0hBUkRMT0NLVVBfREVURUNUT1JfQVJDSCBzZWxlY3RzIEhBVkVfTk1JX1dBVENIRE9HLgo+IAo+ IGQpIERvbid0IGltcGxlbWVudCBfYW55XyBoYXJkbG9ja3VwIGRldGVjdG9yIG9mIGFueSBzb3J0 LiBBZnRlciBteQo+IHBhdGNoc2V0IHlvdSBjYW4gc3RpbGwgZW5kIHVwIHdpdGggImJ1ZGR5IiBp ZiB5b3UgaGF2ZSBTTVAuCj4KPiBPbmUgdGhpbmcgdGhhdCB3b3VsZCBwcm9iYWJseSBoZWxwIHdv dWxkIGJlIHRvIGJyaW5nIHNwYXJjNjQgdG8gYSBmdWxsCj4gImFyY2giIGhhcmRsb2NrdXAgaW1w bGVtZW50YXRpb24gYW5kIHRoZW4gZ2V0IHJpZCBvZiB0aGUgc3BlY2lhbCBjYXNlLgo+IFRoYXQg c2VlbXMgYSBiaXQgb3V0c2lkZSBteSBzY29wZSwgdGhvdWdoIGlmIHNvbWVvbmUgd2FudGVkIHRv IHBvc3QKPiBwYXRjaGVzIGZvciB0aGF0IEknZCBiZSB3aWxsaW5nIHRvIGdpdmUgdGhlbSBhIHJl dmlldy4KCkl0IHdvdWxkIGJlIG5pY2UgYnV0IGl0IG1pZ2h0IGJlIHByb2JsZW1hdGljIGlmIHdl IGRvIG5vdCBoYXZlCmFjY2VzcyB0byB0aGUgaGFyZHdhcmUuCgo+IEkgZ3Vlc3Mgb3RoZXIgdGhh biB0aGF0LCB0aGUgYmVzdCB3ZSBjb3VsZCB0cnkgdG8gZG8gaXMgdG8gcmVuYW1lIHNvbWUKPiBj b25maWdzIGFuZC9vciBhZGQgc29tZSBzdWJjb25maWdzIHRvIGRlc2NyaWJlIGNlcnRhaW4gZmVh dHVyZXM/IE1heWJlCj4gSEFWRV9OTUlfV0FUQ0hET0cgPT4gSEFWRV9IQVJETE9DS1VQX0RFVEVD VE9SX0FSQ0hfQkFTSUNfRkVBVFVSRVMKPiB3b3VsZCBoZWxwPyBJJ2QgbG92ZSB0byBjb21lIHVw IHdpdGggYSBiZXR0ZXIgbmFtZSBmb3IKPiBIQVZFX0hBUkRMT0NLVVBfREVURUNUT1JfTk9OX0FS Q0ggYnV0IEkgY291bGRuJ3QgY29tZSB1cCB3aXRoIG9uZS4KPiBNYXliZSB0aGUgdW53aWVsZHkg ICJIQVZFX0hBUkRMT0NLVVBfREVURUNUT1JfVEhBVF9DT1VOVFNfSFJUSU1FUiI/CgpSZW5hbWlu ZyB0aGUgY29uZmlnIHZhcmlhYmxlcyBzZWVtcyB0byBiZSB0aGUgYmVzdCBzb2x1dGlvbiBhdCB0 aGUgbW9tZW50LgpJTUhPLCBpdCB3b3VsZCBiZSBuaWNlIHRvIGhhdmUgc29tZXRoaW5nIGxpa2U6 CgogICsgQ09ORklHX0hBUkRMT0NLVVBfREVURUNUT1IgZm9yIGNvZGUgc2hhcmVkIGJ5IGFsbCBo YXJkbG9ja3VwCiAgICBkZXRlY3RvcnMKCiAgKyBDT05GSUdfSEFSRExPQ0tVUF9ERVRFQ1RPUl9Q RVJGIGZvciBjb2RlIHVzaW5nIGtlcm5lbC93YXRjaGRvZ19wZXJmLmMKCiAgKyBDT05GSUdfSEFS RExPQ0tVUF9ERVRFQ1RPUl9CVUREWSBmb3IgY29kZSB1c2luZwoJICBrZXJuZWwvd2F0Y2hkb2df YnVkZHkuYwoKICArIEhBVkVfSEFSRExPQ0tVUF9ERVRFQ1RPUl9QRVJGIHNldCBieSBhcmNoaXRl Y3R1cmVzIHRoYXQgc3VwcG9ydAoJICB1c2luZyBrZXJuZWwvd2F0Y2hkb2dfcGVyZi5jCgogICsg SEFWRV9IQVJETE9DS1VQX0RFVEVDVE9SX0FSQ0ggc2V0IGJ5IGFyY2hpdGVjdHVyZXMgdGhhdCBo YXZlCglhbHRlcm5hdGl2ZSBpbXBsZW1lbnRhdGlvbiBvZiB0aGUgaGFyZGxvY2t1cCBkZXRlY3Rv cgoKICArIENPTkZJR19IQVJETE9DS1VQX0RFVEVDVE9SX1BSRUZFUl9CVUREWQoJQWxsb3cgdG8g cHJlZmVyIHRoZSBidWRkeSBkZXRlY3RvciB3aGVuIF9QRVJGIG9yIF9BUkNICglpcyBhdmFpbGFi bGUgYXMgd2VsbC4KCiAgKyBIQVZFX0hBUkRMT0NLVVBfUEFOSUMKICArIEhBVkVfSEFSRExPQ0tV UF9BTExfQ1BVX0JBQ0tUUkFDRQoJc2V0IHdoZW4gdGhlIGVhcmNoaXRlY3R1cmUgc3VwcG9ydCB0 aGVzZSBmZWF0dXJlcyBhbmQKCXVzZWQgZm9yIHRoZSBzeXNmcyBpbnRlcmZhY2UKCj4gSWYgeW91 IGhhdmUgY29uY3JldGUgc3VnZ2VzdGlvbnMgZm9yIHdoYXQgd291bGQgYmUgY2xlYW5lciwgbGV0 IG1lCj4ga25vdyBhbmQgSSBjYW4gcXVldWUgdXAgYSBwYXRjaC4gLi4ub3IgSSdtIGhhcHB5IHRv IHJldmlldyBhIHBhdGNoLgoKSSBhbSBub3Qgc3VyZSBob3cgY29tcGxpY2F0ZWQgaXQgd291bGQg YmUgdG8gcmVuYW1lIHRoZSBjb25maWcKdmFyaWFibGVzIHRvIHNvbWVoaW5nIHNhbmUuIEkgYW0g c29ycnkgSSBkbyBub3QgaGF2ZSB0aW1lIHRvCnByZXBhcmUgdGhlIHBhdGNoZXMgYXQgdGhlIG1v bWVudC4KCkknbGwgbGV0IEFuZHJldyB0byBkZWNpZGUgaWYgaGUgd291bGQgcmVxdWlyZSB0aGlz IGNsZWFudXAgdG8gYWNjZXB0CnRoZSBwYXRjaHNldC4KCkJlc3QgUmVnYXJkcywKUGV0cgoKX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KbGludXgtYXJtLWtl cm5lbCBtYWlsaW5nIGxpc3QKbGludXgtYXJtLWtlcm5lbEBsaXN0cy5pbmZyYWRlYWQub3JnCmh0 dHA6Ly9saXN0cy5pbmZyYWRlYWQub3JnL21haWxtYW4vbGlzdGluZm8vbGludXgtYXJtLWtlcm5l bAo=