From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,HK_RANDOM_FROM,INCLUDES_PATCH, MAILING_LIST_MULTI,NICE_REPLY_A,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0C801C43457 for ; Tue, 20 Oct 2020 13:05:51 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 9174022475 for ; Tue, 20 Oct 2020 13:05:50 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9174022475 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.intel.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=intel-gfx-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id D4DF36EC81; Tue, 20 Oct 2020 13:05:49 +0000 (UTC) Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by gabe.freedesktop.org (Postfix) with ESMTPS id 6789C6EC81 for ; Tue, 20 Oct 2020 13:05:49 +0000 (UTC) IronPort-SDR: 7pEKKmDRQVdiVOoklbTl5WwyQLRHz2tpjVy0RixHpkS4u/wv+p8SL7Mws5J61ntpuRX4fRAQix 7jdXMqizRA6Q== X-IronPort-AV: E=McAfee;i="6000,8403,9779"; a="146497275" X-IronPort-AV: E=Sophos;i="5.77,397,1596524400"; d="scan'208";a="146497275" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Oct 2020 06:05:47 -0700 IronPort-SDR: Ws44ILj8JoYK9CvYqWU5LDd/AMKQC4q+kxSIWdWNZ9xgO24qzntZzWeHCS7Nc8cHcT33HtvBtA AN/8i2U5xdhQ== X-IronPort-AV: E=Sophos;i="5.77,397,1596524400"; d="scan'208";a="533052524" Received: from bszymanx-mobl.ger.corp.intel.com (HELO [10.252.55.244]) ([10.252.55.244]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Oct 2020 06:05:46 -0700 To: Chris Wilson , Intel-gfx@lists.freedesktop.org References: <20201020100822.543332-1-tvrtko.ursulin@linux.intel.com> <20201020100822.543332-2-tvrtko.ursulin@linux.intel.com> <160319519741.15830.12777651851324275501@build.alporthouse.com> <160319580270.15830.9644634406956362493@build.alporthouse.com> <46422560-ca2e-474a-ab07-1107153f1229@linux.intel.com> <160319763357.15830.6451291888634667832@build.alporthouse.com> From: Tvrtko Ursulin Organization: Intel Corporation UK Plc Message-ID: Date: Tue, 20 Oct 2020 14:05:43 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0 MIME-Version: 1.0 In-Reply-To: <160319763357.15830.6451291888634667832@build.alporthouse.com> Content-Language: en-US Subject: Re: [Intel-gfx] [PATCH 2/2] drm/i915/pmu: Fix CPU hotplug with multiple GPUs X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Daniel Vetter Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii"; Format="flowed" Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" On 20/10/2020 13:40, Chris Wilson wrote: > Quoting Tvrtko Ursulin (2020-10-20 13:33:12) >> >> On 20/10/2020 13:10, Chris Wilson wrote: >>> Quoting Chris Wilson (2020-10-20 12:59:57) >>>> Quoting Tvrtko Ursulin (2020-10-20 11:08:22) >>>>> From: Tvrtko Ursulin >>>>> >>>>> Since we keep a driver global mask of online CPUs and base the decision >>>>> whether PMU needs to be migrated upon it, we need to make sure the >>>>> migration is done for all registered PMUs (so GPUs). >>>>> >>>>> To do this we need to track the current CPU for each PMU and base the >>>>> decision on whether to migrate on a comparison between global and local >>>>> state. >>>>> >>>>> At the same time, since dynamic CPU hotplug notification slots are a >>>>> scarce resource and given how we already register the multi instance type >>>>> state, we can and should add multiple instance of the i915 PMU to this >>>>> same state and not allocate a new one for every GPU. >>>>> >>>>> v2: >>>>> * Use pr_notice. (Chris) >>>>> >>>>> Signed-off-by: Tvrtko Ursulin >>>>> Suggested-by: Daniel Vetter # dynamic slot optimisation >>>>> Cc: Chris Wilson >>>>> --- >>>>> drivers/gpu/drm/i915/i915_pci.c | 7 ++++- >>>>> drivers/gpu/drm/i915/i915_pmu.c | 50 ++++++++++++++++++++------------- >>>>> drivers/gpu/drm/i915/i915_pmu.h | 6 +++- >>>>> 3 files changed, 41 insertions(+), 22 deletions(-) >>>>> >>>>> diff --git a/drivers/gpu/drm/i915/i915_pci.c b/drivers/gpu/drm/i915/i915_pci.c >>>>> index 27964ac0638a..a384f51c91c1 100644 >>>>> --- a/drivers/gpu/drm/i915/i915_pci.c >>>>> +++ b/drivers/gpu/drm/i915/i915_pci.c >>>>> @@ -1150,9 +1150,13 @@ static int __init i915_init(void) >>>>> return 0; >>>>> } >>>>> >>>>> + i915_pmu_init(); >>>>> + >>>>> err = pci_register_driver(&i915_pci_driver); >>>>> - if (err) >>>>> + if (err) { >>>>> + i915_pmu_exit(); >>>>> return err; >>>>> + } >>>>> >>>>> i915_perf_sysctl_register(); >>>>> return 0; >>>>> @@ -1166,6 +1170,7 @@ static void __exit i915_exit(void) >>>>> i915_perf_sysctl_unregister(); >>>>> pci_unregister_driver(&i915_pci_driver); >>>>> i915_globals_exit(); >>>>> + i915_pmu_exit(); >>>>> } >>>>> >>>>> module_init(i915_init); >>>>> diff --git a/drivers/gpu/drm/i915/i915_pmu.c b/drivers/gpu/drm/i915/i915_pmu.c >>>>> index 51ed7d0efcdc..0d6c0945621e 100644 >>>>> --- a/drivers/gpu/drm/i915/i915_pmu.c >>>>> +++ b/drivers/gpu/drm/i915/i915_pmu.c >>>>> @@ -30,6 +30,7 @@ >>>>> #define ENGINE_SAMPLE_BITS (1 << I915_PMU_SAMPLE_BITS) >>>>> >>>>> static cpumask_t i915_pmu_cpumask; >>>>> +static unsigned int i915_pmu_target_cpu = -1; >>>>> >>>>> static u8 engine_config_sample(u64 config) >>>>> { >>>>> @@ -1049,25 +1050,32 @@ static int i915_pmu_cpu_online(unsigned int cpu, struct hlist_node *node) >>>>> static int i915_pmu_cpu_offline(unsigned int cpu, struct hlist_node *node) >>>>> { >>>>> struct i915_pmu *pmu = hlist_entry_safe(node, typeof(*pmu), cpuhp.node); >>>>> - unsigned int target; >>>>> + unsigned int target = i915_pmu_target_cpu; >>>> >>>> So we still have multiple callbacks, one per pmu. But each callback is >>>> now stored in a list from the cpuhp_slot instead of each callback having >>>> its own slot. >>>> >>>>> >>>>> GEM_BUG_ON(!pmu->base.event_init); >>>>> >>>>> if (cpumask_test_and_clear_cpu(cpu, &i915_pmu_cpumask)) { >>>> >>>> On first callback... >>>> >>>>> target = cpumask_any_but(topology_sibling_cpumask(cpu), cpu); >>>> >>>> Pick any other cpu. >>>> >>>>> + >>>>> /* Migrate events if there is a valid target */ >>>>> if (target < nr_cpu_ids) { >>>>> cpumask_set_cpu(target, &i915_pmu_cpumask); >>>>> - perf_pmu_migrate_context(&pmu->base, cpu, target); >>>>> + i915_pmu_target_cpu = target; >>>> >>>> Store target for all callbacks. >>>> >>>>> } >>>>> } >>>>> >>>>> + if (target < nr_cpu_ids && target != pmu->cpuhp.cpu) { >>>> >>>> If global [i915_pmu_target_cpu] target has changed, update perf. >>>> >>>>> + perf_pmu_migrate_context(&pmu->base, cpu, target); >>>>> + pmu->cpuhp.cpu = target; >>>> >>>> It is claimed that cpuhp_state_remove_instance() will call the offline >>>> callback for all online cpus... Do we need a pmu->base.state != STOPPED >>>> guard? >>> >>> s/claimed/it definitely does :)/ >>> >>> Or rather pmu->closed. >> >> Hm why? You think perf_pmu_migrate_context accesses something in the PMU >> outside of the already protected entry points? > > If this callback is being called for every online when we unplug one > device, we then believe that no cpus remain online for all other devices. > Should a cpu then be offlined, target is -1u so greater than > nr_cpu_online and we move the perf context to the void, worst case, in > the best case we fail to migrate the perf context off the dying cpu. Well spotted nasty interaction, thanks. Regards, Tvrtko _______________________________________________ Intel-gfx mailing list Intel-gfx@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/intel-gfx