From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.5 required=3.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,HTML_MESSAGE,INCLUDES_PATCH,MAILING_LIST_MULTI, MIME_HTML_MOSTLY,NICE_REPLY_A,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS, USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 21EF7C433E2 for ; Fri, 28 Aug 2020 06:56:29 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E763920825 for ; Fri, 28 Aug 2020 06:56:28 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="XfzPZeHV" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E763920825 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=amd-gfx-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id A27706E13C; Fri, 28 Aug 2020 06:56:28 +0000 (UTC) Received: from mail-qk1-x744.google.com (mail-qk1-x744.google.com [IPv6:2607:f8b0:4864:20::744]) by gabe.freedesktop.org (Postfix) with ESMTPS id AAA416E13C for ; Fri, 28 Aug 2020 06:56:26 +0000 (UTC) Received: by mail-qk1-x744.google.com with SMTP id w186so133905qkd.1 for ; Thu, 27 Aug 2020 23:56:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=reply-to:subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-language; bh=LqNvP5thK4PK0J0vLFSaXD8y+7e6bP9uczU1Ou/r0C8=; b=XfzPZeHVu2ErlS949rIPtgd0huwItZYhw4kAzQ2p94XM/uGjRFelxtJAjFqK6RaLT8 RW8KG4tOztKYIKDE5FA6k9KyOVuBcUYS2+2uTlnu9D+oCx1NrdAgRhRx2TmrtIqUNyFB +iPj0PHyrA+sbfuq/MokcrLkCpMOFcTbqONpDPamkFxCg+RnlorCnpJgKue3JztWyNbm SAkmxSumBcUIcNK/MsLsPa12pZt02PSZJQmUKRlxBfNK3qEHlIAk/oYDNsuwsbWJWCfV 3+xqC+LEy+WpiEuiAIplcGyGpxkzEtaHpf9+dF5eXn95Y8f5mit6adGNainOUtvdAFh0 0hYg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:reply-to:subject:to:cc:references:from :message-id:date:user-agent:mime-version:in-reply-to :content-language; bh=LqNvP5thK4PK0J0vLFSaXD8y+7e6bP9uczU1Ou/r0C8=; b=BDDLN2zxEtp3UueUqJkofRMvOvVBFfrjzRGKTIov3yjsVqnAUxxMZKnKyc+rA6lVsV zfUiWJmQsSQiHJLvMBSB8wyqbX6+NvmV6evzxPKg7pbALpE1/7n20rmf9rhEGqSEUGCk RoulY3ajsXzJ6yNpis/GegpAGAvFdrZNVqV1lM1Icn95hX2VCdGdjEcV2WpZg1/TlDT/ BbDqg5/WvMOru0VmPexVX8pph13wabd0uGBThupvCY7WYHvj6uKFXrdcAwtDmbnaujFv DZtNKs3Xu6sIZB+JfF79snCGjm7RuhAHG7ZjP/ZHUarhJo5FYkeryvg6SIsyBHk3duMy kTkw== X-Gm-Message-State: AOAM5310lEEoGrALVNXh7kIPP6LXAmJArSrwQH2Y+Ibaj4N2yb2RAAgw OKykXpdak3Nyt6b33UBZkS/aQcuMVik= X-Google-Smtp-Source: ABdhPJzV+XKtFlSq4b8jf1KpZXxCz61QsnPAVP1JYIoCh3ITbpAXqgAFREw9KrdN/bzT7EoOJpwhWw== X-Received: by 2002:a05:620a:21d9:: with SMTP id h25mr7461619qka.444.1598597785483; Thu, 27 Aug 2020 23:56:25 -0700 (PDT) Received: from ?IPv6:2a02:908:1252:fb60:be8a:bd56:1f94:86e7? ([2a02:908:1252:fb60:be8a:bd56:1f94:86e7]) by smtp.gmail.com with ESMTPSA id e63sm56319qkf.29.2020.08.27.23.56.23 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 27 Aug 2020 23:56:24 -0700 (PDT) Subject: Re: [PATCH 5/7] drm/amdgpu: Fix consecutive DPC recoveries failure. To: "Grodzovsky, Andrey" , Alex Deucher References: <1598453182-6946-1-git-send-email-andrey.grodzovsky@amd.com> <1598453182-6946-6-git-send-email-andrey.grodzovsky@amd.com> <0c2ada8e-5dde-0165-c8b3-cf6d6d3fba46@amd.com> From: =?UTF-8?Q?Christian_K=c3=b6nig?= Message-ID: Date: Fri, 28 Aug 2020 08:56:21 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0 MIME-Version: 1.0 In-Reply-To: Content-Language: en-US X-BeenThere: amd-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion list for AMD gfx List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: christian.koenig@amd.com Cc: "Deucher, Alexander" , "Das, Nirmoy" , amd-gfx list Content-Type: multipart/mixed; boundary="===============1662151734==" Errors-To: amd-gfx-bounces@lists.freedesktop.org Sender: "amd-gfx" This is a multi-part message in MIME format. --===============1662151734== Content-Type: multipart/alternative; boundary="------------0C563480B3FB1C8A5C1B2D30" Content-Language: en-US This is a multi-part message in MIME format. --------------0C563480B3FB1C8A5C1B2D30 Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 8bit Am 28.08.20 um 02:00 schrieb Grodzovsky, Andrey: > Ping > > Andrey > ------------------------------------------------------------------------ > *From:* amd-gfx on behalf of > Andrey Grodzovsky > *Sent:* 27 August 2020 10:54 > *To:* Alex Deucher > *Cc:* Deucher, Alexander ; Das, Nirmoy > ; amd-gfx list > *Subject:* Re: [PATCH 5/7] drm/amdgpu: Fix consecutive DPC recoveries > failure. > > On 8/26/20 11:20 AM, Alex Deucher wrote: > > On Wed, Aug 26, 2020 at 10:46 AM Andrey Grodzovsky > > wrote: > >> DPC recovery after prev. DPC recovery or after prev. MODE1 reset fails > >> unles you save the cashe the saved PCI confspace to load it after > >> each new reset. > >> Also use same cached state for other use case of restoring PCI > confspace > >> such as GPU mode1 or VGA switheroo. > >> > > We don't want to keep the saved state around in the pci core > > otherwise, the pci core will assume we are managing the saved state > > for suspend and resume.  I think we want logic like this: > > > > At driver load time: > > pci_save_state(pdev); > > adev->pci_state = pci_store_saved_state(pdev); > > pci_restore_state(adev->pdev); > > > > then in the case of dpc, do: > > pci_load_saved_state(pdev, adev->pci_state); > > > > For all the other cases, just leave the code as is. > > > Actually, as we already discussed - caching the PCI confspace only > once on boot > and not doing it again after each subsequent > controlled or spontaneous reset runs the risk of loading back outdated > confspace > settings. I am not sure if and when but, is it indeed > possible we make changes to PCI confspace registers during runtime and > so the > cached state from boot might be outdated > to load back ? Well we change the confspace settings for quite a bunch of things. For example during BAR resize or link speed adjustment. Christian. > > Andrey > > > > > > Alex > > > > > >> Signed-off-by: Andrey Grodzovsky > >> --- > >>   drivers/gpu/drm/amd/amdgpu/amdgpu.h        |  6 +++ > >>   drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 60 > +++++++++++++++++++++++++++--- > >>   drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c    |  4 +- > >>   drivers/gpu/drm/amd/amdgpu/nv.c            |  4 +- > >>   drivers/gpu/drm/amd/amdgpu/soc15.c         |  4 +- > >>   5 files changed, 66 insertions(+), 12 deletions(-) > >> > >> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h > b/drivers/gpu/drm/amd/amdgpu/amdgpu.h > >> index 3489622..42ee208 100644 > >> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h > >> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h > >> @@ -992,7 +992,9 @@ struct amdgpu_device { > >>          atomic_t throttling_logging_enabled; > >>          struct ratelimit_state throttling_logging_rs; > >>          uint32_t ras_features; > >> + > > Unrelated whitespace changes. > > > >>          bool                            in_dpc; > >> +       struct pci_saved_state *pci_state; > >>   }; > >> > >>   static inline struct amdgpu_device *drm_to_adev(struct drm_device > *ddev) > >> @@ -1272,6 +1274,10 @@ pci_ers_result_t > amdgpu_pci_mmio_enabled(struct pci_dev *pdev); > >>   pci_ers_result_t amdgpu_pci_slot_reset(struct pci_dev *pdev); > >>   void amdgpu_pci_resume(struct pci_dev *pdev); > >> > >> +bool amdgpu_device_cache_pci_state(struct pci_dev *pdev); > >> +bool amdgpu_device_load_pci_state(struct pci_dev *pdev); > >> + > >> + > >> > >>   #include "amdgpu_object.h" > >> > >> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c > b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c > >> index d9e3994..2c088df 100644 > >> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c > >> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c > >> @@ -1283,7 +1283,7 @@ static void > amdgpu_switcheroo_set_state(struct pci_dev *pdev, > >>                  dev->switch_power_state = DRM_SWITCH_POWER_CHANGING; > >> > >> pci_set_power_state(dev->pdev, PCI_D0); > >> -               pci_restore_state(dev->pdev); > >> + amdgpu_device_load_pci_state(dev->pdev); > >>                  r = pci_enable_device(dev->pdev); > >>                  if (r) > >> DRM_WARN("pci_enable_device failed (%d)\n", r); > >> @@ -1296,7 +1296,7 @@ static void > amdgpu_switcheroo_set_state(struct pci_dev *pdev, > >> drm_kms_helper_poll_disable(dev); > >>                  dev->switch_power_state = DRM_SWITCH_POWER_CHANGING; > >>                  amdgpu_device_suspend(dev, true); > >> -               pci_save_state(dev->pdev); > >> + amdgpu_device_cache_pci_state(dev->pdev); > >>                  /* Shut down the device */ > >> pci_disable_device(dev->pdev); > >> pci_set_power_state(dev->pdev, PCI_D3cold); > >> @@ -3401,8 +3401,9 @@ int amdgpu_device_init(struct amdgpu_device > *adev, > >>          if (r) > >>                  dev_err(adev->dev, "amdgpu_pmu_init failed\n"); > >> > >> -       if (pci_save_state(pdev)) > >> -               DRM_ERROR("Failed to save PCI state!!\n"); > >> +       /* Have stored pci confspace at hand for restore in sudden > PCI error */ > >> +       if (!amdgpu_device_cache_pci_state(adev->pdev)) > >> +               DRM_WARN("Failed to cache PCI state!"); > >> > >>          return 0; > >> > >> @@ -3430,6 +3431,8 @@ void amdgpu_device_fini(struct amdgpu_device > *adev) > >> flush_delayed_work(&adev->delayed_init_work); > >>          adev->shutdown = true; > >> > >> +       kfree(adev->pci_state); > >> + > >>          /* make sure IB test finished before entering exclusive mode > >>           * to avoid preemption on IB test > >>           * */ > >> @@ -4855,7 +4858,7 @@ pci_ers_result_t amdgpu_pci_slot_reset(struct > pci_dev *pdev) > >>          /* wait for asic to come out of reset */ > >>          msleep(500); > >> > >> -       pci_restore_state(pdev); > >> +       amdgpu_device_load_pci_state(pdev); > >> > >>          /* confirm  ASIC came out of reset */ > >>          for (i = 0; i < adev->usec_timeout; i++) { > >> @@ -4934,8 +4937,10 @@ pci_ers_result_t > amdgpu_pci_slot_reset(struct pci_dev *pdev) > >> > >>   out: > >> > >> -       if (!r) > >> +       if (!r) { > >> + amdgpu_device_cache_pci_state(adev->pdev); > >>                  DRM_INFO("PCIe error recovery succeeded\n"); > >> +       } > >>          else { > >>                  DRM_ERROR("PCIe error recovery failed, err:%d", r); > >>                  amdgpu_device_unlock_adev(adev); > >> @@ -4974,3 +4979,46 @@ void amdgpu_pci_resume(struct pci_dev *pdev) > >> > >>          amdgpu_device_unlock_adev(adev); > >>   } > >> + > >> +bool amdgpu_device_cache_pci_state(struct pci_dev *pdev) > >> +{ > >> +       struct drm_device *dev = pci_get_drvdata(pdev); > >> +       struct amdgpu_device *adev = drm_to_adev(dev); > >> +       int r; > >> + > >> +       r = pci_save_state(pdev); > >> +       if (!r) { > >> +               kfree(adev->pci_state); > >> + > >> +               adev->pci_state = pci_store_saved_state(pdev); > >> + > >> +               if (!adev->pci_state) { > >> +                       DRM_ERROR("Failed to store PCI saved state"); > >> +                       return false; > >> +               } > >> +       } else { > >> +               DRM_WARN("Failed to save PCI state, err:%d\n", r); > >> +               return false; > >> +       } > >> + > >> +       return true; > >> +} > >> + > >> +bool amdgpu_device_load_pci_state(struct pci_dev *pdev) > >> +{ > >> +       struct drm_device *dev = pci_get_drvdata(pdev); > >> +       struct amdgpu_device *adev = drm_to_adev(dev); > >> +       int r; > >> + > >> +       if (!adev->pci_state) > >> +               return false; > >> + > >> +       r = pci_load_saved_state(pdev, adev->pci_state); > >> + > >> +       if (!r) { > >> +               pci_restore_state(pdev); > >> +       } else { > >> +               DRM_WARN("Failed to load PCI state, err:%d\n", r); > >> +               return false; > >> +       } > >> +} > >> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c > b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c > >> index 4bbcc70..7a6482a 100644 > >> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c > >> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c > >> @@ -1320,7 +1320,7 @@ static int > amdgpu_pmops_runtime_suspend(struct device *dev) > >>                  if (amdgpu_is_atpx_hybrid()) { > >> pci_ignore_hotplug(pdev); > >>                  } else { > >> -                       pci_save_state(pdev); > >> + amdgpu_device_cache_pci_state(pdev); > >> pci_disable_device(pdev); > >> pci_ignore_hotplug(pdev); > >> pci_set_power_state(pdev, PCI_D3cold); > >> @@ -1353,7 +1353,7 @@ static int amdgpu_pmops_runtime_resume(struct > device *dev) > >>                          pci_set_master(pdev); > >>                  } else { > >> pci_set_power_state(pdev, PCI_D0); > >> -                       pci_restore_state(pdev); > >> + amdgpu_device_load_pci_state(pdev); > >>                          ret = pci_enable_device(pdev); > >>                          if (ret) > >>                                  return ret; > >> diff --git a/drivers/gpu/drm/amd/amdgpu/nv.c > b/drivers/gpu/drm/amd/amdgpu/nv.c > >> index 4d14023..0ec6603 100644 > >> --- a/drivers/gpu/drm/amd/amdgpu/nv.c > >> +++ b/drivers/gpu/drm/amd/amdgpu/nv.c > >> @@ -311,7 +311,7 @@ static int nv_asic_mode1_reset(struct > amdgpu_device *adev) > >>          /* disable BM */ > >>          pci_clear_master(adev->pdev); > >> > >> -       pci_save_state(adev->pdev); > >> + amdgpu_device_cache_pci_state(adev->pdev); > >> > >>          if (amdgpu_dpm_is_mode1_reset_supported(adev)) { > >>                  dev_info(adev->dev, "GPU smu mode1 reset\n"); > >> @@ -323,7 +323,7 @@ static int nv_asic_mode1_reset(struct > amdgpu_device *adev) > >> > >>          if (ret) > >>                  dev_err(adev->dev, "GPU mode1 reset failed\n"); > >> -       pci_restore_state(adev->pdev); > >> + amdgpu_device_load_pci_state(adev->pdev); > >> > >>          /* wait for asic to come out of reset */ > >>          for (i = 0; i < adev->usec_timeout; i++) { > >> diff --git a/drivers/gpu/drm/amd/amdgpu/soc15.c > b/drivers/gpu/drm/amd/amdgpu/soc15.c > >> index 2f93c47..ddd55e3 100644 > >> --- a/drivers/gpu/drm/amd/amdgpu/soc15.c > >> +++ b/drivers/gpu/drm/amd/amdgpu/soc15.c > >> @@ -484,13 +484,13 @@ static int soc15_asic_mode1_reset(struct > amdgpu_device *adev) > >>          /* disable BM */ > >>          pci_clear_master(adev->pdev); > >> > >> -       pci_save_state(adev->pdev); > >> + amdgpu_device_cache_pci_state(adev->pdev); > >> > >>          ret = psp_gpu_reset(adev); > >>          if (ret) > >>                  dev_err(adev->dev, "GPU mode1 reset failed\n"); > >> > >> -       pci_restore_state(adev->pdev); > >> + amdgpu_device_load_pci_state(adev->pdev); > >> > >>          /* wait for asic to come out of reset */ > >>          for (i = 0; i < adev->usec_timeout; i++) { > >> -- > >> 2.7.4 > >> > >> _______________________________________________ > >> amd-gfx mailing list > >> amd-gfx@lists.freedesktop.org > >> > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Fmailman%2Flistinfo%2Famd-gfx&data=02%7C01%7Candrey.grodzovsky%40amd.com%7Cf7e5e09330074d176b5b08d84a99236e%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637341368926287137&sdata=HTTP5xt%2BFTc%2F%2BlcI37wkRQ0BqTHOXOQVpSlR%2FDEnFlM%3D&reserved=0 > _______________________________________________ > amd-gfx mailing list > amd-gfx@lists.freedesktop.org > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Fmailman%2Flistinfo%2Famd-gfx&data=02%7C01%7Candrey.grodzovsky%40amd.com%7Cf7e5e09330074d176b5b08d84a99236e%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637341368926287137&sdata=HTTP5xt%2BFTc%2F%2BlcI37wkRQ0BqTHOXOQVpSlR%2FDEnFlM%3D&reserved=0 > > _______________________________________________ > amd-gfx mailing list > amd-gfx@lists.freedesktop.org > https://lists.freedesktop.org/mailman/listinfo/amd-gfx --------------0C563480B3FB1C8A5C1B2D30 Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: 8bit
Am 28.08.20 um 02:00 schrieb Grodzovsky, Andrey:
Ping

Andrey
From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org> on behalf of Andrey Grodzovsky <Andrey.Grodzovsky@amd.com>
Sent: 27 August 2020 10:54
To: Alex Deucher <alexdeucher@gmail.com>
Cc: Deucher, Alexander <Alexander.Deucher@amd.com>; Das, Nirmoy <Nirmoy.Das@amd.com>; amd-gfx list <amd-gfx@lists.freedesktop.org>
Subject: Re: [PATCH 5/7] drm/amdgpu: Fix consecutive DPC recoveries failure.
 

On 8/26/20 11:20 AM, Alex Deucher wrote:
> On Wed, Aug 26, 2020 at 10:46 AM Andrey Grodzovsky
> <andrey.grodzovsky@amd.com> wrote:
>> DPC recovery after prev. DPC recovery or after prev. MODE1 reset fails
>> unles you save the cashe the saved PCI confspace to load it after
>> each new reset.
>> Also use same cached state for other use case of restoring PCI confspace
>> such as GPU mode1 or VGA switheroo.
>>
> We don't want to keep the saved state around in the pci core
> otherwise, the pci core will assume we are managing the saved state
> for suspend and resume.  I think we want logic like this:
>
> At driver load time:
> pci_save_state(pdev);
> adev->pci_state = pci_store_saved_state(pdev);
> pci_restore_state(adev->pdev);
>
> then in the case of dpc, do:
> pci_load_saved_state(pdev, adev->pci_state);
>
> For all the other cases, just leave the code as is.


Actually, as we already discussed - caching the PCI confspace only once on boot
and not doing it again after each subsequent
controlled or spontaneous reset runs the risk of loading back outdated confspace
settings. I am not sure if and when but, is it indeed
possible we make changes to PCI confspace registers during runtime and so the
cached state from boot might be outdated
to load back ?

Well we change the confspace settings for quite a bunch of things. For example during BAR resize or link speed adjustment.

Christian.


Andrey


>
> Alex
>
>
>> Signed-off-by: Andrey Grodzovsky <andrey.grodzovsky@amd.com>
>> ---
>>   drivers/gpu/drm/amd/amdgpu/amdgpu.h        |  6 +++
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 60 +++++++++++++++++++++++++++---
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c    |  4 +-
>>   drivers/gpu/drm/amd/amdgpu/nv.c            |  4 +-
>>   drivers/gpu/drm/amd/amdgpu/soc15.c         |  4 +-
>>   5 files changed, 66 insertions(+), 12 deletions(-)
>>
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
>> index 3489622..42ee208 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
>> @@ -992,7 +992,9 @@ struct amdgpu_device {
>>          atomic_t                        throttling_logging_enabled;
>>          struct ratelimit_state          throttling_logging_rs;
>>          uint32_t                        ras_features;
>> +
> Unrelated whitespace changes.
>
>>          bool                            in_dpc;
>> +       struct pci_saved_state          *pci_state;
>>   };
>>
>>   static inline struct amdgpu_device *drm_to_adev(struct drm_device *ddev)
>> @@ -1272,6 +1274,10 @@ pci_ers_result_t amdgpu_pci_mmio_enabled(struct pci_dev *pdev);
>>   pci_ers_result_t amdgpu_pci_slot_reset(struct pci_dev *pdev);
>>   void amdgpu_pci_resume(struct pci_dev *pdev);
>>
>> +bool amdgpu_device_cache_pci_state(struct pci_dev *pdev);
>> +bool amdgpu_device_load_pci_state(struct pci_dev *pdev);
>> +
>> +
>>
>>   #include "amdgpu_object.h"
>>
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
>> index d9e3994..2c088df 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
>> @@ -1283,7 +1283,7 @@ static void amdgpu_switcheroo_set_state(struct pci_dev *pdev,
>>                  dev->switch_power_state = DRM_SWITCH_POWER_CHANGING;
>>
>>                  pci_set_power_state(dev->pdev, PCI_D0);
>> -               pci_restore_state(dev->pdev);
>> +               amdgpu_device_load_pci_state(dev->pdev);
>>                  r = pci_enable_device(dev->pdev);
>>                  if (r)
>>                          DRM_WARN("pci_enable_device failed (%d)\n", r);
>> @@ -1296,7 +1296,7 @@ static void amdgpu_switcheroo_set_state(struct pci_dev *pdev,
>>                  drm_kms_helper_poll_disable(dev);
>>                  dev->switch_power_state = DRM_SWITCH_POWER_CHANGING;
>>                  amdgpu_device_suspend(dev, true);
>> -               pci_save_state(dev->pdev);
>> +               amdgpu_device_cache_pci_state(dev->pdev);
>>                  /* Shut down the device */
>>                  pci_disable_device(dev->pdev);
>>                  pci_set_power_state(dev->pdev, PCI_D3cold);
>> @@ -3401,8 +3401,9 @@ int amdgpu_device_init(struct amdgpu_device *adev,
>>          if (r)
>>                  dev_err(adev->dev, "amdgpu_pmu_init failed\n");
>>
>> -       if (pci_save_state(pdev))
>> -               DRM_ERROR("Failed to save PCI state!!\n");
>> +       /* Have stored pci confspace at hand for restore in sudden PCI error */
>> +       if (!amdgpu_device_cache_pci_state(adev->pdev))
>> +               DRM_WARN("Failed to cache PCI state!");
>>
>>          return 0;
>>
>> @@ -3430,6 +3431,8 @@ void amdgpu_device_fini(struct amdgpu_device *adev)
>>          flush_delayed_work(&adev->delayed_init_work);
>>          adev->shutdown = true;
>>
>> +       kfree(adev->pci_state);
>> +
>>          /* make sure IB test finished before entering exclusive mode
>>           * to avoid preemption on IB test
>>           * */
>> @@ -4855,7 +4858,7 @@ pci_ers_result_t amdgpu_pci_slot_reset(struct pci_dev *pdev)
>>          /* wait for asic to come out of reset */
>>          msleep(500);
>>
>> -       pci_restore_state(pdev);
>> +       amdgpu_device_load_pci_state(pdev);
>>
>>          /* confirm  ASIC came out of reset */
>>          for (i = 0; i < adev->usec_timeout; i++) {
>> @@ -4934,8 +4937,10 @@ pci_ers_result_t amdgpu_pci_slot_reset(struct pci_dev *pdev)
>>
>>   out:
>>
>> -       if (!r)
>> +       if (!r) {
>> +               amdgpu_device_cache_pci_state(adev->pdev);
>>                  DRM_INFO("PCIe error recovery succeeded\n");
>> +       }
>>          else {
>>                  DRM_ERROR("PCIe error recovery failed, err:%d", r);
>>                  amdgpu_device_unlock_adev(adev);
>> @@ -4974,3 +4979,46 @@ void amdgpu_pci_resume(struct pci_dev *pdev)
>>
>>          amdgpu_device_unlock_adev(adev);
>>   }
>> +
>> +bool amdgpu_device_cache_pci_state(struct pci_dev *pdev)
>> +{
>> +       struct drm_device *dev = pci_get_drvdata(pdev);
>> +       struct amdgpu_device *adev = drm_to_adev(dev);
>> +       int r;
>> +
>> +       r = pci_save_state(pdev);
>> +       if (!r) {
>> +               kfree(adev->pci_state);
>> +
>> +               adev->pci_state = pci_store_saved_state(pdev);
>> +
>> +               if (!adev->pci_state) {
>> +                       DRM_ERROR("Failed to store PCI saved state");
>> +                       return false;
>> +               }
>> +       } else {
>> +               DRM_WARN("Failed to save PCI state, err:%d\n", r);
>> +               return false;
>> +       }
>> +
>> +       return true;
>> +}
>> +
>> +bool amdgpu_device_load_pci_state(struct pci_dev *pdev)
>> +{
>> +       struct drm_device *dev = pci_get_drvdata(pdev);
>> +       struct amdgpu_device *adev = drm_to_adev(dev);
>> +       int r;
>> +
>> +       if (!adev->pci_state)
>> +               return false;
>> +
>> +       r = pci_load_saved_state(pdev, adev->pci_state);
>> +
>> +       if (!r) {
>> +               pci_restore_state(pdev);
>> +       } else {
>> +               DRM_WARN("Failed to load PCI state, err:%d\n", r);
>> +               return false;
>> +       }
>> +}
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
>> index 4bbcc70..7a6482a 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
>> @@ -1320,7 +1320,7 @@ static int amdgpu_pmops_runtime_suspend(struct device *dev)
>>                  if (amdgpu_is_atpx_hybrid()) {
>>                          pci_ignore_hotplug(pdev);
>>                  } else {
>> -                       pci_save_state(pdev);
>> +                       amdgpu_device_cache_pci_state(pdev);
>>                          pci_disable_device(pdev);
>>                          pci_ignore_hotplug(pdev);
>>                          pci_set_power_state(pdev, PCI_D3cold);
>> @@ -1353,7 +1353,7 @@ static int amdgpu_pmops_runtime_resume(struct device *dev)
>>                          pci_set_master(pdev);
>>                  } else {
>>                          pci_set_power_state(pdev, PCI_D0);
>> -                       pci_restore_state(pdev);
>> +                       amdgpu_device_load_pci_state(pdev);
>>                          ret = pci_enable_device(pdev);
>>                          if (ret)
>>                                  return ret;
>> diff --git a/drivers/gpu/drm/amd/amdgpu/nv.c b/drivers/gpu/drm/amd/amdgpu/nv.c
>> index 4d14023..0ec6603 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/nv.c
>> +++ b/drivers/gpu/drm/amd/amdgpu/nv.c
>> @@ -311,7 +311,7 @@ static int nv_asic_mode1_reset(struct amdgpu_device *adev)
>>          /* disable BM */
>>          pci_clear_master(adev->pdev);
>>
>> -       pci_save_state(adev->pdev);
>> +       amdgpu_device_cache_pci_state(adev->pdev);
>>
>>          if (amdgpu_dpm_is_mode1_reset_supported(adev)) {
>>                  dev_info(adev->dev, "GPU smu mode1 reset\n");
>> @@ -323,7 +323,7 @@ static int nv_asic_mode1_reset(struct amdgpu_device *adev)
>>
>>          if (ret)
>>                  dev_err(adev->dev, "GPU mode1 reset failed\n");
>> -       pci_restore_state(adev->pdev);
>> +       amdgpu_device_load_pci_state(adev->pdev);
>>
>>          /* wait for asic to come out of reset */
>>          for (i = 0; i < adev->usec_timeout; i++) {
>> diff --git a/drivers/gpu/drm/amd/amdgpu/soc15.c b/drivers/gpu/drm/amd/amdgpu/soc15.c
>> index 2f93c47..ddd55e3 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/soc15.c
>> +++ b/drivers/gpu/drm/amd/amdgpu/soc15.c
>> @@ -484,13 +484,13 @@ static int soc15_asic_mode1_reset(struct amdgpu_device *adev)
>>          /* disable BM */
>>          pci_clear_master(adev->pdev);
>>
>> -       pci_save_state(adev->pdev);
>> +       amdgpu_device_cache_pci_state(adev->pdev);
>>
>>          ret = psp_gpu_reset(adev);
>>          if (ret)
>>                  dev_err(adev->dev, "GPU mode1 reset failed\n");
>>
>> -       pci_restore_state(adev->pdev);
>> +       amdgpu_device_load_pci_state(adev->pdev);
>>
>>          /* wait for asic to come out of reset */
>>          for (i = 0; i < adev->usec_timeout; i++) {
>> --
>> 2.7.4
>>
>> _______________________________________________
>> amd-gfx mailing list
>> amd-gfx@lists.freedesktop.org
>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Fmailman%2Flistinfo%2Famd-gfx&amp;data=02%7C01%7Candrey.grodzovsky%40amd.com%7Cf7e5e09330074d176b5b08d84a99236e%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637341368926287137&amp;sdata=HTTP5xt%2BFTc%2F%2BlcI37wkRQ0BqTHOXOQVpSlR%2FDEnFlM%3D&amp;reserved=0
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Fmailman%2Flistinfo%2Famd-gfx&amp;data=02%7C01%7Candrey.grodzovsky%40amd.com%7Cf7e5e09330074d176b5b08d84a99236e%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637341368926287137&amp;sdata=HTTP5xt%2BFTc%2F%2BlcI37wkRQ0BqTHOXOQVpSlR%2FDEnFlM%3D&amp;reserved=0

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

--------------0C563480B3FB1C8A5C1B2D30-- --===============1662151734== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/amd-gfx --===============1662151734==--