From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga09.intel.com ([134.134.136.24]:46837 "EHLO mga09.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751147AbdFBFvK (ORCPT ); Fri, 2 Jun 2017 01:51:10 -0400 Date: Fri, 2 Jun 2017 13:51:38 +0800 From: Chen Yu To: "Rafael J. Wysocki" Cc: Bjorn Helgaas , "Rafael J . Wysocki" , Len Brown , Dan Williams , Rui Zhang , Ying Huang , Linux PCI , Linux PM , Linux Kernel Mailing List Subject: Re: [PATCH] PCI / PM: Restore the status of PCI devices across hibernation Message-ID: <20170602055138.GA32161@yu-desktop-1.sh.intel.com> References: <1495702147-31427-1-git-send-email-yu.c.chen@intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii In-Reply-To: Sender: linux-pci-owner@vger.kernel.org List-ID: On Fri, Jun 02, 2017 at 01:22:28AM +0200, Rafael J. Wysocki wrote: > On Thu, May 25, 2017 at 10:49 AM, Chen Yu wrote: > > Currently we saw a lot of "No irq handler" errors during hibernation, > > which caused the system hang finally: > > > > [ 710.141581] ata4.00: qc timeout (cmd 0xec) > > [ 710.147135] ata4.00: failed to IDENTIFY (I/O error, err_mask=0x4) > > [ 710.154593] ata4.00: revalidation failed (errno=-5) > > [ 710.468124] ata4: SATA link up 6.0 Gbps (SStatus 133 SControl 300) > > [ 710.477746] do_IRQ: 31.151 No irq handler for vector > > > > According to above logs, there is an interrupt triggered and it is > > dispatched to CPU31 with a vector number 151, but there is no handler > > for it, thus this irq will not get acked and caused irq flood which kill > > the system. To be more specific, the 31.151 is an interrupt from the ahci > > host controller. > > > > After some investigation, the reason why this issue is triggered is > > because the thaw_noirq() function does not restore the MSI/MSIX settings > > across hibernation. > > > > The scenario is illustrated below: > > > > 1. Before the hibernation starts, the irq 34 is the handler for the ahci device, > > which is binded on cpu31. > > 2. Hibernation starts, the ahci device is put into low power state. > > 3. All the nonboot CPUs are put offline, so the irq 34 has to be migrated to > > the last alive one - CPU0. > > 4. After the snapshot has been created, all the nonboot CPUs are brought up again, > > the CPU affinity for IRQ 34 remains to be 0. > > 5. ahci device are put into D0. > > 6. The snapshot is written to the disk. > > > > The issue is triggered in step 6, in theory the ahci interrupt should be > > delivered to CPU0, however the actually result is that this interrupt is > > delivered to the original CPU31 instead, which cause the "No irq handler" issue. > > > > Ying Huang has has provided a clue that, in step 3 it is possible that the writing > > to the register might not take effect as the PCI devices have been put suspended. > > Actually it is true: > > In step 3, the irq 34 affinity is supposed to be modified from 31 to 0, > > but actually it did not. In __pci_write_msi_msg(), if the device is already > > in low power state, the low level msi message entry will not be updated > > but cached. So in theory during the device restore process, the cached msi > > modification information should be written back to the hardware, and this > > is what pci_restore_msi_state() do during normal suspend-resume. > > But this is not the case for hibernation, pci_restore_msi_state() is not > > invoked currently, to be more specific, pci_restore_state() is not invoked > > in pci_pm_thaw_noirq(), although pci_save_state() has saved the necessary > > pci cached information in pci_pm_freeze_noirq(). > > > > This patch tries to restore the pci status for the device during hibernation, > > otherwise the status might be lost across hibernation(for example, the MSI/MSIX > > message settings), which might cause problems during hibernation. > > > > Suggested-by: Ying Huang > > Suggested-by: Rafael J. Wysocki > > Cc: Rafael J. Wysocki > > Cc: Bjorn Helgaas > > Cc: Len Brown > > Cc: Dan Williams > > Cc: Rui Zhang > > Cc: Ying Huang > > Cc: linux-pci@vger.kernel.org > > Cc: linux-pm@vger.kernel.org > > Cc: linux-kernel@vger.kernel.org > > Signed-off-by: Chen Yu > > Thanks for the detailed description of what's going on! > > The fix is correct IMO, so > > Reviewed-by: Rafael J. Wysocki > Thanks for the review! Please also help add a tag for stable: Cc: stable thanks, Yu