From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3356AC4332F for ; Wed, 12 Oct 2022 14:10:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:Content-Type: Content-Transfer-Encoding:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:From:References:CC:To:Subject: MIME-Version:Date:Message-ID:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=bNNZfXp5mq03BF5POtQHL6g/VkDMhDDtA9PLRmyaLnw=; b=mAH0wl60KZOKOz qJdb3WH1P5FG97lHeKB6B0DPHV9fpSSzslPtK08usmrjRcJnQgXFBKE+fyUqNfXV+nGDA7pBOSB7g 3TZvxCE5SI+mjV9clYtOfLk9rdG9i4+QoNrBDq54eDPP12ZkZFq1T4yUZ2TfLrIuihP6jjRnc4GBj pgSmWYyxMlaIBQKrsg5BSfhIXUna3saUdQ8jA6elEd4f6IGlgx+CdHXygy3PIZf+9IjBi7sM4TXH0 m8ruyGawHKRvu0tQFEdS83H92Ubnsvu7FTPtsEvXyhxEaiIpS33gFi40vF9PS7E56JPwM5DV16aqA Dzbz9zvCZ/jscpD3jL7A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oicRX-008Kfm-JO; Wed, 12 Oct 2022 14:10:43 +0000 Received: from mx0a-0031df01.pphosted.com ([205.220.168.131]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oicRU-008Ke7-0Q for linux-phy@lists.infradead.org; Wed, 12 Oct 2022 14:10:42 +0000 Received: from pps.filterd (m0279863.ppops.net [127.0.0.1]) by mx0a-0031df01.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 29C9jlge004563; Wed, 12 Oct 2022 14:10:28 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quicinc.com; h=message-id : date : mime-version : subject : to : cc : references : from : in-reply-to : content-type : content-transfer-encoding; s=qcppdkim1; bh=X56T0Y4BdSLVT4DObSFyKor0Immxx7awcwH20gDFW3A=; b=VSIUaIYH9BfAwE+KxU6/vwSmrLRfJRxC18QVIV4+nWqRjJGnuJNZgKAdPXYhbtFsOary KWcrQbPFWw7HzoqxkZheiv08AxpM1g1qb2Ua68yqOVYrK9K3jnc7pCGkoitvyt7fZayC fcOFvIkci55mRsLDbMEK3U0ywlmI7sbV71wz25SM57nzJkHAyy9x1aBJ3Z/d6Yz3k5wm laH67oEVj3JjCDXmSvdJ2mva00oJDMhKGMrrETZTX2u9ckh1rLNDjQsRQ9HRwhttdTDI CEQTXsAU//sULjVkYtZls3H1h5TbxpGdLUKQq99lrcJkItTM5TsDMr7pcmNabSsRA0um /A== Received: from nalasppmta02.qualcomm.com (Global_NAT1.qualcomm.com [129.46.96.20]) by mx0a-0031df01.pphosted.com (PPS) with ESMTPS id 3k5j7b244p-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 12 Oct 2022 14:10:28 +0000 Received: from nalasex01a.na.qualcomm.com (nalasex01a.na.qualcomm.com [10.47.209.196]) by NALASPPMTA02.qualcomm.com (8.17.1.5/8.17.1.5) with ESMTPS id 29CEARlm024769 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 12 Oct 2022 14:10:27 GMT Received: from [10.216.32.191] (10.80.80.8) by nalasex01a.na.qualcomm.com (10.47.209.196) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.29; Wed, 12 Oct 2022 07:10:18 -0700 Message-ID: <04ace1ae-26d0-4157-b7eb-8dff29895180@quicinc.com> Date: Wed, 12 Oct 2022 19:36:52 +0530 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Thunderbird/91.9.1 Subject: Re: [PATCH v7 1/5] PCI: qcom: Add system suspend and resume support Content-Language: en-US To: Bjorn Helgaas CC: , , , , , , , , , , , , , , , , , , , , , , , , , Bjorn Andersson , "Rafael J. Wysocki" , References: <20221005211323.GA2390992@bhelgaas> From: Krishna Chaitanya Chundru In-Reply-To: <20221005211323.GA2390992@bhelgaas> X-Originating-IP: [10.80.80.8] X-ClientProxiedBy: nasanex01a.na.qualcomm.com (10.52.223.231) To nalasex01a.na.qualcomm.com (10.47.209.196) X-QCInternal: smtphost X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=5800 signatures=585085 X-Proofpoint-GUID: 6dfLb38IxSjGoasAJ9pMdvLlzZF_6S-6 X-Proofpoint-ORIG-GUID: 6dfLb38IxSjGoasAJ9pMdvLlzZF_6S-6 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-10-12_06,2022-10-12_01,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxscore=0 phishscore=0 spamscore=0 suspectscore=0 clxscore=1015 impostorscore=0 adultscore=0 mlxlogscore=951 malwarescore=0 priorityscore=1501 bulkscore=0 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2209130000 definitions=main-2210120093 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221012_071040_089869_9224BF76 X-CRM114-Status: GOOD ( 43.54 ) X-BeenThere: linux-phy@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Linux Phy Mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii"; Format="flowed" Sender: "linux-phy" Errors-To: linux-phy-bounces+linux-phy=archiver.kernel.org@lists.infradead.org On 10/6/2022 2:43 AM, Bjorn Helgaas wrote: > On Mon, Oct 03, 2022 at 05:40:21PM +0530, Krishna Chaitanya Chundru wrote: >> On 9/30/2022 12:23 AM, Bjorn Helgaas wrote: >>> On Mon, Sep 26, 2022 at 09:00:11PM +0530, Krishna Chaitanya Chundru wrote: >>>> On 9/23/2022 7:56 PM, Bjorn Helgaas wrote: >>>>> On Fri, Sep 23, 2022 at 07:29:31AM +0530, Krishna Chaitanya Chundru wrote: >>>>>> On 9/23/2022 12:12 AM, Bjorn Helgaas wrote: >>>>>>> On Thu, Sep 22, 2022 at 09:09:28PM +0530, Krishna Chaitanya Chundru wrote: >>>>>>>> On 9/21/2022 10:26 PM, Bjorn Helgaas wrote: >>>>>>>>> On Wed, Sep 21, 2022 at 03:23:35PM +0530, Krishna Chaitanya Chundru wrote: >>>>>>>>>> On 9/20/2022 11:46 PM, Bjorn Helgaas wrote: >>>>>>>>>>> On Tue, Sep 20, 2022 at 03:52:23PM +0530, Krishna chaitanya chundru wrote: >>>>>>>>>>>> In qcom platform PCIe resources( clocks, phy >>>>>>>>>>>> etc..) can released when the link is in L1ss to >>>>>>>>>>>> reduce the power consumption. So if the link is >>>>>>>>>>>> in L1ss, release the PCIe resources. And when >>>>>>>>>>>> the system resumes, enable the PCIe resources if >>>>>>>>>>>> they released in the suspend path. >>>>>>>>>>> What's the connection with L1.x? Links enter L1.x >>>>>>>>>>> based on activity and timing. That doesn't seem >>>>>>>>>>> like a reliable indicator to turn PHYs off and >>>>>>>>>>> disable clocks. >>>>>>>>>> This is a Qcom PHY-specific feature (retaining the >>>>>>>>>> link state in L1.x with clocks turned off). It is >>>>>>>>>> possible only with the link being in l1.x. PHY can't >>>>>>>>>> retain the link state in L0 with the clocks turned >>>>>>>>>> off and we need to re-train the link if it's in L2 >>>>>>>>>> or L3. So we can support this feature only with >>>>>>>>>> L1.x. That is the reason we are taking l1.x as the >>>>>>>>>> trigger to turn off clocks (in only suspend path). >>>>>>>>> This doesn't address my question. L1.x is an ASPM >>>>>>>>> feature, which means hardware may enter or leave L1.x >>>>>>>>> autonomously at any time without software >>>>>>>>> intervention. Therefore, I don't think reading the >>>>>>>>> current state is a reliable way to decide anything. >>>>>>>> After the link enters the L1.x it will come out only if >>>>>>>> there is some activity on the link. As system is >>>>>>>> suspended and NVMe driver is also suspended (queues >>>>>>>> will freeze in suspend) who else can initiate any data. >>>>>>> I don't think we can assume that nothing will happen to >>>>>>> cause exit from L1.x. For instance, PCIe Messages for >>>>>>> INTx signaling, LTR, OBFF, PTM, etc., may be sent even >>>>>>> though we think the device is idle and there should be no >>>>>>> link activity. >>>>>> I don't think after the link enters into L1.x there will >>>>>> some activity on the link as you mentioned, except for PCIe >>>>>> messages like INTx/MSI/MSIX. These messages also will not >>>>>> come because the client drivers like NVMe will keep their >>>>>> device in the lowest power mode. >>>>>> >>>>>> The link will come out of L1.x only when there is config or >>>>>> memory access or some messages to trigger the interrupts >>>>>> from the devices. We are already making sure this access >>>>>> will not be there in S3. If the link is in L0 or L0s what >>>>>> you said is expected but not in L1.x >>>>> Forgive me for being skeptical, but we just spent a few months >>>>> untangling the fact that some switches send PTM request >>>>> messages even when they're in a non-D0 state. We expected >>>>> that devices in D3hot would not send such messages because >>>>> "why would they?" But it turns out the spec allows that, and >>>>> they actually *do*. >>>>> >>>>> I don't think it's robust interoperable design for a PCI >>>>> controller driver like qcom to assume anything about PCI >>>>> devices unless it's required by the spec. >>>> From pci spec 4, in sec 5.5 "Ports that support L1 PM Substates >>>> must not require a reference clock while in L1 PM Substates >>>> other than L1.0". If there is no reference clk we can say >>>> there is no activity on the link. If anything needs to be sent >>>> (such as LTR, or some messages ), the link needs to be back in >>>> L0 before it sends the packet to the link partner. >>>> >>>> To exit from L1.x clkreq pin should be asserted. >>>> >>>> In suspend after turning off clocks and phy we can enable to >>>> trigger an interrupt whenever the clk req pin asserts. In that >>>> interrupt handler, we can enable the pcie resources back. >>> From the point of view of the endpoint driver, ASPM should be >>> invisible -- no software intervention required. I think you're >>> suggesting that the PCIe controller driver could help exit L1.x by >>> handling a clk req interrupt and enabling clock and PHY then. >>> >>> But doesn't L1.x exit also have to happen within the time the >>> endpoint can tolerate? E.g., I think L1.2 exit has to happen >>> within the LTR time advertised by the endpoint (PCIe r6.0, sec >>> 5.5.5). How can we guarantee that if software is involved? >> It is true that it is difficult to guarantee those delays. On our >> internal boards, we are able to achieve this but that is not with >> linux kernel. >> >> With NVMe attach we have connected the protocol analyzer and tried >> to see if there are any transactions over the link. We found there >> are no transactions on the link once the link enters L1.x till we >> resume the system. As the NVMe is a passive system it is not >> initiating any transactions. >> >> This whole requirement came from the NVMe driver, it requires >> keeping the link active state when the system is suspended. >> >> There are only two things we can in do in PCIe suspend as we have to >> turn off PCIe clocks to allow the system to the lowest possible >> power state. >> >> 1) Keep the device in D3 cold and turn off all the clocks and phy >> etc. (It is not an ideal one as this decreases the NVMe lifetime >> because link-down and link-up is treated as a power cycle by a few >> NVMe devices). >> >> 2) This is the one we are proposing where we turn off the clocks, >> phy once the link enters L1ss. > It sounds like both options turn off the clocks and PHY. But > apparently they do not look the same to the NVMe endpoint? I guess > NVMe is in D3cold for 1), but it's in D0 for 2), right? > >> Can you please suggest us any other possible solutions to meet NVMe >> requirement (That is to keep the link active during suspend) and the >> Qcom platform requirement (that is to turn off all the clocks to >> allow a lower possible power state)? Qcom PCIe controller is >> compatible with v3.1 specification only. > The PCIe spec clearly envisions Refclk being turned off > (sec 5.5.3.3.1) and PHYs being powered off (sec 5.5.3.2) while in > L1.2. > > I've been assuming L1.2 exit (which includes Refclk being turned on > and PHYs being powered up) is completely handled by hardware, but it > sounds like the Qcom controller needs software assistance which fields > an interrupt when CLKREQ# is asserted and turns on Refclk and the > PHYs? > > 5.5.3 does say "All Link and PHY state must be maintained during L1.2, > or must be restored upon exit using implementation specific means", > and maybe Qcom counts as using implementation specific means. > > I *am* concerned about whether software can do the L1.2 exit fast > enough, but the biggest reason I'm struggling with this is because > using the syscore framework to work around IRQ affinity changes that > happen late in suspend just seems kind of kludgy and it doesn't seem > like it fits cleanly in the power management model. > > Bjorn Bjorn, Can you please suggest any another way to work around IRQ affinity changes. Thanks & Regards, Krishna Chaitanya. -- linux-phy mailing list linux-phy@lists.infradead.org https://lists.infradead.org/mailman/listinfo/linux-phy