From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.3 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4D906C10F25 for ; Wed, 11 Mar 2020 06:48:49 +0000 (UTC) Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 218A22146E for ; Wed, 11 Mar 2020 06:48:47 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ozlabs-ru.20150623.gappssmtp.com header.i=@ozlabs-ru.20150623.gappssmtp.com header.b="zHFANCuS" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 218A22146E Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=ozlabs.ru Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Received: from lists.ozlabs.org (unknown [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 48cjGk21h5zDqRK for ; Wed, 11 Mar 2020 17:48:46 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=ozlabs.ru (client-ip=209.85.214.193; helo=mail-pl1-f193.google.com; envelope-from=aik@ozlabs.ru; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=ozlabs.ru Received: from mail-pl1-f193.google.com (mail-pl1-f193.google.com [209.85.214.193]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 48cjF54nG0zDqNJ for ; Wed, 11 Mar 2020 17:47:16 +1100 (AEDT) Received: by mail-pl1-f193.google.com with SMTP id j20so624656pll.6 for ; Tue, 10 Mar 2020 23:47:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ozlabs-ru.20150623.gappssmtp.com; s=20150623; h=subject:to:cc:references:from:autocrypt:message-id:date:user-agent :mime-version:in-reply-to:content-language:content-transfer-encoding; bh=pX1+oQrgOMippVWX547i6yVi7EXAjGTxuQ05bV6cwkM=; b=zHFANCuSBc+1PY5liREXkmNjG/Udk86E2pr5eQOz3fZnwy0ALtRU0M5vd4XKoHRYhe de4NqgLlKRZoVRjaKbfcLiVhX7k286sKTOX0OcLfDFkQCvdjbDKJ1EEVGpMmKj/O/0i0 n/6ozc51X07uCOGwp2YzGATXBoSj1lNjPOmuCxwLRBSQWiws0ZkAKf8e/4ljZPKX08mR tubtuYjItoiklaK11ib4ZKQCAisKdwmJUJAeDVaxyrW/sYNFzR1EK8En6JzAVuAiDEg0 7UfHHbFw0z/sZ4quTuCH0mtHzcokJkGdUN3qOL84RMDU5m8OpV5fF59UJ//CiEfa7qzk GdEA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:autocrypt :message-id:date:user-agent:mime-version:in-reply-to :content-language:content-transfer-encoding; bh=pX1+oQrgOMippVWX547i6yVi7EXAjGTxuQ05bV6cwkM=; b=ALPvOXkymkVP1xbujtOzEwA/g5BQvXZBlbRXHfcUf6RPJnQN4kk4sQ7yWwHvgDNwww eccbJAftv1c2dzn94WAx/KenDZIJEvxngq40jiWWS/aBrK37pAmNfzcgJ7PnyRDgVIK9 j7PhmdIrd4nIdRUraOTj88DXs4chIBS2CvK8e0DE++nQGylbFCPG25Nxo05ClZV6zxez 0F3fLf6oGY+XkDNCWx2xP2l0Geq+FmkImCHkR9Pqan0eWj4FMEXNNma3p8OJsBUJ5cld yEEJpqivYPyBlJeKtJSj2pJ5l9WyTOGIwtqwMWASJn+J401QNG8Rqha80HTTTxk5IKGr hNWg== X-Gm-Message-State: ANhLgQ0FiAh+5RDvN5831mzN1kGQCNGk1p/VDXA55YE2B9719AKIDugm iQcS+bDWmQ8FwEqR9Xxle3qTpg== X-Google-Smtp-Source: ADFU+vsvDIPE7wUqeW5gL1lb4mCdCSko+7oyQ7eo0ws4GtY3P+0erxXmQsP/YOBaP1c77lvpizhLYw== X-Received: by 2002:a17:90a:2e05:: with SMTP id q5mr1852827pjd.68.1583909234050; Tue, 10 Mar 2020 23:47:14 -0700 (PDT) Received: from [10.61.2.175] ([122.99.82.10]) by smtp.gmail.com with ESMTPSA id u23sm49520221pfm.29.2020.03.10.23.47.11 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 10 Mar 2020 23:47:13 -0700 (PDT) Subject: Re: [PATCH] pseries/iommu: Tweak ddw behavior in presence of pmem To: "Aneesh Kumar K.V" , Vaibhav Jain , linuxppc-dev@lists.ozlabs.org References: <20200311055050.161556-1-vaibhav@linux.ibm.com> <87y2s7mnhx.fsf@linux.ibm.com> From: Alexey Kardashevskiy Autocrypt: addr=aik@ozlabs.ru; keydata= mQINBE+rT0sBEADFEI2UtPRsLLvnRf+tI9nA8T91+jDK3NLkqV+2DKHkTGPP5qzDZpRSH6mD EePO1JqpVuIow/wGud9xaPA5uvuVgRS1q7RU8otD+7VLDFzPRiRE4Jfr2CW89Ox6BF+q5ZPV /pS4v4G9eOrw1v09lEKHB9WtiBVhhxKK1LnUjPEH3ifkOkgW7jFfoYgTdtB3XaXVgYnNPDFo PTBYsJy+wr89XfyHr2Ev7BB3Xaf7qICXdBF8MEVY8t/UFsesg4wFWOuzCfqxFmKEaPDZlTuR tfLAeVpslNfWCi5ybPlowLx6KJqOsI9R2a9o4qRXWGP7IwiMRAC3iiPyk9cknt8ee6EUIxI6 t847eFaVKI/6WcxhszI0R6Cj+N4y+1rHfkGWYWupCiHwj9DjILW9iEAncVgQmkNPpUsZECLT WQzMuVSxjuXW4nJ6f4OFHqL2dU//qR+BM/eJ0TT3OnfLcPqfucGxubhT7n/CXUxEy+mvWwnm s9p4uqVpTfEuzQ0/bE6t7dZdPBua7eYox1AQnk8JQDwC3Rn9kZq2O7u5KuJP5MfludMmQevm pHYEMF4vZuIpWcOrrSctJfIIEyhDoDmR34bCXAZfNJ4p4H6TPqPh671uMQV82CfTxTrMhGFq 8WYU2AH86FrVQfWoH09z1WqhlOm/KZhAV5FndwVjQJs1MRXD8QARAQABtCRBbGV4ZXkgS2Fy ZGFzaGV2c2tpeSA8YWlrQG96bGFicy5ydT6JAjgEEwECACIFAk+rT0sCGwMGCwkIBwMCBhUI AgkKCwQWAgMBAh4BAheAAAoJEIYTPdgrwSC5fAIP/0wf/oSYaCq9PhO0UP9zLSEz66SSZUf7 AM9O1rau1lJpT8RoNa0hXFXIVbqPPKPZgorQV8SVmYRLr0oSmPnTiZC82x2dJGOR8x4E01gK TanY53J/Z6+CpYykqcIpOlGsytUTBA+AFOpdaFxnJ9a8p2wA586fhCZHVpV7W6EtUPH1SFTQ q5xvBmr3KkWGjz1FSLH4FeB70zP6uyuf/B2KPmdlPkyuoafl2UrU8LBADi/efc53PZUAREih sm3ch4AxaL4QIWOmlE93S+9nHZSRo9jgGXB1LzAiMRII3/2Leg7O4hBHZ9Nki8/fbDo5///+ kD4L7UNbSUM/ACWHhd4m1zkzTbyRzvL8NAVQ3rckLOmju7Eu9whiPueGMi5sihy9VQKHmEOx OMEhxLRQbzj4ypRLS9a+oxk1BMMu9cd/TccNy0uwx2UUjDQw/cXw2rRWTRCxoKmUsQ+eNWEd iYLW6TCfl9CfHlT6A7Zmeqx2DCeFafqEd69DqR9A8W5rx6LQcl0iOlkNqJxxbbW3ddDsLU/Y r4cY20++WwOhSNghhtrroP+gouTOIrNE/tvG16jHs8nrYBZuc02nfX1/gd8eguNfVX/ZTHiR gHBWe40xBKwBEK2UeqSpeVTohYWGBkcd64naGtK9qHdo1zY1P55lHEc5Uhlk743PgAnOi27Q ns5zuQINBE+rT0sBEACnV6GBSm+25ACT+XAE0t6HHAwDy+UKfPNaQBNTTt31GIk5aXb2Kl/p AgwZhQFEjZwDbl9D/f2GtmUHWKcCmWsYd5M/6Ljnbp0Ti5/xi6FyfqnO+G/wD2VhGcKBId1X Em/B5y1kZVbzcGVjgD3HiRTqE63UPld45bgK2XVbi2+x8lFvzuFq56E3ZsJZ+WrXpArQXib2 hzNFwQleq/KLBDOqTT7H+NpjPFR09Qzfa7wIU6pMNF2uFg5ihb+KatxgRDHg70+BzQfa6PPA o1xioKXW1eHeRGMmULM0Eweuvpc7/STD3K7EJ5bBq8svoXKuRxoWRkAp9Ll65KTUXgfS+c0x gkzJAn8aTG0z/oEJCKPJ08CtYQ5j7AgWJBIqG+PpYrEkhjzSn+DZ5Yl8r+JnZ2cJlYsUHAB9 jwBnWmLCR3gfop65q84zLXRQKWkASRhBp4JK3IS2Zz7Nd/Sqsowwh8x+3/IUxVEIMaVoUaxk Wt8kx40h3VrnLTFRQwQChm/TBtXqVFIuv7/Mhvvcq11xnzKjm2FCnTvCh6T2wJw3de6kYjCO 7wsaQ2y3i1Gkad45S0hzag/AuhQJbieowKecuI7WSeV8AOFVHmgfhKti8t4Ff758Z0tw5Fpc BFDngh6Lty9yR/fKrbkkp6ux1gJ2QncwK1v5kFks82Cgj+DSXK6GUQARAQABiQIfBBgBAgAJ BQJPq09LAhsMAAoJEIYTPdgrwSC5NYEP/2DmcEa7K9A+BT2+G5GXaaiFa098DeDrnjmRvumJ BhA1UdZRdfqICBADmKHlJjj2xYo387sZpS6ABbhrFxM6s37g/pGPvFUFn49C47SqkoGcbeDz Ha7JHyYUC+Tz1dpB8EQDh5xHMXj7t59mRDgsZ2uVBKtXj2ZkbizSHlyoeCfs1gZKQgQE8Ffc F8eWKoqAQtn3j4nE3RXbxzTJJfExjFB53vy2wV48fUBdyoXKwE85fiPglQ8bU++0XdOr9oyy j1llZlB9t3tKVv401JAdX8EN0++ETiOovQdzE1m+6ioDCtKEx84ObZJM0yGSEGEanrWjiwsa nzeK0pJQM9EwoEYi8TBGhHC9ksaAAQipSH7F2OHSYIlYtd91QoiemgclZcSgrxKSJhyFhmLr QEiEILTKn/pqJfhHU/7R7UtlDAmFMUp7ByywB4JLcyD10lTmrEJ0iyRRTVfDrfVP82aMBXgF tKQaCxcmLCaEtrSrYGzd1sSPwJne9ssfq0SE/LM1J7VdCjm6OWV33SwKrfd6rOtvOzgadrG6 3bgUVBw+bsXhWDd8tvuCXmdY4bnUblxF2B6GOwSY43v6suugBttIyW5Bl2tXSTwP+zQisOJo +dpVG2pRr39h+buHB3NY83NEPXm1kUOhduJUA17XUY6QQCAaN4sdwPqHq938S3EmtVhsuQIN BFq54uIBEACtPWrRdrvqfwQF+KMieDAMGdWKGSYSfoEGGJ+iNR8v255IyCMkty+yaHafvzpl PFtBQ/D7Fjv+PoHdFq1BnNTk8u2ngfbre9wd9MvTDsyP/TmpF0wyyTXhhtYvE267Av4X/BQT lT9IXKyAf1fP4BGYdTNgQZmAjrRsVUW0j6gFDrN0rq2J9emkGIPvt9rQt6xGzrd6aXonbg5V j6Uac1F42ESOZkIh5cN6cgnGdqAQb8CgLK92Yc8eiCVCH3cGowtzQ2m6U32qf30cBWmzfSH0 HeYmTP9+5L8qSTA9s3z0228vlaY0cFGcXjdodBeVbhqQYseMF9FXiEyRs28uHAJEyvVZwI49 CnAgVV/n1eZa5qOBpBL+ZSURm8Ii0vgfvGSijPGbvc32UAeAmBWISm7QOmc6sWa1tobCiVmY SNzj5MCNk8z4cddoKIc7Wt197+X/X5JPUF5nQRvg3SEHvfjkS4uEst9GwQBpsbQYH9MYWq2P PdxZ+xQE6v7cNB/pGGyXqKjYCm6v70JOzJFmheuUq0Ljnfhfs15DmZaLCGSMC0Amr+rtefpA y9FO5KaARgdhVjP2svc1F9KmTUGinSfuFm3quadGcQbJw+lJNYIfM7PMS9fftq6vCUBoGu3L j4xlgA/uQl/LPneu9mcvit8JqcWGS3fO+YeagUOon1TRqQARAQABiQRsBBgBCAAgFiEEZSrP ibrORRTHQ99dhhM92CvBILkFAlq54uICGwICQAkQhhM92CvBILnBdCAEGQEIAB0WIQQIhvWx rCU+BGX+nH3N7sq0YorTbQUCWrni4gAKCRDN7sq0YorTbVVSD/9V1xkVFyUCZfWlRuryBRZm S4GVaNtiV2nfUfcThQBfF0sSW/aFkLP6y+35wlOGJE65Riw1C2Ca9WQYk0xKvcZrmuYkK3DZ 0M9/Ikkj5/2v0vxz5Z5w/9+IaCrnk7pTnHZuZqOh23NeVZGBls/IDIvvLEjpD5UYicH0wxv+ X6cl1RoP2Kiyvenf0cS73O22qSEw0Qb9SId8wh0+ClWet2E7hkjWFkQfgJ3hujR/JtwDT/8h 3oCZFR0KuMPHRDsCepaqb/k7VSGTLBjVDOmr6/C9FHSjq0WrVB9LGOkdnr/xcISDZcMIpbRm EkIQ91LkT/HYIImL33ynPB0SmA+1TyMgOMZ4bakFCEn1vxB8Ir8qx5O0lHMOiWMJAp/PAZB2 r4XSSHNlXUaWUg1w3SG2CQKMFX7vzA31ZeEiWO8tj/c2ZjQmYjTLlfDK04WpOy1vTeP45LG2 wwtMA1pKvQ9UdbYbovz92oyZXHq81+k5Fj/YA1y2PI4MdHO4QobzgREoPGDkn6QlbJUBf4To pEbIGgW5LRPLuFlOPWHmIS/sdXDrllPc29aX2P7zdD/ivHABslHmt7vN3QY+hG0xgsCO1JG5 pLORF2N5XpM95zxkZqvYfC5tS/qhKyMcn1kC0fcRySVVeR3tUkU8/caCqxOqeMe2B6yTiU1P aNDq25qYFLeYxg67D/4w/P6BvNxNxk8hx6oQ10TOlnmeWp1q0cuutccblU3ryRFLDJSngTEu ZgnOt5dUFuOZxmMkqXGPHP1iOb+YDznHmC0FYZFG2KAc9pO0WuO7uT70lL6larTQrEneTDxQ CMQLP3qAJ/2aBH6SzHIQ7sfbsxy/63jAiHiT3cOaxAKsWkoV2HQpnmPOJ9u02TPjYmdpeIfa X2tXyeBixa3i/6dWJ4nIp3vGQicQkut1YBwR7dJq67/FCV3Mlj94jI0myHT5PIrCS2S8LtWX ikTJSxWUKmh7OP5mrqhwNe0ezgGiWxxvyNwThOHc5JvpzJLd32VDFilbxgu4Hhnf6LcgZJ2c Zd44XWqUu7FzVOYaSgIvTP0hNrBYm/E6M7yrLbs3JY74fGzPWGRbBUHTZXQEqQnZglXaVB5V ZhSFtHopZnBSCUSNDbB+QGy4B/E++Bb02IBTGl/JxmOwG+kZUnymsPvTtnNIeTLHxN/H/ae0 c7E5M+/NpslPCmYnDjs5qg0/3ihh6XuOGggZQOqrYPC3PnsNs3NxirwOkVPQgO6mXxpuifvJ DG9EMkK8IBXnLulqVk54kf7fE0jT/d8RTtJIA92GzsgdK2rpT1MBKKVffjRFGwN7nQVOzi4T XrB5p+6ML7Bd84xOEGsj/vdaXmz1esuH7BOZAGEZfLRCHJ0GVCSssg== Message-ID: <03ccbe79-49ea-0f03-a5b0-60abababe0dc@ozlabs.ru> Date: Wed, 11 Mar 2020 17:47:08 +1100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.5.0 MIME-Version: 1.0 In-Reply-To: <87y2s7mnhx.fsf@linux.ibm.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Michael Ellerman Errors-To: linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Sender: "Linuxppc-dev" On 11/03/2020 17:09, Aneesh Kumar K.V wrote: > Vaibhav Jain writes: > >> Recently we discovered an issue on pseries guests that prevents pci >> devices from accessing pmem memory via DMA. Performing such an >> operation will cause PHB to freeze the corresponding partition >> endpoint and in some scenarios will shutdown the disk that hosts the >> rootfs. >> >> A fix for this is in works until then this patch proposes to disable >> DDW if pmem nodes are present in the device-tree. This would force all >> DMA from I/O adapters through default 2-GB window and prevent direct >> access of pmem memory which is located beyond 4-TB guest physical >> address. >> >> Since this change can have performance penalty for cases where its >> known that i/o adapters wont be performing DMA to pmem, the patch >> adds new args to the 'disable_ddw' kernel commanline flag with >> following possible values: >> >> 'default' : Enable DDW only if no Pmem nodes present in device-tree >> 'Yes' : Disable DDW always >> 'No' : Force enable DDW even if Pmem nodes present in device-tree > > This is wrong, if we find pmem attached to LPAR we should not enable ddw > at all. Enabling ddw results in us using direct dma ops and that cause > crashes with vpmem address as you explained earlier. I suggest using existing device::dma_mask/coherent_dma_mask and/or device::bus_dma_limit - we could actually set these up in enable_ddw() (where we know the window size) and then the rest of the code would use it naturally and switch between "direct" or IOMMU DMA. The immediate problem with this is that the generic dma_set_mask()/dma_set_coherent_mask() do not ask the platform about the maximum mask supported, it is either what the device requested (often it is simply 64bit) or we fall back to 32bit, we could probably fix that. Thanks, > > >> >> Signed-off-by: Vaibhav Jain >> --- >> .../admin-guide/kernel-parameters.txt | 10 ++- >> arch/powerpc/platforms/pseries/iommu.c | 67 +++++++++++++++++-- >> 2 files changed, 70 insertions(+), 7 deletions(-) >> >> diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt >> index c07815d230bc..58e09f7a2cb9 100644 >> --- a/Documentation/admin-guide/kernel-parameters.txt >> +++ b/Documentation/admin-guide/kernel-parameters.txt >> @@ -901,9 +901,13 @@ >> The feature only exists starting from >> Arch Perfmon v4 (Skylake and newer). >> >> - disable_ddw [PPC/PSERIES] >> - Disable Dynamic DMA Window support. Use this if >> - to workaround buggy firmware. >> + disable_ddw= [PPC/PSERIES] >> + Controls weather Dynamic DMA Window support is enabled. >> + Use this if to workaround buggy firmware. Following >> + values are supported: >> + on Disable ddw always >> + off Enable ddw always >> + default Enable ddw if no Persistent memory present (default) >> >> disable_ipv6= [IPV6] >> See Documentation/networking/ipv6.txt. >> diff --git a/arch/powerpc/platforms/pseries/iommu.c b/arch/powerpc/platforms/pseries/iommu.c >> index 2e0a8eab5588..97498cc25c9f 100644 >> --- a/arch/powerpc/platforms/pseries/iommu.c >> +++ b/arch/powerpc/platforms/pseries/iommu.c >> @@ -12,6 +12,7 @@ >> >> #include >> #include >> +#include >> #include >> #include >> #include >> @@ -755,12 +756,39 @@ static void pci_dma_dev_setup_pSeries(struct pci_dev *dev) >> pci_name(dev)); >> } >> >> +/* >> + * Following values for the variable are handled >> + * '-1': Force enable ddw even if Persistent memory is present >> + * '0' : Enable ddw if no Persistent memory present (default) >> + * '1' : Disable ddw always >> + */ >> static int __read_mostly disable_ddw; >> >> -static int __init disable_ddw_setup(char *str) >> +static int __init disable_ddw_setup(char *param) >> { >> - disable_ddw = 1; >> - printk(KERN_INFO "ppc iommu: disabling ddw.\n"); >> + bool val; >> + int res; >> + >> + /* Maintain old behaviour that disables DDW when flag given */ >> + if (!param) { >> + disable_ddw = 1; >> + return 0; >> + } >> + >> + res = strtobool(param, &val); >> + >> + if (!res) { >> + if (val) { >> + disable_ddw = 1; >> + pr_info("ppc iommu: disabling ddw.\n"); >> + } else if (!val) { >> + /* Force enable of DDW even if pmem is available */ >> + disable_ddw = -1; >> + pr_info("ppc iommu: will force enable ddw.\n"); >> + } >> + } else if (strcmp(param, "default") == 0) { >> + disable_ddw = 0; >> + } >> >> return 0; >> } >> @@ -1313,6 +1341,37 @@ static struct notifier_block iommu_reconfig_nb = { >> .notifier_call = iommu_reconfig_notifier, >> }; >> >> +/* Check if DDW can be supported for this lpar */ >> +int ddw_supported(void) >> +{ >> + struct device_node *dn; >> + >> + if (disable_ddw == -1) /* force enable ddw */ >> + goto out; >> + >> + if (disable_ddw == 1) >> + return 0; >> + >> + /* >> + * Due to DMA window limitations currently DDW is not supported >> + * for persistent memory. This is due 1 TiB size of direct mapped >> + * DMA window size limitation enforce by phyp. Since pmem memory >> + * will be mapped at phy address > 4TiB, we cannot accmodate pmem >> + * in the DDW window and DMA's to/from the pmem memory will result in >> + * PHBs getting frozen triggering EEH. Hence for the the time being >> + * disable DDW in presence of a 'ibm,pmemory' node. >> + */ > > > That comment is not complete. It is not a phyp limitations it is a linux > kernel implementation detail where we have one offset for direct map > dma. > >> + dn = of_find_compatible_node(NULL, NULL, "ibm,pmemory"); >> + if (dn) { >> + pr_info("IOMMU: Disabling DDW as pmem memory available\n"); >> + of_node_put(dn); >> + return 0; >> + } >> + out: >> + pr_info("IOMMU: Enabling DDW support\n"); >> + return 1; >> +} >> + >> /* These are called very early. */ >> void iommu_init_early_pSeries(void) >> { >> @@ -1322,7 +1381,7 @@ void iommu_init_early_pSeries(void) >> if (firmware_has_feature(FW_FEATURE_LPAR)) { >> pseries_pci_controller_ops.dma_bus_setup = pci_dma_bus_setup_pSeriesLP; >> pseries_pci_controller_ops.dma_dev_setup = pci_dma_dev_setup_pSeriesLP; >> - if (!disable_ddw) >> + if (ddw_supported()) >> pseries_pci_controller_ops.iommu_bypass_supported = >> iommu_bypass_supported_pSeriesLP; >> } else { >> -- >> 2.24.1 > > I guess a much simpler patch is below. > > diff --git a/arch/powerpc/platforms/pseries/iommu.c b/arch/powerpc/platforms/pseries/iommu.c > index 2e0a8eab5588..99f72162dd85 100644 > --- a/arch/powerpc/platforms/pseries/iommu.c > +++ b/arch/powerpc/platforms/pseries/iommu.c > @@ -1313,6 +1313,31 @@ static struct notifier_block iommu_reconfig_nb = { > .notifier_call = iommu_reconfig_notifier, > }; > > +static bool pmem_attached(void) > +{ > + struct device_node *dn; > + /* > + * Depending on different partitionable endpoints, dynamic dma window > + * implementation in Linux restrict the max addressable memory that > + * can be DMA mapped using dynamic dma window feature. This limit > + * depends on the I/O map page size and max TCE entries enabled > + * by Phyp. With 64K page size with some PE this limit is 1TB. > + * When persistent memory is bound above this address, we need to > + * disable ddw feature for this PE. > + * > + * Since kernel won't know the pmem bind-address early during boot, > + * force disable dynamic dma window feature for all PEs if pmem is > + * attached to the LPAR. > + */ > + dn = of_find_compatible_node(NULL, NULL, "ibm,pmemory"); > + if (dn) { > + pr_info("IOMMU: Disabling DDW as pmem memory available\n"); > + of_node_put(dn); > + return true; > + } > + return false; > +} > + > /* These are called very early. */ > void iommu_init_early_pSeries(void) > { > @@ -1322,7 +1347,7 @@ void iommu_init_early_pSeries(void) > if (firmware_has_feature(FW_FEATURE_LPAR)) { > pseries_pci_controller_ops.dma_bus_setup = pci_dma_bus_setup_pSeriesLP; > pseries_pci_controller_ops.dma_dev_setup = pci_dma_dev_setup_pSeriesLP; > - if (!disable_ddw) > + if (!(disable_ddw || pmem_attached())) > pseries_pci_controller_ops.iommu_bypass_supported = > iommu_bypass_supported_pSeriesLP; > } else { > -- Alexey