From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pg0-x244.google.com (mail-pg0-x244.google.com [IPv6:2607:f8b0:400e:c05::244]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 418Mty3XD6zF09q for ; Mon, 18 Jun 2018 17:20:24 +1000 (AEST) Received: by mail-pg0-x244.google.com with SMTP id d2-v6so7095076pga.13 for ; Mon, 18 Jun 2018 00:20:24 -0700 (PDT) Date: Mon, 18 Jun 2018 17:20:14 +1000 From: Alexey Kardashevskiy To: Benjamin Herrenschmidt Cc: linuxppc-dev@lists.ozlabs.org, Alistair Popple , Russell Currey Subject: Re: [PATCH kernel 2/2] powerpc/powernv: Define PHB4 type and enable sketchy bypass on POWER9 Message-ID: <20180618172014.2bba467d@aik.ozlabs.ibm.com> In-Reply-To: <82ebaedf4279dce05a7cd6bdb0d9b84bdf232a02.camel@kernel.crashing.org> References: <20180601081028.29401-1-aik@ozlabs.ru> <20180601081028.29401-3-aik@ozlabs.ru> <20180618121307.199cd998@aik.ozlabs.ibm.com> <82ebaedf4279dce05a7cd6bdb0d9b84bdf232a02.camel@kernel.crashing.org> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , On Mon, 18 Jun 2018 14:44:56 +1000 Benjamin Herrenschmidt wrote: > On Mon, 2018-06-18 at 12:13 +1000, Alexey Kardashevskiy wrote: > > On Sat, 16 Jun 2018 11:05:19 +1000 > > Benjamin Herrenschmidt wrote: > > > > > On Fri, 2018-06-01 at 18:10 +1000, Alexey Kardashevskiy wrote: > > > > These are found in POWER9 chips. Right now these PHBs have unknown type > > > > so changing it to PHB4 won't make much of a difference except enabling > > > > sketchy bypass for POWER9 as this does below. > > > > > > And that will break on multi-chip systems since P9 doesn't have the > > > memory contiguous (it has the chip ID in the top bits). > > > > > > This did not break mine and it is hard to see why it would break at all > > if we use 1G pages and the maximum we need to cover is 48 bits (this > > is what we are trying to support here - all these gpus, right?), or is > > it more now? If so, I have posted v2 of tce multilevel dynamic > > allocation which helps with enormous tce tables. > > The whole point of sketchy bypass is to deal with devices with small > amount of DMA bits... most Radeon's have 40 for example. So that won't > work terribly well. 40 with 1GB pages needs 40-30=10 bits i.e. (1<<10)*8 = 8KB for the entire TCE table. I am definitely missing something here, what is it? > > Cheers, > Ben. > > > > > > > > > > > Russell is working on a different implementation that should be much > > > more imune to the system physical memory layout. > > > > > > > Signed-off-by: Alexey Kardashevskiy > > > > --- > > > > arch/powerpc/platforms/powernv/pci.h | 1 + > > > > arch/powerpc/platforms/powernv/pci-ioda.c | 5 ++++- > > > > 2 files changed, 5 insertions(+), 1 deletion(-) > > > > > > > > diff --git a/arch/powerpc/platforms/powernv/pci.h b/arch/powerpc/platforms/powernv/pci.h > > > > index eada4b6..1408247 100644 > > > > --- a/arch/powerpc/platforms/powernv/pci.h > > > > +++ b/arch/powerpc/platforms/powernv/pci.h > > > > @@ -23,6 +23,7 @@ enum pnv_phb_model { > > > > PNV_PHB_MODEL_UNKNOWN, > > > > PNV_PHB_MODEL_P7IOC, > > > > PNV_PHB_MODEL_PHB3, > > > > + PNV_PHB_MODEL_PHB4, > > > > PNV_PHB_MODEL_NPU, > > > > PNV_PHB_MODEL_NPU2, > > > > }; > > > > diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c > > > > index 9239142..66c2804 100644 > > > > --- a/arch/powerpc/platforms/powernv/pci-ioda.c > > > > +++ b/arch/powerpc/platforms/powernv/pci-ioda.c > > > > @@ -1882,7 +1882,8 @@ static int pnv_pci_ioda_dma_set_mask(struct pci_dev *pdev, u64 dma_mask) > > > > if (dma_mask >> 32 && > > > > dma_mask > (memory_hotplug_max() + (1ULL << 32)) && > > > > pnv_pci_ioda_pe_single_vendor(pe) && > > > > - phb->model == PNV_PHB_MODEL_PHB3) { > > > > + (phb->model == PNV_PHB_MODEL_PHB3 || > > > > + phb->model == PNV_PHB_MODEL_PHB4)) { > > > > /* Configure the bypass mode */ > > > > rc = pnv_pci_ioda_dma_64bit_bypass(pe); > > > > if (rc) > > > > @@ -3930,6 +3931,8 @@ static void __init pnv_pci_init_ioda_phb(struct device_node *np, > > > > phb->model = PNV_PHB_MODEL_P7IOC; > > > > else if (of_device_is_compatible(np, "ibm,power8-pciex")) > > > > phb->model = PNV_PHB_MODEL_PHB3; > > > > + else if (of_device_is_compatible(np, "ibm,power9-pciex")) > > > > + phb->model = PNV_PHB_MODEL_PHB4; > > > > else if (of_device_is_compatible(np, "ibm,power8-npu-pciex")) > > > > phb->model = PNV_PHB_MODEL_NPU; > > > > else if (of_device_is_compatible(np, "ibm,power9-npu-pciex")) > > > > > > > > -- > > Alexey -- Alexey