Re: [PATCH kernel 2/2] powerpc/powernv: Define PHB4 type and enable sketchy bypass on POWER9

[Alexey Kardashevskiy <aik@ozlabs.ru>]

On Mon, 18 Jun 2018 14:44:56 +1000
Benjamin Herrenschmidt <benh@kernel.crashing.org> wrote:

> On Mon, 2018-06-18 at 12:13 +1000, Alexey Kardashevskiy wrote:
> > On Sat, 16 Jun 2018 11:05:19 +1000
> > Benjamin Herrenschmidt <benh@kernel.crashing.org> wrote:
> >   
> > > On Fri, 2018-06-01 at 18:10 +1000, Alexey Kardashevskiy wrote:  
> > > > These are found in POWER9 chips. Right now these PHBs have unknown type
> > > > so changing it to PHB4 won't make much of a difference except enabling
> > > > sketchy bypass for POWER9 as this does below.    
> > > 
> > > And that will break on multi-chip systems since P9 doesn't have the
> > > memory contiguous (it has the chip ID in the top bits).  
> > 
> > 
> > This did not break mine and it is hard to see why it would break at all
> > if we use 1G pages and the maximum we need to cover is 48 bits (this
> > is what we are trying to support here - all these gpus, right?), or is
> > it more now? If so, I have posted v2 of tce multilevel dynamic
> > allocation which helps with enormous tce tables.  
> 
> The whole point of sketchy bypass is to deal with devices with small
> amount of DMA bits... most Radeon's have 40 for example. So that won't
> work terribly well.


40 with 1GB pages needs 40-30=10 bits i.e. (1<<10)*8 = 8KB for the
entire TCE table. I am definitely missing something here, what is it?



> 
> Cheers,
> Ben.
> 
> > 
> >   
> > > 
> > > Russell is working on a different implementation that should be much
> > > more imune to the system physical memory layout.
> > >   
> > > > Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
> > > > ---
> > > >  arch/powerpc/platforms/powernv/pci.h      | 1 +
> > > >  arch/powerpc/platforms/powernv/pci-ioda.c | 5 ++++-
> > > >  2 files changed, 5 insertions(+), 1 deletion(-)
> > > > 
> > > > diff --git a/arch/powerpc/platforms/powernv/pci.h b/arch/powerpc/platforms/powernv/pci.h
> > > > index eada4b6..1408247 100644
> > > > --- a/arch/powerpc/platforms/powernv/pci.h
> > > > +++ b/arch/powerpc/platforms/powernv/pci.h
> > > > @@ -23,6 +23,7 @@ enum pnv_phb_model {
> > > >  	PNV_PHB_MODEL_UNKNOWN,
> > > >  	PNV_PHB_MODEL_P7IOC,
> > > >  	PNV_PHB_MODEL_PHB3,
> > > > +	PNV_PHB_MODEL_PHB4,
> > > >  	PNV_PHB_MODEL_NPU,
> > > >  	PNV_PHB_MODEL_NPU2,
> > > >  };
> > > > diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c
> > > > index 9239142..66c2804 100644
> > > > --- a/arch/powerpc/platforms/powernv/pci-ioda.c
> > > > +++ b/arch/powerpc/platforms/powernv/pci-ioda.c
> > > > @@ -1882,7 +1882,8 @@ static int pnv_pci_ioda_dma_set_mask(struct pci_dev *pdev, u64 dma_mask)
> > > >  		if (dma_mask >> 32 &&
> > > >  		    dma_mask > (memory_hotplug_max() + (1ULL << 32)) &&
> > > >  		    pnv_pci_ioda_pe_single_vendor(pe) &&
> > > > -		    phb->model == PNV_PHB_MODEL_PHB3) {
> > > > +		    (phb->model == PNV_PHB_MODEL_PHB3 ||
> > > > +		     phb->model == PNV_PHB_MODEL_PHB4)) {
> > > >  			/* Configure the bypass mode */
> > > >  			rc = pnv_pci_ioda_dma_64bit_bypass(pe);
> > > >  			if (rc)
> > > > @@ -3930,6 +3931,8 @@ static void __init pnv_pci_init_ioda_phb(struct device_node *np,
> > > >  		phb->model = PNV_PHB_MODEL_P7IOC;
> > > >  	else if (of_device_is_compatible(np, "ibm,power8-pciex"))
> > > >  		phb->model = PNV_PHB_MODEL_PHB3;
> > > > +	else if (of_device_is_compatible(np, "ibm,power9-pciex"))
> > > > +		phb->model = PNV_PHB_MODEL_PHB4;
> > > >  	else if (of_device_is_compatible(np, "ibm,power8-npu-pciex"))
> > > >  		phb->model = PNV_PHB_MODEL_NPU;
> > > >  	else if (of_device_is_compatible(np, "ibm,power9-npu-pciex"))    
> > 
> > 
> > 
> > --
> > Alexey  



--
Alexey