Re: [PATCH v3] pseries/drmem: don't cache node id in drmem_lmb struct

[Scott Cheloha <cheloha@linux.ibm.com>]

On Fri, Aug 21, 2020 at 10:33:10AM +0200, Laurent Dufour wrote:
> Le 11/08/2020 à 03:51, Scott Cheloha a écrit :
> > 
> > [...]
> > 
> > @@ -631,7 +638,7 @@ static int dlpar_memory_remove_by_ic(u32 lmbs_to_remove, u32 drc_index)
> >   static int dlpar_add_lmb(struct drmem_lmb *lmb)
> >   {
> >   	unsigned long block_sz;
> > -	int rc;
> > +	int nid, rc;
> > 
> >   	if (lmb->flags & DRCONF_MEM_ASSIGNED)
> >   		return -EINVAL;
> > @@ -642,11 +649,13 @@ static int dlpar_add_lmb(struct drmem_lmb *lmb)
> >   		return rc;
> >   	}
> > 
> > -	lmb_set_nid(lmb);
> >   	block_sz = memory_block_size_bytes();
> > 
> > +	/* Find the node id for this address. */
> > +	nid = memory_add_physaddr_to_nid(lmb->base_addr);
> 
> I think we could be more efficient here.
> Here is the call stack behind memory_add_physaddr_to_nid():
> 
>   memory_add_physaddr_to_nid(lmb->base_addr)
>     hot_add_scn_to_nid()
>       if (of_find_node_by_path("/ibm,dynamic-reconfiguration-memory")) == true*
>       then
> 	  hot_add_drconf_scn_to_nid()
> 	    for_each_drmem_lmb() to find the LMB based on lmb->base_addr
> 	      of_drconf_to_nid_single(found LMB)
> 		use lmb->aa_index to get the nid.
> 
> * that test is necessarily true when called from dlpar_add_lmb()
> otherwise the call to update_lmb_associativity_index() would have
> failed earlier.
> 
> Basically, we have a LMB and we later walk all the LMBs to find that lmb
> again.  In the case of dlpar_add_lmb(), it would be more efficient to
> directly call of_drconf_to_nid_single(). That function is not exported
> from arch/powerpc/mm/numa.c but it may be good to export it through that
> patch.

I've posted a patch for this:

https://lore.kernel.org/linuxppc-dev/20200910175637.2865160-1-cheloha@linux.ibm.com/T/#u

The speedup is nice, especially for LMBs at the end of the array.