Re: [RFC PATCH net-next v5 05/11] net/smc: implement DMB-related operations of loopback-ism

[Niklas Schnelle <schnelle@linux.ibm.com>]

On Thu, 2024-04-04 at 18:20 +0800, Wen Gu wrote:
> 
> On 2024/4/4 01:20, Gerd Bayer wrote:
> > On Sun, 2024-03-24 at 21:55 +0800, Wen Gu wrote:
> > 
> > When I instrumented this to see, why I still see tons of my other
> > temporary instrumentation messages from the "ism" driver, I found that
> > in my setup loopback-ism is used rather infrequently.
> > 
> > I suspect this is due to how the SMC proposals are constructed in
> > net/smc/af_smc.c and net/smc/smc_pnet.c - and later evaluated in
> > smc_check_ism_v2_match() - where there is a first-come-first-serve
> > selection.
> > 
> > I wonder if one should change that to favour loopback-ism over "real"
> > ISM devices - and how this could be achieved elegantly.
> > 
> > Just some food for thought... Probably little you can do on x86.
> > 
> 
> Yes, it is about the priority of available ISM devices, and now it
> is decided by their order in the smcd_dev_list. The later registered
> ISMv2 devices(without pnetid) will be added to the beginning of the
> list (see smcd_register_dev()). So there is a probability that
> loopback-ism will not be ranked first, since it is added into list
> earlier during smc_init().
> 
> If we have the runtime switch of loopback-ism, we can re-active the
> loopback-ism, that make it be re-added into the beginning of the dev
> list and be chosen first. Or a new netlink command to adjust the slot
> order of available ISM devices in the list. As we discussed before,
> that could be tasks in stage 1 or stage 2.
> 
> Thanks!

Maybe when adding the ISM devices we could instead make sure that all
ISM devices are added after loopback and loopback is added in the
beginning. I think loopback should always be preferred and would
consider it a bug if it isn't faster too. Between virtio-ism and ISM it
may be less clear so maybe for stage 2 we would want a priority setting
and then insert ordered by priority. Thoughts?