Re: [PATCH v2] RDMA/cma: Handle ARPHRD_NONE devices for iWARP

[Jason Gunthorpe <jgg@nvidia.com>]

On Thu, Jun 08, 2023 at 04:05:54PM -0400, Chuck Lever wrote:
> From: Chuck Lever <chuck.lever@oracle.com>
> 
> We would like to enable the use of siw on top of a VPN that is
> constructed and managed via a tun device. That hasn't worked up
> until now because ARPHRD_NONE devices (such as tun devices) have
> no GID for the RDMA/core to look up.
> 
> But it turns out that the egress device has already been picked for
> us -- no GID is necessary. addr_handler() just has to do the right
> thing with it.
> 
> Suggested-by: Jason Gunthorpe <jgg@nvidia.com>
> Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
> ---
>  drivers/infiniband/core/cma.c |   27 ++++++++++++++++++++++-----
>  1 file changed, 22 insertions(+), 5 deletions(-)
> 
> Further testing convinced me of the necessity of confirming that
> the ndev and ib_device are properly related. This version works
> on systems with multiple RDMA devices present.
> 
> 
> diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c
> index 56e568fcd32b..44ef0539957a 100644
> --- a/drivers/infiniband/core/cma.c
> +++ b/drivers/infiniband/core/cma.c
> @@ -686,30 +686,47 @@ cma_validate_port(struct ib_device *device, u32 port,
>  		  struct rdma_id_private *id_priv)
>  {
>  	struct rdma_dev_addr *dev_addr = &id_priv->id.route.addr.dev_addr;
> +	const struct ib_gid_attr *sgid_attr = ERR_PTR(-ENODEV);
>  	int bound_if_index = dev_addr->bound_dev_if;
> -	const struct ib_gid_attr *sgid_attr;
>  	int dev_type = dev_addr->dev_type;
>  	struct net_device *ndev = NULL;
>  
>  	if (!rdma_dev_access_netns(device, id_priv->id.route.addr.dev_addr.net))
> -		return ERR_PTR(-ENODEV);
> +		goto out;
> +
> +	if (rdma_protocol_iwarp(device, port)) {
> +		struct ib_device *base_dev;
> +
> +		ndev = dev_get_by_index(dev_addr->net, bound_if_index);
> +		if (!ndev)
> +			goto out;
> +		base_dev = ib_device_get_by_netdev(ndev, RDMA_DRIVER_UNKNOWN);
> +		if (base_dev)
> +			ib_device_put(base_dev);
> +		dev_put(ndev);
> +
> +		if (device == base_dev)
> +			sgid_attr = rdma_get_gid_attr(device, port, 0);
> +		goto out;
> +	}

Oy, this is kind of ugly - did you look at having the iwarp side
properly set the ndev in the sgid_attrs instead?

Then you can just check the sgid_attrs->ndev->'net && bound_if_indx' == dev_addr

Jason