[PATCH v2] RDMA/cma: Handle ARPHRD_NONE devices for iWARP

[PATCH v2] RDMA/cma: Handle ARPHRD_NONE devices for iWARP

[Chuck Lever]

From: Chuck Lever <chuck.lever@oracle.com>

We would like to enable the use of siw on top of a VPN that is
constructed and managed via a tun device. That hasn't worked up
until now because ARPHRD_NONE devices (such as tun devices) have
no GID for the RDMA/core to look up.

But it turns out that the egress device has already been picked for
us -- no GID is necessary. addr_handler() just has to do the right
thing with it.

Suggested-by: Jason Gunthorpe <jgg@nvidia.com>
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
---
 drivers/infiniband/core/cma.c |   27 ++++++++++++++++++++++-----
 1 file changed, 22 insertions(+), 5 deletions(-)

Further testing convinced me of the necessity of confirming that
the ndev and ib_device are properly related. This version works
on systems with multiple RDMA devices present.


diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c
index 56e568fcd32b..44ef0539957a 100644
--- a/drivers/infiniband/core/cma.c
+++ b/drivers/infiniband/core/cma.c
@@ -686,30 +686,47 @@ cma_validate_port(struct ib_device *device, u32 port,
 		  struct rdma_id_private *id_priv)
 {
 	struct rdma_dev_addr *dev_addr = &id_priv->id.route.addr.dev_addr;
+	const struct ib_gid_attr *sgid_attr = ERR_PTR(-ENODEV);
 	int bound_if_index = dev_addr->bound_dev_if;
-	const struct ib_gid_attr *sgid_attr;
 	int dev_type = dev_addr->dev_type;
 	struct net_device *ndev = NULL;
 
 	if (!rdma_dev_access_netns(device, id_priv->id.route.addr.dev_addr.net))
-		return ERR_PTR(-ENODEV);
+		goto out;
+
+	if (rdma_protocol_iwarp(device, port)) {
+		struct ib_device *base_dev;
+
+		ndev = dev_get_by_index(dev_addr->net, bound_if_index);
+		if (!ndev)
+			goto out;
+		base_dev = ib_device_get_by_netdev(ndev, RDMA_DRIVER_UNKNOWN);
+		if (base_dev)
+			ib_device_put(base_dev);
+		dev_put(ndev);
+
+		if (device == base_dev)
+			sgid_attr = rdma_get_gid_attr(device, port, 0);
+		goto out;
+	}
 
 	if ((dev_type == ARPHRD_INFINIBAND) && !rdma_protocol_ib(device, port))
-		return ERR_PTR(-ENODEV);
+		goto out;
 
 	if ((dev_type != ARPHRD_INFINIBAND) && rdma_protocol_ib(device, port))
-		return ERR_PTR(-ENODEV);
+		goto out;
 
 	if (dev_type == ARPHRD_ETHER && rdma_protocol_roce(device, port)) {
 		ndev = dev_get_by_index(dev_addr->net, bound_if_index);
 		if (!ndev)
-			return ERR_PTR(-ENODEV);
+			goto out;
 	} else {
 		gid_type = IB_GID_TYPE_IB;
 	}
 
 	sgid_attr = rdma_find_gid_by_port(device, gid, gid_type, port, ndev);
 	dev_put(ndev);
+out:
 	return sgid_attr;
 }
 

Re: [PATCH v2] RDMA/cma: Handle ARPHRD_NONE devices for iWARP

[Jason Gunthorpe]

On Thu, Jun 08, 2023 at 04:05:54PM -0400, Chuck Lever wrote:
> From: Chuck Lever <chuck.lever@oracle.com>
> 
> We would like to enable the use of siw on top of a VPN that is
> constructed and managed via a tun device. That hasn't worked up
> until now because ARPHRD_NONE devices (such as tun devices) have
> no GID for the RDMA/core to look up.
> 
> But it turns out that the egress device has already been picked for
> us -- no GID is necessary. addr_handler() just has to do the right
> thing with it.
> 
> Suggested-by: Jason Gunthorpe <jgg@nvidia.com>
> Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
> ---
>  drivers/infiniband/core/cma.c |   27 ++++++++++++++++++++++-----
>  1 file changed, 22 insertions(+), 5 deletions(-)
> 
> Further testing convinced me of the necessity of confirming that
> the ndev and ib_device are properly related. This version works
> on systems with multiple RDMA devices present.
> 
> 
> diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c
> index 56e568fcd32b..44ef0539957a 100644
> --- a/drivers/infiniband/core/cma.c
> +++ b/drivers/infiniband/core/cma.c
> @@ -686,30 +686,47 @@ cma_validate_port(struct ib_device *device, u32 port,
>  		  struct rdma_id_private *id_priv)
>  {
>  	struct rdma_dev_addr *dev_addr = &id_priv->id.route.addr.dev_addr;
> +	const struct ib_gid_attr *sgid_attr = ERR_PTR(-ENODEV);
>  	int bound_if_index = dev_addr->bound_dev_if;
> -	const struct ib_gid_attr *sgid_attr;
>  	int dev_type = dev_addr->dev_type;
>  	struct net_device *ndev = NULL;
>  
>  	if (!rdma_dev_access_netns(device, id_priv->id.route.addr.dev_addr.net))
> -		return ERR_PTR(-ENODEV);
> +		goto out;
> +
> +	if (rdma_protocol_iwarp(device, port)) {
> +		struct ib_device *base_dev;
> +
> +		ndev = dev_get_by_index(dev_addr->net, bound_if_index);
> +		if (!ndev)
> +			goto out;
> +		base_dev = ib_device_get_by_netdev(ndev, RDMA_DRIVER_UNKNOWN);
> +		if (base_dev)
> +			ib_device_put(base_dev);
> +		dev_put(ndev);
> +
> +		if (device == base_dev)
> +			sgid_attr = rdma_get_gid_attr(device, port, 0);
> +		goto out;
> +	}

Oy, this is kind of ugly - did you look at having the iwarp side
properly set the ndev in the sgid_attrs instead?

Then you can just check the sgid_attrs->ndev->'net && bound_if_indx' == dev_addr

Jason

Re: [PATCH v2] RDMA/cma: Handle ARPHRD_NONE devices for iWARP

[Chuck Lever III]

> On Jun 12, 2023, at 10:40 AM, Jason Gunthorpe <jgg@nvidia.com> wrote:
> 
> On Thu, Jun 08, 2023 at 04:05:54PM -0400, Chuck Lever wrote:
>> From: Chuck Lever <chuck.lever@oracle.com>
>> 
>> We would like to enable the use of siw on top of a VPN that is
>> constructed and managed via a tun device. That hasn't worked up
>> until now because ARPHRD_NONE devices (such as tun devices) have
>> no GID for the RDMA/core to look up.
>> 
>> But it turns out that the egress device has already been picked for
>> us -- no GID is necessary. addr_handler() just has to do the right
>> thing with it.
>> 
>> Suggested-by: Jason Gunthorpe <jgg@nvidia.com>
>> Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
>> ---
>> drivers/infiniband/core/cma.c |   27 ++++++++++++++++++++++-----
>> 1 file changed, 22 insertions(+), 5 deletions(-)
>> 
>> Further testing convinced me of the necessity of confirming that
>> the ndev and ib_device are properly related. This version works
>> on systems with multiple RDMA devices present.
>> 
>> 
>> diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c
>> index 56e568fcd32b..44ef0539957a 100644
>> --- a/drivers/infiniband/core/cma.c
>> +++ b/drivers/infiniband/core/cma.c
>> @@ -686,30 +686,47 @@ cma_validate_port(struct ib_device *device, u32 port,
>>  struct rdma_id_private *id_priv)
>> {
>> struct rdma_dev_addr *dev_addr = &id_priv->id.route.addr.dev_addr;
>> + const struct ib_gid_attr *sgid_attr = ERR_PTR(-ENODEV);
>> int bound_if_index = dev_addr->bound_dev_if;
>> - const struct ib_gid_attr *sgid_attr;
>> int dev_type = dev_addr->dev_type;
>> struct net_device *ndev = NULL;
>> 
>> if (!rdma_dev_access_netns(device, id_priv->id.route.addr.dev_addr.net))
>> - return ERR_PTR(-ENODEV);
>> + goto out;
>> +
>> + if (rdma_protocol_iwarp(device, port)) {
>> + struct ib_device *base_dev;
>> +
>> + ndev = dev_get_by_index(dev_addr->net, bound_if_index);
>> + if (!ndev)
>> + goto out;
>> + base_dev = ib_device_get_by_netdev(ndev, RDMA_DRIVER_UNKNOWN);
>> + if (base_dev)
>> + ib_device_put(base_dev);
>> + dev_put(ndev);
>> +
>> + if (device == base_dev)
>> + sgid_attr = rdma_get_gid_attr(device, port, 0);
>> + goto out;
>> + }
> 
> Oy, this is kind of ugly -

Yeah, not 100% elegant.


> did you look at having the iwarp side
> properly set the ndev in the sgid_attrs instead?

Interesting. I'll have a look.


> Then you can just check the sgid_attrs->ndev->'net && bound_if_indx' == dev_addr


--
Chuck Lever