Re: [PATCH 2/4] net/smc: Add netlink net namespace support

Re: [PATCH 2/4] net/smc: Add netlink net namespace support

[Dmitry V. Levin]

On Tue, Dec 28, 2021 at 09:06:10PM +0800, Tony Lu wrote:
> This adds net namespace ID to diag of linkgroup, helps us to distinguish
> different namespaces, and net_cookie is unique in the whole system.
> 
> Signed-off-by: Tony Lu <tonylu@linux.alibaba.com>
> ---
>  include/uapi/linux/smc.h      |  2 ++
>  include/uapi/linux/smc_diag.h | 11 ++++++-----
>  net/smc/smc_core.c            |  3 +++
>  net/smc/smc_diag.c            | 16 +++++++++-------
>  4 files changed, 20 insertions(+), 12 deletions(-)
> 
> diff --git a/include/uapi/linux/smc.h b/include/uapi/linux/smc.h
> index 20f33b27787f..6c2874fd2c00 100644
> --- a/include/uapi/linux/smc.h
> +++ b/include/uapi/linux/smc.h
> @@ -119,6 +119,8 @@ enum {
>  	SMC_NLA_LGR_R_CONNS_NUM,	/* u32 */
>  	SMC_NLA_LGR_R_V2_COMMON,	/* nest */
>  	SMC_NLA_LGR_R_V2,		/* nest */
> +	SMC_NLA_LGR_R_NET_COOKIE,	/* u64 */
> +	SMC_NLA_LGR_R_PAD,		/* flag */
>  	__SMC_NLA_LGR_R_MAX,
>  	SMC_NLA_LGR_R_MAX = __SMC_NLA_LGR_R_MAX - 1
>  };
> diff --git a/include/uapi/linux/smc_diag.h b/include/uapi/linux/smc_diag.h
> index 8cb3a6fef553..c7008d87f1a4 100644
> --- a/include/uapi/linux/smc_diag.h
> +++ b/include/uapi/linux/smc_diag.h
> @@ -84,11 +84,12 @@ struct smc_diag_conninfo {
>  /* SMC_DIAG_LINKINFO */
>  
>  struct smc_diag_linkinfo {
> -	__u8 link_id;			/* link identifier */
> -	__u8 ibname[IB_DEVICE_NAME_MAX]; /* name of the RDMA device */
> -	__u8 ibport;			/* RDMA device port number */
> -	__u8 gid[40];			/* local GID */
> -	__u8 peer_gid[40];		/* peer GID */
> +	__u8		link_id;		    /* link identifier */
> +	__u8		ibname[IB_DEVICE_NAME_MAX]; /* name of the RDMA device */
> +	__u8		ibport;			    /* RDMA device port number */
> +	__u8		gid[40];		    /* local GID */
> +	__u8		peer_gid[40];		    /* peer GID */
> +	__aligned_u64	net_cookie;                 /* RDMA device net namespace */
>  };
>  
>  struct smc_diag_lgrinfo {

I'm sorry but this is an ABI regression.

Since struct smc_diag_lgrinfo contains an object of type "struct smc_diag_linkinfo",
offset of all subsequent members of struct smc_diag_lgrinfo is changed by
this patch.

As result, applications compiled with the old version of struct smc_diag_linkinfo
will receive garbage in struct smc_diag_lgrinfo.role if the kernel implements
this new version of struct smc_diag_linkinfo.


-- 
ldv

Re: [PATCH 2/4] net/smc: Add netlink net namespace support

[Karsten Graul]

On 31/01/2022 01:24, Dmitry V. Levin wrote:
> On Tue, Dec 28, 2021 at 09:06:10PM +0800, Tony Lu wrote:
>> This adds net namespace ID to diag of linkgroup, helps us to distinguish
>> different namespaces, and net_cookie is unique in the whole system.
>>
> 
> I'm sorry but this is an ABI regression.
> 
> Since struct smc_diag_lgrinfo contains an object of type "struct smc_diag_linkinfo",
> offset of all subsequent members of struct smc_diag_lgrinfo is changed by
> this patch.
> 
> As result, applications compiled with the old version of struct smc_diag_linkinfo
> will receive garbage in struct smc_diag_lgrinfo.role if the kernel implements
> this new version of struct smc_diag_linkinfo.
> 

Good catch! This patch adds 2 ways to provide the net_cookie to user space, one is over the new
netlink interface, and the other is using the old smc_diag way. 
Imho to use the new netlink interface is good enough, there is no need to touch the smc_diag ABI.
We already started adding new fields to the netlink interface only, this flexibility is 
the reason why we added this interface initially.

So a patch that removes
	__aligned_u64	net_cookie;
and
	.lnk[0].net_cookie = net->net_cookie,
should solve the issue. 

Thoughts?

[PATCH] Partially revert "net/smc: Add netlink net namespace support"

[Dmitry V. Levin]

The change of sizeof(struct smc_diag_linkinfo) by commit 79d39fc503b4
("net/smc: Add netlink net namespace support") introduced an ABI
regression: since struct smc_diag_lgrinfo contains an object of
type "struct smc_diag_linkinfo", offset of all subsequent members
of struct smc_diag_lgrinfo was changed by that change.

As result, applications compiled with the old version
of struct smc_diag_linkinfo will receive garbage in
struct smc_diag_lgrinfo.role if the kernel implements
this new version of struct smc_diag_linkinfo.

Fix this regression by reverting the part of commit 79d39fc503b4 that
changes struct smc_diag_linkinfo.  After all, there is SMC_GEN_NETLINK
interface which is good enough, so there is probably no need to touch
the smc_diag ABI in the first place.

Fixes: 79d39fc503b4 ("net/smc: Add netlink net namespace support")
Signed-off-by: Dmitry V. Levin <ldv@altlinux.org>
---
 include/uapi/linux/smc_diag.h | 11 +++++------
 net/smc/smc_diag.c            |  2 --
 2 files changed, 5 insertions(+), 8 deletions(-)

diff --git a/include/uapi/linux/smc_diag.h b/include/uapi/linux/smc_diag.h
index c7008d87f1a4..8cb3a6fef553 100644
--- a/include/uapi/linux/smc_diag.h
+++ b/include/uapi/linux/smc_diag.h
@@ -84,12 +84,11 @@ struct smc_diag_conninfo {
 /* SMC_DIAG_LINKINFO */
 
 struct smc_diag_linkinfo {
-	__u8		link_id;		    /* link identifier */
-	__u8		ibname[IB_DEVICE_NAME_MAX]; /* name of the RDMA device */
-	__u8		ibport;			    /* RDMA device port number */
-	__u8		gid[40];		    /* local GID */
-	__u8		peer_gid[40];		    /* peer GID */
-	__aligned_u64	net_cookie;                 /* RDMA device net namespace */
+	__u8 link_id;			/* link identifier */
+	__u8 ibname[IB_DEVICE_NAME_MAX]; /* name of the RDMA device */
+	__u8 ibport;			/* RDMA device port number */
+	__u8 gid[40];			/* local GID */
+	__u8 peer_gid[40];		/* peer GID */
 };
 
 struct smc_diag_lgrinfo {
diff --git a/net/smc/smc_diag.c b/net/smc/smc_diag.c
index b8898c787d23..1fca2f90a9c7 100644
--- a/net/smc/smc_diag.c
+++ b/net/smc/smc_diag.c
@@ -146,13 +146,11 @@ static int __smc_diag_dump(struct sock *sk, struct sk_buff *skb,
 	    (req->diag_ext & (1 << (SMC_DIAG_LGRINFO - 1))) &&
 	    !list_empty(&smc->conn.lgr->list)) {
 		struct smc_link *link = smc->conn.lnk;
-		struct net *net = read_pnet(&link->smcibdev->ibdev->coredev.rdma_net);
 
 		struct smc_diag_lgrinfo linfo = {
 			.role = smc->conn.lgr->role,
 			.lnk[0].ibport = link->ibport,
 			.lnk[0].link_id = link->link_id,
-			.lnk[0].net_cookie = net->net_cookie,
 		};
 
 		memcpy(linfo.lnk[0].ibname,
-- 
ldv

Re: [PATCH] Partially revert "net/smc: Add netlink net namespace support"

[Karsten Graul]

On 02/02/2022 04:09, Dmitry V. Levin wrote:
> The change of sizeof(struct smc_diag_linkinfo) by commit 79d39fc503b4
> ("net/smc: Add netlink net namespace support") introduced an ABI
> regression: since struct smc_diag_lgrinfo contains an object of
> type "struct smc_diag_linkinfo", offset of all subsequent members
> of struct smc_diag_lgrinfo was changed by that change.
> 
> As result, applications compiled with the old version
> of struct smc_diag_linkinfo will receive garbage in
> struct smc_diag_lgrinfo.role if the kernel implements
> this new version of struct smc_diag_linkinfo.
> 
> Fix this regression by reverting the part of commit 79d39fc503b4 that
> changes struct smc_diag_linkinfo.  After all, there is SMC_GEN_NETLINK
> interface which is good enough, so there is probably no need to touch
> the smc_diag ABI in the first place.

Reviewed-by: Karsten Graul <kgraul@linux.ibm.com>

Thank you Dmitry.

Re: [PATCH] Partially revert "net/smc: Add netlink net namespace support"

[Tony Lu]

On Wed, Feb 02, 2022 at 06:09:04AM +0300, Dmitry V. Levin wrote:
> The change of sizeof(struct smc_diag_linkinfo) by commit 79d39fc503b4
> ("net/smc: Add netlink net namespace support") introduced an ABI
> regression: since struct smc_diag_lgrinfo contains an object of
> type "struct smc_diag_linkinfo", offset of all subsequent members
> of struct smc_diag_lgrinfo was changed by that change.
> 
> As result, applications compiled with the old version
> of struct smc_diag_linkinfo will receive garbage in
> struct smc_diag_lgrinfo.role if the kernel implements
> this new version of struct smc_diag_linkinfo.
> 
> Fix this regression by reverting the part of commit 79d39fc503b4 that
> changes struct smc_diag_linkinfo.  After all, there is SMC_GEN_NETLINK
> interface which is good enough, so there is probably no need to touch
> the smc_diag ABI in the first place.
> 
> Fixes: 79d39fc503b4 ("net/smc: Add netlink net namespace support")
> Signed-off-by: Dmitry V. Levin <ldv@altlinux.org>

Thank you and Karsten.

It was my negligence that caused the ABI incompatibility issue.
I will consider to fix it completely. And we are starting to build
smc-tools and other userspace test for potential ABI modifications.

Best regards,
Tony Lu