[PATCH net v3 0/2] net: tcp: lookup the best matched listen socket

[PATCH net v3 0/2] net: tcp: lookup the best matched listen socket

[Menglong Dong]

For now, the tcp socket lookup will terminate if the socket is reuse port
in inet_lhash2_lookup(), which makes the socket is not the best match.

For example, we have socket1 and socket2 both listen on "0.0.0.0:1234",
but socket1 bind on "eth0". We create socket1 first, and then socket2.
Then, all connections will goto socket2, which is not expected, as socket1
has higher priority.

The 1st patch fix this problem, and the 2nd patch is a selftests for this
problem. Without the 1st patch, the selftests will fail with:

$ ./tcp_reuseport.py
TAP version 13
1..1
FAIL: wrong assignment
not ok 1 tcp_reuseport.test_reuseport_select
Totals: pass:0 fail:1 xfail:0 xpass:0 skip:0 error:0

With the 1st patch, it will success:
$ ./tcp_reuseport.py
TAP version 13
1..1
SUCCESS: assigned properly: (<socket.socket fd=6, family=2, type=1, proto=0, laddr=('127.0.0.1', 33787), raddr=('127.0.0.1', 43140)>, ('127.0.0.1', 43140))
SUCCESS: assigned properly: (<socket.socket fd=5, family=2, type=1, proto=0, laddr=('127.0.0.1', 33787), raddr=('127.0.0.1', 43146)>, ('127.0.0.1', 43146))
SUCCESS: assigned properly: (<socket.socket fd=6, family=2, type=1, proto=0, laddr=('127.0.0.1', 33787), raddr=('127.0.0.1', 43162)>, ('127.0.0.1', 43162))
ok 1 tcp_reuseport.test_reuseport_select
Totals: pass:1 fail:0 xfail:0 xpass:0 skip:0 error:0

Changes since V2:
* use the approach in V1
* add the Fixes tag in the 1st patch
* introduce the selftests

Menglong Dong (2):
  net: tcp: lookup the best matched listen socket
  selftests/net: test TCP reuseport socket selection

 net/ipv4/inet_hashtables.c                   | 13 +++----
 net/ipv6/inet6_hashtables.c                  | 13 +++----
 tools/testing/selftests/net/Makefile         |  1 +
 tools/testing/selftests/net/tcp_reuseport.py | 36 ++++++++++++++++++++
 4 files changed, 51 insertions(+), 12 deletions(-)
 create mode 100755 tools/testing/selftests/net/tcp_reuseport.py

-- 
2.50.1

[PATCH net v3 1/2] net: tcp: lookup the best matched listen socket

[Menglong Dong]

For now, the tcp socket lookup will terminate if the socket is reuse port
in inet_lhash2_lookup(), which makes the socket is not the best match.

For example, we have socket1 and socket2 both listen on "0.0.0.0:1234",
but socket1 bind on "eth0". We create socket1 first, and then socket2.
Then, all connections will goto socket2, which is not expected, as socket1
has higher priority.

This can cause unexpected behavior if TCP MD5 keys is used, as described
in Documentation/networking/vrf.rst -> Applications.

Therefor, we lookup the best matched socket first, and then do the reuse
port logic. This can increase some overhead if there are many reuse port
socket :/

Fixes: c125e80b8868 ("soreuseport: fast reuseport TCP socket selection")
Signed-off-by: Menglong Dong <dongml2@chinatelecom.cn>
---
v3:
* use the approach in V1
* add the Fixes tag
---
 net/ipv4/inet_hashtables.c  | 13 +++++++------
 net/ipv6/inet6_hashtables.c | 13 +++++++------
 2 files changed, 14 insertions(+), 12 deletions(-)

diff --git a/net/ipv4/inet_hashtables.c b/net/ipv4/inet_hashtables.c
index ceeeec9b7290..51751337f394 100644
--- a/net/ipv4/inet_hashtables.c
+++ b/net/ipv4/inet_hashtables.c
@@ -389,17 +389,18 @@ static struct sock *inet_lhash2_lookup(const struct net *net,
 	sk_nulls_for_each_rcu(sk, node, &ilb2->nulls_head) {
 		score = compute_score(sk, net, hnum, daddr, dif, sdif);
 		if (score > hiscore) {
-			result = inet_lookup_reuseport(net, sk, skb, doff,
-						       saddr, sport, daddr, hnum, inet_ehashfn);
-			if (result)
-				return result;
-
 			result = sk;
 			hiscore = score;
 		}
 	}
 
-	return result;
+	if (!result)
+		return NULL;
+
+	sk = inet_lookup_reuseport(net, result, skb, doff,
+				   saddr, sport, daddr, hnum, inet_ehashfn);
+
+	return sk ? sk : result;
 }
 
 struct sock *inet_lookup_run_sk_lookup(const struct net *net,
diff --git a/net/ipv6/inet6_hashtables.c b/net/ipv6/inet6_hashtables.c
index 76ee521189eb..2554f26d6c20 100644
--- a/net/ipv6/inet6_hashtables.c
+++ b/net/ipv6/inet6_hashtables.c
@@ -161,17 +161,18 @@ static struct sock *inet6_lhash2_lookup(const struct net *net,
 	sk_nulls_for_each_rcu(sk, node, &ilb2->nulls_head) {
 		score = compute_score(sk, net, hnum, daddr, dif, sdif);
 		if (score > hiscore) {
-			result = inet6_lookup_reuseport(net, sk, skb, doff,
-							saddr, sport, daddr, hnum, inet6_ehashfn);
-			if (result)
-				return result;
-
 			result = sk;
 			hiscore = score;
 		}
 	}
 
-	return result;
+	if (!result)
+		return NULL;
+
+	sk = inet6_lookup_reuseport(net, result, skb, doff,
+				    saddr, sport, daddr, hnum, inet6_ehashfn);
+
+	return sk ? sk : result;
 }
 
 struct sock *inet6_lookup_run_sk_lookup(const struct net *net,
-- 
2.50.1

[PATCH net v3 2/2] selftests/net: test TCP reuseport socket selection

[Menglong Dong]

The test script is provided by Kuniyuki in [1], which is used to test the
selection of the TCP reuseport socket problem.

Link: https://lore.kernel.org/netdev/20250801040757.1599996-1-kuniyu@google.com/ [1]
Signed-off-by: Menglong Dong <dongml2@chinatelecom.cn>
---
 tools/testing/selftests/net/Makefile         |  1 +
 tools/testing/selftests/net/tcp_reuseport.py | 36 ++++++++++++++++++++
 2 files changed, 37 insertions(+)
 create mode 100755 tools/testing/selftests/net/tcp_reuseport.py

diff --git a/tools/testing/selftests/net/Makefile b/tools/testing/selftests/net/Makefile
index b31a71f2b372..0f4c3eea9709 100644
--- a/tools/testing/selftests/net/Makefile
+++ b/tools/testing/selftests/net/Makefile
@@ -117,6 +117,7 @@ TEST_GEN_FILES += tfo
 TEST_PROGS += tfo_passive.sh
 TEST_PROGS += broadcast_pmtu.sh
 TEST_PROGS += ipv6_force_forwarding.sh
+TEST_PROGS += tcp_reuseport.py
 
 # YNL files, must be before "include ..lib.mk"
 YNL_GEN_FILES := busy_poller netlink-dumps
diff --git a/tools/testing/selftests/net/tcp_reuseport.py b/tools/testing/selftests/net/tcp_reuseport.py
new file mode 100755
index 000000000000..eaeb7096382e
--- /dev/null
+++ b/tools/testing/selftests/net/tcp_reuseport.py
@@ -0,0 +1,36 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: GPL-2.0
+
+import os
+
+from lib.py import ksft_run, ksft_exit
+from socket import *
+
+def test_reuseport_select() -> None:
+    s1 = socket()
+    s1.setsockopt(SOL_SOCKET, SO_REUSEPORT, 1)
+    s1.setsockopt(SOL_SOCKET, SO_BINDTODEVICE, b'lo')
+    s1.listen()
+    s1.setblocking(False)
+
+    s2 = socket()
+    s2.setsockopt(SOL_SOCKET, SO_REUSEPORT, 1)
+    s2.bind(s1.getsockname())
+    s2.listen()
+    s2.setblocking(False)
+
+    for i in range(3):
+        c = socket()
+        c.connect(s1.getsockname())
+        try:
+            print("SUCCESS: assigned properly:", s1.accept())
+        except:
+            print("FAIL: wrong assignment")
+            os.sys.exit(1)
+
+def main() -> None:
+    ksft_run([test_reuseport_select])
+    ksft_exit()
+
+if __name__ == "__main__":
+    main()
-- 
2.50.1

Re: [PATCH net v3 1/2] net: tcp: lookup the best matched listen socket

[Jason Xing]

Hi Menglong,

On Sat, Aug 2, 2025 at 5:28 PM Menglong Dong <menglong8.dong@gmail.com> wrote:
>
> For now, the tcp socket lookup will terminate if the socket is reuse port
> in inet_lhash2_lookup(), which makes the socket is not the best match.
>
> For example, we have socket1 and socket2 both listen on "0.0.0.0:1234",
> but socket1 bind on "eth0". We create socket1 first, and then socket2.
> Then, all connections will goto socket2, which is not expected, as socket1
> has higher priority.
>
> This can cause unexpected behavior if TCP MD5 keys is used, as described
> in Documentation/networking/vrf.rst -> Applications.
>
> Therefor, we lookup the best matched socket first, and then do the reuse

s/Therefor/Therefore

> port logic. This can increase some overhead if there are many reuse port
> socket :/
>
> Fixes: c125e80b8868 ("soreuseport: fast reuseport TCP socket selection")
> Signed-off-by: Menglong Dong <dongml2@chinatelecom.cn>
> ---
> v3:
> * use the approach in V1
> * add the Fixes tag
> ---
>  net/ipv4/inet_hashtables.c  | 13 +++++++------
>  net/ipv6/inet6_hashtables.c | 13 +++++++------
>  2 files changed, 14 insertions(+), 12 deletions(-)
>
> diff --git a/net/ipv4/inet_hashtables.c b/net/ipv4/inet_hashtables.c
> index ceeeec9b7290..51751337f394 100644
> --- a/net/ipv4/inet_hashtables.c
> +++ b/net/ipv4/inet_hashtables.c
> @@ -389,17 +389,18 @@ static struct sock *inet_lhash2_lookup(const struct net *net,
>         sk_nulls_for_each_rcu(sk, node, &ilb2->nulls_head) {
>                 score = compute_score(sk, net, hnum, daddr, dif, sdif);
>                 if (score > hiscore) {
> -                       result = inet_lookup_reuseport(net, sk, skb, doff,
> -                                                      saddr, sport, daddr, hnum, inet_ehashfn);
> -                       if (result)
> -                               return result;
> -
>                         result = sk;
>                         hiscore = score;
>                 }
>         }
>
> -       return result;
> +       if (!result)
> +               return NULL;
> +
> +       sk = inet_lookup_reuseport(net, result, skb, doff,
> +                                  saddr, sport, daddr, hnum, inet_ehashfn);
> +
> +       return sk ? sk : result;
>  }

IMHO, I don't see it as a bugfix. So can you elaborate on what the exact
side effect you're faced with is when the algorithm finally prefers
socket2 (without
this patch)?

AFAIK, the current approach breaks the initial design and might make
the whole lookup process take a longer time in certain cases like you mentioned.

Thanks,
Jason

Re: [PATCH net v3 1/2] net: tcp: lookup the best matched listen socket

[Menglong Dong]

On Sat, Aug 2, 2025 at 9:06 PM Jason Xing <kerneljasonxing@gmail.com> wrote:
>
> Hi Menglong,
>
> On Sat, Aug 2, 2025 at 5:28 PM Menglong Dong <menglong8.dong@gmail.com> wrote:
> >
> > For now, the tcp socket lookup will terminate if the socket is reuse port
> > in inet_lhash2_lookup(), which makes the socket is not the best match.
> >
> > For example, we have socket1 and socket2 both listen on "0.0.0.0:1234",
> > but socket1 bind on "eth0". We create socket1 first, and then socket2.
> > Then, all connections will goto socket2, which is not expected, as socket1
> > has higher priority.
> >
> > This can cause unexpected behavior if TCP MD5 keys is used, as described
> > in Documentation/networking/vrf.rst -> Applications.
> >
> > Therefor, we lookup the best matched socket first, and then do the reuse
>
> s/Therefor/Therefore
>
> > port logic. This can increase some overhead if there are many reuse port
> > socket :/
> >
> > Fixes: c125e80b8868 ("soreuseport: fast reuseport TCP socket selection")
> > Signed-off-by: Menglong Dong <dongml2@chinatelecom.cn>
> > ---
> > v3:
> > * use the approach in V1
> > * add the Fixes tag
> > ---
> >  net/ipv4/inet_hashtables.c  | 13 +++++++------
> >  net/ipv6/inet6_hashtables.c | 13 +++++++------
> >  2 files changed, 14 insertions(+), 12 deletions(-)
> >
> > diff --git a/net/ipv4/inet_hashtables.c b/net/ipv4/inet_hashtables.c
> > index ceeeec9b7290..51751337f394 100644
> > --- a/net/ipv4/inet_hashtables.c
> > +++ b/net/ipv4/inet_hashtables.c
> > @@ -389,17 +389,18 @@ static struct sock *inet_lhash2_lookup(const struct net *net,
> >         sk_nulls_for_each_rcu(sk, node, &ilb2->nulls_head) {
> >                 score = compute_score(sk, net, hnum, daddr, dif, sdif);
> >                 if (score > hiscore) {
> > -                       result = inet_lookup_reuseport(net, sk, skb, doff,
> > -                                                      saddr, sport, daddr, hnum, inet_ehashfn);
> > -                       if (result)
> > -                               return result;
> > -
> >                         result = sk;
> >                         hiscore = score;
> >                 }
> >         }
> >
> > -       return result;
> > +       if (!result)
> > +               return NULL;
> > +
> > +       sk = inet_lookup_reuseport(net, result, skb, doff,
> > +                                  saddr, sport, daddr, hnum, inet_ehashfn);
> > +
> > +       return sk ? sk : result;
> >  }
>
> IMHO, I don't see it as a bugfix. So can you elaborate on what the exact
> side effect you're faced with is when the algorithm finally prefers
> socket2 (without
> this patch)?

Hi, Jason. The case is that the user has several NIC,
and there are some sockets that are binded to them,
who listen on TCP port 6666. And a global socket doesn't
bind any NIC and listens on TCP port 6666.

In theory, the connection request from the NIC will goto
the listen socket that is binded on it. When on socket
is binded on the NIC, it goto the global socket. However,
the connection request always goto the global socket, which
is not expected.

What's worse is that when TCP MD5 is used on the socket,
the connection will fail :/

More discussion can be found here:
https://lore.kernel.org/netdev/20250801090949.129941-1-dongml2@chinatelecom.cn/
https://lore.kernel.org/netdev/20250731123309.184496-1-dongml2@chinatelecom.cn/

>
> AFAIK, the current approach breaks the initial design and might make
> the whole lookup process take a longer time in certain cases like you mentioned.

Kuniyuki said he will post some patch when the net-next is
open, which reorder the socket in __inet_hash() to make
the reuseport socket lookup keep O(1).

So this patch is more like a temporary solution :/

Thanks!
Menglong Dong

>
> Thanks,
> Jason

Re: [PATCH net v3 1/2] net: tcp: lookup the best matched listen socket

[Jason Xing]

On Sun, Aug 3, 2025 at 9:59 AM Menglong Dong <menglong8.dong@gmail.com> wrote:
>
> On Sat, Aug 2, 2025 at 9:06 PM Jason Xing <kerneljasonxing@gmail.com> wrote:
> >
> > Hi Menglong,
> >
> > On Sat, Aug 2, 2025 at 5:28 PM Menglong Dong <menglong8.dong@gmail.com> wrote:
> > >
> > > For now, the tcp socket lookup will terminate if the socket is reuse port
> > > in inet_lhash2_lookup(), which makes the socket is not the best match.
> > >
> > > For example, we have socket1 and socket2 both listen on "0.0.0.0:1234",
> > > but socket1 bind on "eth0". We create socket1 first, and then socket2.
> > > Then, all connections will goto socket2, which is not expected, as socket1
> > > has higher priority.
> > >
> > > This can cause unexpected behavior if TCP MD5 keys is used, as described
> > > in Documentation/networking/vrf.rst -> Applications.
> > >
> > > Therefor, we lookup the best matched socket first, and then do the reuse
> >
> > s/Therefor/Therefore
> >
> > > port logic. This can increase some overhead if there are many reuse port
> > > socket :/
> > >
> > > Fixes: c125e80b8868 ("soreuseport: fast reuseport TCP socket selection")
> > > Signed-off-by: Menglong Dong <dongml2@chinatelecom.cn>
> > > ---
> > > v3:
> > > * use the approach in V1
> > > * add the Fixes tag
> > > ---
> > >  net/ipv4/inet_hashtables.c  | 13 +++++++------
> > >  net/ipv6/inet6_hashtables.c | 13 +++++++------
> > >  2 files changed, 14 insertions(+), 12 deletions(-)
> > >
> > > diff --git a/net/ipv4/inet_hashtables.c b/net/ipv4/inet_hashtables.c
> > > index ceeeec9b7290..51751337f394 100644
> > > --- a/net/ipv4/inet_hashtables.c
> > > +++ b/net/ipv4/inet_hashtables.c
> > > @@ -389,17 +389,18 @@ static struct sock *inet_lhash2_lookup(const struct net *net,
> > >         sk_nulls_for_each_rcu(sk, node, &ilb2->nulls_head) {
> > >                 score = compute_score(sk, net, hnum, daddr, dif, sdif);
> > >                 if (score > hiscore) {
> > > -                       result = inet_lookup_reuseport(net, sk, skb, doff,
> > > -                                                      saddr, sport, daddr, hnum, inet_ehashfn);
> > > -                       if (result)
> > > -                               return result;
> > > -
> > >                         result = sk;
> > >                         hiscore = score;
> > >                 }
> > >         }
> > >
> > > -       return result;
> > > +       if (!result)
> > > +               return NULL;
> > > +
> > > +       sk = inet_lookup_reuseport(net, result, skb, doff,
> > > +                                  saddr, sport, daddr, hnum, inet_ehashfn);
> > > +
> > > +       return sk ? sk : result;
> > >  }
> >
> > IMHO, I don't see it as a bugfix. So can you elaborate on what the exact
> > side effect you're faced with is when the algorithm finally prefers
> > socket2 (without
> > this patch)?
>
> Hi, Jason. The case is that the user has several NIC,
> and there are some sockets that are binded to them,
> who listen on TCP port 6666. And a global socket doesn't
> bind any NIC and listens on TCP port 6666.
>
> In theory, the connection request from the NIC will goto
> the listen socket that is binded on it. When on socket
> is binded on the NIC, it goto the global socket. However,
> the connection request always goto the global socket, which
> is not expected.
>
> What's worse is that when TCP MD5 is used on the socket,
> the connection will fail :/

I'm trying to picture what the usage can be in the userland as you
pointed out in the MD5 case. As to the client side, it seems weird
since it cannot detect and know the priority of the other side where a
few sockets listen on the same address and port.

I'm not saying the priority problem doesn't exist, just not knowing
how severe the case could be. It doesn't look that bad at least until
now. Only the selection policy itself matters more to the server side
than to the client side.

Thanks,
Jason

Re: [PATCH net v3 1/2] net: tcp: lookup the best matched listen socket

[Menglong Dong]

On Sun, Aug 3, 2025 at 10:54 AM Jason Xing <kerneljasonxing@gmail.com> wrote:
>
> On Sun, Aug 3, 2025 at 9:59 AM Menglong Dong <menglong8.dong@gmail.com> wrote:
> >
> > On Sat, Aug 2, 2025 at 9:06 PM Jason Xing <kerneljasonxing@gmail.com> wrote:
> > >
> > > Hi Menglong,
> > >
> > > On Sat, Aug 2, 2025 at 5:28 PM Menglong Dong <menglong8.dong@gmail.com> wrote:
> > > >
> > > > For now, the tcp socket lookup will terminate if the socket is reuse port
> > > > in inet_lhash2_lookup(), which makes the socket is not the best match.
> > > >
> > > > For example, we have socket1 and socket2 both listen on "0.0.0.0:1234",
> > > > but socket1 bind on "eth0". We create socket1 first, and then socket2.
> > > > Then, all connections will goto socket2, which is not expected, as socket1
> > > > has higher priority.
> > > >
> > > > This can cause unexpected behavior if TCP MD5 keys is used, as described
> > > > in Documentation/networking/vrf.rst -> Applications.
> > > >
> > > > Therefor, we lookup the best matched socket first, and then do the reuse
> > >
> > > s/Therefor/Therefore
> > >
> > > > port logic. This can increase some overhead if there are many reuse port
> > > > socket :/
> > > >
> > > > Fixes: c125e80b8868 ("soreuseport: fast reuseport TCP socket selection")
> > > > Signed-off-by: Menglong Dong <dongml2@chinatelecom.cn>
> > > > ---
> > > > v3:
> > > > * use the approach in V1
> > > > * add the Fixes tag
> > > > ---
> > > >  net/ipv4/inet_hashtables.c  | 13 +++++++------
> > > >  net/ipv6/inet6_hashtables.c | 13 +++++++------
> > > >  2 files changed, 14 insertions(+), 12 deletions(-)
> > > >
> > > > diff --git a/net/ipv4/inet_hashtables.c b/net/ipv4/inet_hashtables.c
> > > > index ceeeec9b7290..51751337f394 100644
> > > > --- a/net/ipv4/inet_hashtables.c
> > > > +++ b/net/ipv4/inet_hashtables.c
> > > > @@ -389,17 +389,18 @@ static struct sock *inet_lhash2_lookup(const struct net *net,
> > > >         sk_nulls_for_each_rcu(sk, node, &ilb2->nulls_head) {
> > > >                 score = compute_score(sk, net, hnum, daddr, dif, sdif);
> > > >                 if (score > hiscore) {
> > > > -                       result = inet_lookup_reuseport(net, sk, skb, doff,
> > > > -                                                      saddr, sport, daddr, hnum, inet_ehashfn);
> > > > -                       if (result)
> > > > -                               return result;
> > > > -
> > > >                         result = sk;
> > > >                         hiscore = score;
> > > >                 }
> > > >         }
> > > >
> > > > -       return result;
> > > > +       if (!result)
> > > > +               return NULL;
> > > > +
> > > > +       sk = inet_lookup_reuseport(net, result, skb, doff,
> > > > +                                  saddr, sport, daddr, hnum, inet_ehashfn);
> > > > +
> > > > +       return sk ? sk : result;
> > > >  }
> > >
> > > IMHO, I don't see it as a bugfix. So can you elaborate on what the exact
> > > side effect you're faced with is when the algorithm finally prefers
> > > socket2 (without
> > > this patch)?
> >
> > Hi, Jason. The case is that the user has several NIC,
> > and there are some sockets that are binded to them,
> > who listen on TCP port 6666. And a global socket doesn't
> > bind any NIC and listens on TCP port 6666.
> >
> > In theory, the connection request from the NIC will goto
> > the listen socket that is binded on it. When on socket
> > is binded on the NIC, it goto the global socket. However,
> > the connection request always goto the global socket, which
> > is not expected.
> >
> > What's worse is that when TCP MD5 is used on the socket,
> > the connection will fail :/
>
> I'm trying to picture what the usage can be in the userland as you
> pointed out in the MD5 case. As to the client side, it seems weird
> since it cannot detect and know the priority of the other side where a
> few sockets listen on the same address and port.

For the server side, it needs to add all the clients information
with the TCP_MD5SIG option. For socket1 that binded on the
eth0, it will add all the client addresses that come from eth0 to the
socket1 with TCP_MD5SIG. So the server knows the clients.

And in my use case, the TCP MD5 + VRF is used. The details are
a little fuzzy for me, which I need to do some recalling :/

>
> I'm not saying the priority problem doesn't exist, just not knowing
> how severe the case could be. It doesn't look that bad at least until
> now. Only the selection policy itself matters more to the server side
> than to the client side.
>
> Thanks,
> Jason

Re: [PATCH net v3 1/2] net: tcp: lookup the best matched listen socket

[Jason Xing]

On Sun, Aug 3, 2025 at 12:00 PM Menglong Dong <menglong8.dong@gmail.com> wrote:
>
> On Sun, Aug 3, 2025 at 10:54 AM Jason Xing <kerneljasonxing@gmail.com> wrote:
> >
> > On Sun, Aug 3, 2025 at 9:59 AM Menglong Dong <menglong8.dong@gmail.com> wrote:
> > >
> > > On Sat, Aug 2, 2025 at 9:06 PM Jason Xing <kerneljasonxing@gmail.com> wrote:
> > > >
> > > > Hi Menglong,
> > > >
> > > > On Sat, Aug 2, 2025 at 5:28 PM Menglong Dong <menglong8.dong@gmail.com> wrote:
> > > > >
> > > > > For now, the tcp socket lookup will terminate if the socket is reuse port
> > > > > in inet_lhash2_lookup(), which makes the socket is not the best match.
> > > > >
> > > > > For example, we have socket1 and socket2 both listen on "0.0.0.0:1234",
> > > > > but socket1 bind on "eth0". We create socket1 first, and then socket2.
> > > > > Then, all connections will goto socket2, which is not expected, as socket1
> > > > > has higher priority.
> > > > >
> > > > > This can cause unexpected behavior if TCP MD5 keys is used, as described
> > > > > in Documentation/networking/vrf.rst -> Applications.
> > > > >
> > > > > Therefor, we lookup the best matched socket first, and then do the reuse
> > > >
> > > > s/Therefor/Therefore
> > > >
> > > > > port logic. This can increase some overhead if there are many reuse port
> > > > > socket :/
> > > > >
> > > > > Fixes: c125e80b8868 ("soreuseport: fast reuseport TCP socket selection")
> > > > > Signed-off-by: Menglong Dong <dongml2@chinatelecom.cn>
> > > > > ---
> > > > > v3:
> > > > > * use the approach in V1
> > > > > * add the Fixes tag
> > > > > ---
> > > > >  net/ipv4/inet_hashtables.c  | 13 +++++++------
> > > > >  net/ipv6/inet6_hashtables.c | 13 +++++++------
> > > > >  2 files changed, 14 insertions(+), 12 deletions(-)
> > > > >
> > > > > diff --git a/net/ipv4/inet_hashtables.c b/net/ipv4/inet_hashtables.c
> > > > > index ceeeec9b7290..51751337f394 100644
> > > > > --- a/net/ipv4/inet_hashtables.c
> > > > > +++ b/net/ipv4/inet_hashtables.c
> > > > > @@ -389,17 +389,18 @@ static struct sock *inet_lhash2_lookup(const struct net *net,
> > > > >         sk_nulls_for_each_rcu(sk, node, &ilb2->nulls_head) {
> > > > >                 score = compute_score(sk, net, hnum, daddr, dif, sdif);
> > > > >                 if (score > hiscore) {
> > > > > -                       result = inet_lookup_reuseport(net, sk, skb, doff,
> > > > > -                                                      saddr, sport, daddr, hnum, inet_ehashfn);
> > > > > -                       if (result)
> > > > > -                               return result;
> > > > > -
> > > > >                         result = sk;
> > > > >                         hiscore = score;
> > > > >                 }
> > > > >         }
> > > > >
> > > > > -       return result;
> > > > > +       if (!result)
> > > > > +               return NULL;
> > > > > +
> > > > > +       sk = inet_lookup_reuseport(net, result, skb, doff,
> > > > > +                                  saddr, sport, daddr, hnum, inet_ehashfn);
> > > > > +
> > > > > +       return sk ? sk : result;
> > > > >  }
> > > >
> > > > IMHO, I don't see it as a bugfix. So can you elaborate on what the exact
> > > > side effect you're faced with is when the algorithm finally prefers
> > > > socket2 (without
> > > > this patch)?
> > >
> > > Hi, Jason. The case is that the user has several NIC,
> > > and there are some sockets that are binded to them,
> > > who listen on TCP port 6666. And a global socket doesn't
> > > bind any NIC and listens on TCP port 6666.
> > >
> > > In theory, the connection request from the NIC will goto
> > > the listen socket that is binded on it. When on socket
> > > is binded on the NIC, it goto the global socket. However,
> > > the connection request always goto the global socket, which
> > > is not expected.
> > >
> > > What's worse is that when TCP MD5 is used on the socket,
> > > the connection will fail :/
> >
> > I'm trying to picture what the usage can be in the userland as you
> > pointed out in the MD5 case. As to the client side, it seems weird
> > since it cannot detect and know the priority of the other side where a
> > few sockets listen on the same address and port.
>
> For the server side, it needs to add all the clients information
> with the TCP_MD5SIG option. For socket1 that binded on the
> eth0, it will add all the client addresses that come from eth0 to the
> socket1 with TCP_MD5SIG. So the server knows the clients.

Right, but I meant the _client_ side doesn't know the details of the
other side, that is to say, the server side needs to keep sure the
client server easily connects to your preferred listener regardless of
what the selection algorithm looks like. In terms of what you
depicted, I see your point. One question is if the kernel or API
provides any rules and instructions to the users that on the server
side the sk1 must be selected in your case? Is it possible for other
cases where the sk2 is the preferable/ideal one? I'm not sure if it's
worth 'fixing' it in the kernel.

Thanks,
Jason

Re: [PATCH net v3 2/2] selftests/net: test TCP reuseport socket selection

[Kuniyuki Iwashima]

On Sat, Aug 2, 2025 at 2:24 AM Menglong Dong <menglong8.dong@gmail.com> wrote:
>
> The test script is provided by Kuniyuki in [1], which is used to test the
> selection of the TCP reuseport socket problem.
>
> Link: https://lore.kernel.org/netdev/20250801040757.1599996-1-kuniyu@google.com/ [1]
> Signed-off-by: Menglong Dong <dongml2@chinatelecom.cn>
> ---
>  tools/testing/selftests/net/Makefile         |  1 +
>  tools/testing/selftests/net/tcp_reuseport.py | 36 ++++++++++++++++++++
>  2 files changed, 37 insertions(+)
>  create mode 100755 tools/testing/selftests/net/tcp_reuseport.py
>
> diff --git a/tools/testing/selftests/net/Makefile b/tools/testing/selftests/net/Makefile
> index b31a71f2b372..0f4c3eea9709 100644
> --- a/tools/testing/selftests/net/Makefile
> +++ b/tools/testing/selftests/net/Makefile
> @@ -117,6 +117,7 @@ TEST_GEN_FILES += tfo
>  TEST_PROGS += tfo_passive.sh
>  TEST_PROGS += broadcast_pmtu.sh
>  TEST_PROGS += ipv6_force_forwarding.sh
> +TEST_PROGS += tcp_reuseport.py
>
>  # YNL files, must be before "include ..lib.mk"
>  YNL_GEN_FILES := busy_poller netlink-dumps
> diff --git a/tools/testing/selftests/net/tcp_reuseport.py b/tools/testing/selftests/net/tcp_reuseport.py
> new file mode 100755
> index 000000000000..eaeb7096382e
> --- /dev/null
> +++ b/tools/testing/selftests/net/tcp_reuseport.py
> @@ -0,0 +1,36 @@
> +#!/usr/bin/env python3
> +# SPDX-License-Identifier: GPL-2.0
> +
> +import os
> +
> +from lib.py import ksft_run, ksft_exit
> +from socket import *
> +
> +def test_reuseport_select() -> None:
> +    s1 = socket()
> +    s1.setsockopt(SOL_SOCKET, SO_REUSEPORT, 1)
> +    s1.setsockopt(SOL_SOCKET, SO_BINDTODEVICE, b'lo')
> +    s1.listen()
> +    s1.setblocking(False)
> +
> +    s2 = socket()
> +    s2.setsockopt(SOL_SOCKET, SO_REUSEPORT, 1)
> +    s2.bind(s1.getsockname())
> +    s2.listen()
> +    s2.setblocking(False)
> +
> +    for i in range(3):
> +        c = socket()
> +        c.connect(s1.getsockname())
> +        try:
> +            print("SUCCESS: assigned properly:", s1.accept())
> +        except:
> +            print("FAIL: wrong assignment")
> +            os.sys.exit(1)

It seems you don't need to handle an exception with ksft.
You can see os.sys.exit(1) triggers another exception when
you run it without patch 1.

TAP version 13
1..1
# timeout set to 3600
# selftests: net: tcp_reuseport.py
# TAP version 13
# 1..1
# FAIL: wrong assignment
# # Exception| Traceback (most recent call last):
# # Exception|   File
"/root/linux/tools/testing/selftests/net/./tcp_reuseport.py", line 26,
in test_reuseport_select
# # Exception|     print("SUCCESS: assigned properly:", s1.accept())
# # Exception|                                          ~~~~~~~~~^^
# # Exception|   File "/usr/lib64/python3.13/socket.py", line 295, in accept
# # Exception|     fd, addr = self._accept()
# # Exception|                ~~~~~~~~~~~~^^
# # Exception| BlockingIOError: [Errno 11] Resource temporarily unavailable
# # Exception|
# # Exception| During handling of the above exception, another
exception occurred:
# # Exception|
# # Exception| Traceback (most recent call last):
# # Exception|   File
"/root/linux/tools/testing/selftests/net/lib/py/ksft.py", line 244, in
ksft_run
# # Exception|     case(*args)
# # Exception|     ~~~~^^^^^^^
# # Exception|   File
"/root/linux/tools/testing/selftests/net/./tcp_reuseport.py", line 29,
in test_reuseport_select
# # Exception|     os.sys.exit(1)
# # Exception|     ~~~~~~~~~~~^^^
# # Exception| SystemExit: 1
# not ok 1 tcp_reuseport.test_reuseport_select
# # Totals: pass:0 fail:1 xfail:0 xpass:0 skip:0 error:0
not ok 1 selftests: net: tcp_reuseport.py # exit=1


btw, I'd write an official uAPI selftest in plain C for socket as
python sometimes does a tricky thing and I don't trust it.

For example, this is...

from socket import *

s = socket()
s.listen(-1)

internally translated to:

socket(AF_INET, SOCK_STREAM|SOCK_CLOEXEC, IPPROTO_IP) = 3
listen(3, 0)                            = 0

Re: [PATCH net v3 1/2] net: tcp: lookup the best matched listen socket

[Menglong Dong]

On Sun, Aug 3, 2025 at 4:32 PM Jason Xing <kerneljasonxing@gmail.com> wrote:
>
> On Sun, Aug 3, 2025 at 12:00 PM Menglong Dong <menglong8.dong@gmail.com> wrote:
> >
> > On Sun, Aug 3, 2025 at 10:54 AM Jason Xing <kerneljasonxing@gmail.com> wrote:
> > >
> > > On Sun, Aug 3, 2025 at 9:59 AM Menglong Dong <menglong8.dong@gmail.com> wrote:
> > > >
> > > > On Sat, Aug 2, 2025 at 9:06 PM Jason Xing <kerneljasonxing@gmail.com> wrote:
> > > > >
[......]
> > >
> > > I'm trying to picture what the usage can be in the userland as you
> > > pointed out in the MD5 case. As to the client side, it seems weird
> > > since it cannot detect and know the priority of the other side where a
> > > few sockets listen on the same address and port.
> >
> > For the server side, it needs to add all the clients information
> > with the TCP_MD5SIG option. For socket1 that binded on the
> > eth0, it will add all the client addresses that come from eth0 to the
> > socket1 with TCP_MD5SIG. So the server knows the clients.
>
> Right, but I meant the _client_ side doesn't know the details of the
> other side, that is to say, the server side needs to keep sure the
> client server easily connects to your preferred listener regardless of
> what the selection algorithm looks like. In terms of what you
> depicted, I see your point. One question is if the kernel or API
> provides any rules and instructions to the users that on the server
> side the sk1 must be selected in your case? Is it possible for other
> cases where the sk2 is the preferable/ideal one? I'm not sure if it's
> worth 'fixing' it in the kernel.

What does sk2 represent here? I don't think that the sk2 should
be preferable. For now, it is selected by the creating order, and
there should not be a case that relies on the socket creation
order......would it?

So the socket should be selected by priority, and sk1, which is binded
on eth0, has higher priority, which is awarded by the users.

I noticed that the net-next is open, so we can use Kuniyuki's
solution instead, which is better for the performance ;)

Thanks!
Menglong Dong

Re: [PATCH net v3 2/2] selftests/net: test TCP reuseport socket selection

[Menglong Dong]

On Wed, Aug 6, 2025 at 3:01 AM Kuniyuki Iwashima <kuniyu@google.com> wrote:
>
> On Sat, Aug 2, 2025 at 2:24 AM Menglong Dong <menglong8.dong@gmail.com> wrote:
> >
> > The test script is provided by Kuniyuki in [1], which is used to test the
> > selection of the TCP reuseport socket problem.
> >
> > Link: https://lore.kernel.org/netdev/20250801040757.1599996-1-kuniyu@google.com/ [1]
> > Signed-off-by: Menglong Dong <dongml2@chinatelecom.cn>
> > ---
> >  tools/testing/selftests/net/Makefile         |  1 +
> >  tools/testing/selftests/net/tcp_reuseport.py | 36 ++++++++++++++++++++
> >  2 files changed, 37 insertions(+)
> >  create mode 100755 tools/testing/selftests/net/tcp_reuseport.py
> >
> > diff --git a/tools/testing/selftests/net/Makefile b/tools/testing/selftests/net/Makefile
> > index b31a71f2b372..0f4c3eea9709 100644
> > --- a/tools/testing/selftests/net/Makefile
> > +++ b/tools/testing/selftests/net/Makefile
> > @@ -117,6 +117,7 @@ TEST_GEN_FILES += tfo
> >  TEST_PROGS += tfo_passive.sh
> >  TEST_PROGS += broadcast_pmtu.sh
> >  TEST_PROGS += ipv6_force_forwarding.sh
> > +TEST_PROGS += tcp_reuseport.py
> >
> >  # YNL files, must be before "include ..lib.mk"
> >  YNL_GEN_FILES := busy_poller netlink-dumps
> > diff --git a/tools/testing/selftests/net/tcp_reuseport.py b/tools/testing/selftests/net/tcp_reuseport.py
> > new file mode 100755
> > index 000000000000..eaeb7096382e
> > --- /dev/null
> > +++ b/tools/testing/selftests/net/tcp_reuseport.py
> > @@ -0,0 +1,36 @@
> > +#!/usr/bin/env python3
> > +# SPDX-License-Identifier: GPL-2.0
> > +
> > +import os
> > +
> > +from lib.py import ksft_run, ksft_exit
> > +from socket import *
> > +
> > +def test_reuseport_select() -> None:
> > +    s1 = socket()
> > +    s1.setsockopt(SOL_SOCKET, SO_REUSEPORT, 1)
> > +    s1.setsockopt(SOL_SOCKET, SO_BINDTODEVICE, b'lo')
> > +    s1.listen()
> > +    s1.setblocking(False)
> > +
> > +    s2 = socket()
> > +    s2.setsockopt(SOL_SOCKET, SO_REUSEPORT, 1)
> > +    s2.bind(s1.getsockname())
> > +    s2.listen()
> > +    s2.setblocking(False)
> > +
> > +    for i in range(3):
> > +        c = socket()
> > +        c.connect(s1.getsockname())
> > +        try:
> > +            print("SUCCESS: assigned properly:", s1.accept())
> > +        except:
> > +            print("FAIL: wrong assignment")
> > +            os.sys.exit(1)
>
> It seems you don't need to handle an exception with ksft.
> You can see os.sys.exit(1) triggers another exception when
> you run it without patch 1.
>
> TAP version 13
> 1..1
> # timeout set to 3600
> # selftests: net: tcp_reuseport.py
> # TAP version 13
> # 1..1
> # FAIL: wrong assignment
> # # Exception| Traceback (most recent call last):
> # # Exception|   File
> "/root/linux/tools/testing/selftests/net/./tcp_reuseport.py", line 26,
> in test_reuseport_select
> # # Exception|     print("SUCCESS: assigned properly:", s1.accept())
> # # Exception|                                          ~~~~~~~~~^^
> # # Exception|   File "/usr/lib64/python3.13/socket.py", line 295, in accept
> # # Exception|     fd, addr = self._accept()
> # # Exception|                ~~~~~~~~~~~~^^
> # # Exception| BlockingIOError: [Errno 11] Resource temporarily unavailable
> # # Exception|
> # # Exception| During handling of the above exception, another
> exception occurred:
> # # Exception|
> # # Exception| Traceback (most recent call last):
> # # Exception|   File
> "/root/linux/tools/testing/selftests/net/lib/py/ksft.py", line 244, in
> ksft_run
> # # Exception|     case(*args)
> # # Exception|     ~~~~^^^^^^^
> # # Exception|   File
> "/root/linux/tools/testing/selftests/net/./tcp_reuseport.py", line 29,
> in test_reuseport_select
> # # Exception|     os.sys.exit(1)
> # # Exception|     ~~~~~~~~~~~^^^
> # # Exception| SystemExit: 1
> # not ok 1 tcp_reuseport.test_reuseport_select
> # # Totals: pass:0 fail:1 xfail:0 xpass:0 skip:0 error:0
> not ok 1 selftests: net: tcp_reuseport.py # exit=1
>
>
> btw, I'd write an official uAPI selftest in plain C for socket as
> python sometimes does a tricky thing and I don't trust it.

Yeah, sounds nice, and C has better compatibility and
reliability for the testing.

>
> For example, this is...
>
> from socket import *
>
> s = socket()
> s.listen(-1)
>
> internally translated to:
>
> socket(AF_INET, SOCK_STREAM|SOCK_CLOEXEC, IPPROTO_IP) = 3
> listen(3, 0)                            = 0

Re: [PATCH net v3 1/2] net: tcp: lookup the best matched listen socket

[Kuniyuki Iwashima]

On Wed, Aug 13, 2025 at 5:55 AM Menglong Dong <menglong8.dong@gmail.com> wrote:
>
> On Sun, Aug 3, 2025 at 4:32 PM Jason Xing <kerneljasonxing@gmail.com> wrote:
> >
> > On Sun, Aug 3, 2025 at 12:00 PM Menglong Dong <menglong8.dong@gmail.com> wrote:
> > >
> > > On Sun, Aug 3, 2025 at 10:54 AM Jason Xing <kerneljasonxing@gmail.com> wrote:
> > > >
> > > > On Sun, Aug 3, 2025 at 9:59 AM Menglong Dong <menglong8.dong@gmail.com> wrote:
> > > > >
> > > > > On Sat, Aug 2, 2025 at 9:06 PM Jason Xing <kerneljasonxing@gmail.com> wrote:
> > > > > >
> [......]
> > > >
> > > > I'm trying to picture what the usage can be in the userland as you
> > > > pointed out in the MD5 case. As to the client side, it seems weird
> > > > since it cannot detect and know the priority of the other side where a
> > > > few sockets listen on the same address and port.
> > >
> > > For the server side, it needs to add all the clients information
> > > with the TCP_MD5SIG option. For socket1 that binded on the
> > > eth0, it will add all the client addresses that come from eth0 to the
> > > socket1 with TCP_MD5SIG. So the server knows the clients.
> >
> > Right, but I meant the _client_ side doesn't know the details of the
> > other side, that is to say, the server side needs to keep sure the
> > client server easily connects to your preferred listener regardless of
> > what the selection algorithm looks like. In terms of what you
> > depicted, I see your point. One question is if the kernel or API
> > provides any rules and instructions to the users that on the server
> > side the sk1 must be selected in your case? Is it possible for other
> > cases where the sk2 is the preferable/ideal one? I'm not sure if it's
> > worth 'fixing' it in the kernel.
>
> What does sk2 represent here? I don't think that the sk2 should
> be preferable. For now, it is selected by the creating order, and
> there should not be a case that relies on the socket creation
> order......would it?
>
> So the socket should be selected by priority, and sk1, which is binded
> on eth0, has higher priority, which is awarded by the users.
>
> I noticed that the net-next is open, so we can use Kuniyuki's
> solution instead, which is better for the performance ;)

I'll post a series next week when Eric is back.