[PATCH RFC 0/3] tcp: allow to repair a tcp connections in closing states

[PATCH RFC 0/3] tcp: allow to repair a tcp connections in closing states

[Andrey Vagin]

Currently connections only in the TCP_ESTABLISHED state can be dumped and
restored. This series allows to restore connections in the FIN_WAIT_1,
FIN_WAIT_2, LAST_ACK, CLOSE_WAIT or CLOSING states.

For restoring closing states we need an ability to restore a fin packet
in a queue. In this series I suggest to use the interface of control
messages for that.

Here is an example of user-space code:

msg.msg_control = buf;
msg.msg_controllen = sizeof buf;
cmsg = CMSG_FIRSTHDR(&msg);
cmsg->cmsg_level = SOL_TCP;
cmsg->cmsg_type = TCP_CMSG_SEND_FIN;
cmsg->cmsg_len = CMSG_LEN(0);
msg.msg_controllen = cmsg->cmsg_len;
if (sendmsg(sk, &msg, 0) < 0) {
	pr_perror("sendmsg");
	return -1;
}

Andrey Vagin (3):
  tcp: allow to enable repair mode for sockets in any state
  tcp: check repair before fastopen in tcp_sendmsg
  tcp: add ability to restore a fin packet

 include/net/tcp.h        |  1 +
 include/uapi/linux/tcp.h |  3 +++
 net/ipv4/tcp.c           | 69 +++++++++++++++++++++++++++++++++++++-----------
 net/ipv4/tcp_input.c     |  2 +-
 4 files changed, 59 insertions(+), 16 deletions(-)

Cc: "David S. Miller" <davem@davemloft.net>
Cc: Alexey Kuznetsov <kuznet@ms2.inr.ac.ru>
Cc: James Morris <jmorris@namei.org>
Cc: Hideaki YOSHIFUJI <yoshfuji@linux-ipv6.org>
Cc: Patrick McHardy <kaber@trash.net>
Cc: Eric Dumazet <edumazet@google.com>
Cc: Pavel Emelyanov <xemul@parallels.com>
Cc: Cyrill Gorcunov <gorcunov@openvz.org>
Signed-off-by: Andrey Vagin <avagin@openvz.org>

-- 
1.8.5.3

[PATCH 1/3] tcp: allow to enable repair mode for sockets in any state

[Andrey Vagin]

The repair mode is used for dumping state of tcp connections
(sequence numbers, queues, options, etc).

Currently the repair mode can be enalbed only for sockets in the
TCP_ESTABLISHED state. If a socket in another state, its internal
state can not be dumped.

Same time there is no guarantee that a connection won't be in other
states when we are dumping it, thus to be able to dump and restore
such states we need to get rid of CLOSE,ESTABLISHED in-kernel
limitation.

I see nothing wrong to allow enabling of the repair mode for connected
sockets in any states.

Cc: "David S. Miller" <davem@davemloft.net>
Cc: Alexey Kuznetsov <kuznet@ms2.inr.ac.ru>
Cc: James Morris <jmorris@namei.org>
Cc: Hideaki YOSHIFUJI <yoshfuji@linux-ipv6.org>
Cc: Patrick McHardy <kaber@trash.net>
Cc: Eric Dumazet <edumazet@google.com>
Cc: Pavel Emelyanov <xemul@parallels.com>
Cc: Cyrill Gorcunov <gorcunov@openvz.org>
Signed-off-by: Andrey Vagin <avagin@openvz.org>
---
 net/ipv4/tcp.c | 13 ++++++++-----
 1 file changed, 8 insertions(+), 5 deletions(-)

diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c
index 97c8f56..267adb7 100644
--- a/net/ipv4/tcp.c
+++ b/net/ipv4/tcp.c
@@ -1106,15 +1106,18 @@ int tcp_sendmsg(struct kiocb *iocb, struct sock *sk, struct msghdr *msg,
 	}
 
 	if (unlikely(tp->repair)) {
+		err = -EINVAL;
+		if (tp->repair_queue == TCP_NO_QUEUE)
+			goto out_err;
+
+		if (sk->sk_state != TCP_ESTABLISHED)
+			goto out_err;
+
 		if (tp->repair_queue == TCP_RECV_QUEUE) {
 			copied = tcp_send_rcvq(sk, msg, size);
 			goto out;
 		}
 
-		err = -EINVAL;
-		if (tp->repair_queue == TCP_NO_QUEUE)
-			goto out_err;
-
 		/* 'common' sending to sendq */
 	}
 
@@ -2375,7 +2378,7 @@ void tcp_sock_destruct(struct sock *sk)
 static inline bool tcp_can_repair_sock(const struct sock *sk)
 {
 	return ns_capable(sock_net(sk)->user_ns, CAP_NET_ADMIN) &&
-		((1 << sk->sk_state) & (TCPF_CLOSE | TCPF_ESTABLISHED));
+		(sk->sk_state != TCP_LISTEN);
 }
 
 static int tcp_repair_options_est(struct tcp_sock *tp,
-- 
1.8.5.3

Re: [PATCH 1/3] tcp: allow to enable repair mode for sockets in any state

[Pavel Emelyanov]

> @@ -2375,7 +2378,7 @@ void tcp_sock_destruct(struct sock *sk)
>  static inline bool tcp_can_repair_sock(const struct sock *sk)
>  {
>  	return ns_capable(sock_net(sk)->user_ns, CAP_NET_ADMIN) &&
> -		((1 << sk->sk_state) & (TCPF_CLOSE | TCPF_ESTABLISHED));
> +		(sk->sk_state != TCP_LISTEN);

This set only covers states that may happen after ESTABLISHED, so
I would still exclude others (e.g. syn-sent) from the white list.

>  }
>  
>  static int tcp_repair_options_est(struct tcp_sock *tp,
> 

Thanks,
Pavel

[PATCH 2/3] tcp: check repair before fastopen in tcp_sendmsg

[Andrey Vagin]

First of all it is logical.

And I found that sk_stream_wait_connect can be called if a socket
isn't in the TCP_ESTABLISHED state. In this case we get -EPIPE.

Cc: "David S. Miller" <davem@davemloft.net>
Cc: Alexey Kuznetsov <kuznet@ms2.inr.ac.ru>
Cc: James Morris <jmorris@namei.org>
Cc: Hideaki YOSHIFUJI <yoshfuji@linux-ipv6.org>
Cc: Patrick McHardy <kaber@trash.net>
Cc: Eric Dumazet <edumazet@google.com>
Cc: Pavel Emelyanov <xemul@parallels.com>
Cc: Cyrill Gorcunov <gorcunov@openvz.org>
Signed-off-by: Andrey Vagin <avagin@openvz.org>
---
 net/ipv4/tcp.c | 33 +++++++++++++++++----------------
 1 file changed, 17 insertions(+), 16 deletions(-)

diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c
index 267adb7..4cd0e87 100644
--- a/net/ipv4/tcp.c
+++ b/net/ipv4/tcp.c
@@ -1084,6 +1084,23 @@ int tcp_sendmsg(struct kiocb *iocb, struct sock *sk, struct msghdr *msg,
 	lock_sock(sk);
 
 	flags = msg->msg_flags;
+
+	if (unlikely(tp->repair)) {
+		err = -EINVAL;
+		if (tp->repair_queue == TCP_NO_QUEUE)
+			goto out_err;
+
+		if (sk->sk_state != TCP_ESTABLISHED)
+			goto out_err;
+
+		if (tp->repair_queue == TCP_RECV_QUEUE) {
+			copied = tcp_send_rcvq(sk, msg, size);
+			goto out;
+		}
+
+		/* 'common' sending to sendq */
+	}
+
 	if (flags & MSG_FASTOPEN) {
 		err = tcp_sendmsg_fastopen(sk, msg, &copied_syn, size);
 		if (err == -EINPROGRESS && copied_syn > 0)
@@ -1105,22 +1122,6 @@ int tcp_sendmsg(struct kiocb *iocb, struct sock *sk, struct msghdr *msg,
 			goto do_error;
 	}
 
-	if (unlikely(tp->repair)) {
-		err = -EINVAL;
-		if (tp->repair_queue == TCP_NO_QUEUE)
-			goto out_err;
-
-		if (sk->sk_state != TCP_ESTABLISHED)
-			goto out_err;
-
-		if (tp->repair_queue == TCP_RECV_QUEUE) {
-			copied = tcp_send_rcvq(sk, msg, size);
-			goto out;
-		}
-
-		/* 'common' sending to sendq */
-	}
-
 	/* This should be in poll */
 	clear_bit(SOCK_ASYNC_NOSPACE, &sk->sk_socket->flags);
 
-- 
1.8.5.3

[PATCH 3/3] tcp: add ability to restore a fin packet

[Andrey Vagin]

It's required for restoring sockets in closing states:
TCP_FIN_WAIT{1,2}, TCP_WAIT_STOP, TCP_CLOSING, TCP_LAST_ACK.

A fin packet is restored by sending a control message (ancillary data).
In which queue a packet is restored depends on a value of
tp->repair_queue.

This interface is choosen, because we are goint to use sendmsg for
restoring sockets in TCP_SYN_RECV states.  Requests in the TCP_SYN_RECV
state will be restored by sending messages in a proper listen socket. A
message will contain address and a control messages with sequence
numbers.

Cc: "David S. Miller" <davem@davemloft.net>
Cc: Alexey Kuznetsov <kuznet@ms2.inr.ac.ru>
Cc: James Morris <jmorris@namei.org>
Cc: Hideaki YOSHIFUJI <yoshfuji@linux-ipv6.org>
Cc: Patrick McHardy <kaber@trash.net>
Cc: Eric Dumazet <edumazet@google.com>
Cc: Pavel Emelyanov <xemul@parallels.com>
Cc: Cyrill Gorcunov <gorcunov@openvz.org>
Signed-off-by: Andrey Vagin <avagin@openvz.org>
---
 include/net/tcp.h        |  1 +
 include/uapi/linux/tcp.h |  3 +++
 net/ipv4/tcp.c           | 40 ++++++++++++++++++++++++++++++++++++++++
 net/ipv4/tcp_input.c     |  2 +-
 4 files changed, 45 insertions(+), 1 deletion(-)

diff --git a/include/net/tcp.h b/include/net/tcp.h
index 8c4dd63..4b4d9e8 100644
--- a/include/net/tcp.h
+++ b/include/net/tcp.h
@@ -561,6 +561,7 @@ void tcp_cwnd_application_limited(struct sock *sk);
 void tcp_resume_early_retransmit(struct sock *sk);
 void tcp_rearm_rto(struct sock *sk);
 void tcp_reset(struct sock *sk);
+void tcp_fin(struct sock *sk);
 
 /* tcp_timer.c */
 void tcp_init_xmit_timers(struct sock *);
diff --git a/include/uapi/linux/tcp.h b/include/uapi/linux/tcp.h
index 377f1e5..2cc6876 100644
--- a/include/uapi/linux/tcp.h
+++ b/include/uapi/linux/tcp.h
@@ -199,4 +199,7 @@ struct tcp_md5sig {
 	__u8	tcpm_key[TCP_MD5SIG_MAXKEYLEN];		/* key (binary) */
 };
 
+/* Conntroll message types to repair tcp connections */
+#define TCP_REPAIR_SEND_FIN	1
+
 #endif /* _UAPI_LINUX_TCP_H */
diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c
index 4cd0e87..7f5a15c 100644
--- a/net/ipv4/tcp.c
+++ b/net/ipv4/tcp.c
@@ -1070,6 +1070,34 @@ static int tcp_sendmsg_fastopen(struct sock *sk, struct msghdr *msg,
 	return err;
 }
 
+static int tcp_repair_cmsg(struct sock *sk, struct msghdr *msg)
+{
+	struct tcp_sock *tp = tcp_sk(sk);
+	struct cmsghdr *cmsg;
+
+	for (cmsg = CMSG_FIRSTHDR(msg); cmsg; cmsg = CMSG_NXTHDR(msg, cmsg)) {
+		if (!CMSG_OK(msg, cmsg))
+			return -EINVAL;
+		if (cmsg->cmsg_level != SOL_TCP)
+			continue;
+
+		switch (cmsg->cmsg_type) {
+		case TCP_REPAIR_SEND_FIN:
+			if (tp->repair_queue == TCP_RECV_QUEUE)
+				tcp_fin(sk);
+			else if (tp->repair_queue == TCP_SEND_QUEUE)
+				tcp_shutdown(sk, SEND_SHUTDOWN);
+			else
+				return -EINVAL;
+			break;
+		default:
+			return -EINVAL;
+		}
+	}
+
+	return 0;
+}
+
 int tcp_sendmsg(struct kiocb *iocb, struct sock *sk, struct msghdr *msg,
 		size_t size)
 {
@@ -1090,6 +1118,18 @@ int tcp_sendmsg(struct kiocb *iocb, struct sock *sk, struct msghdr *msg,
 		if (tp->repair_queue == TCP_NO_QUEUE)
 			goto out_err;
 
+		if (msg->msg_controllen) {
+			if (size != 0)
+				return -EINVAL;
+
+			err = tcp_repair_cmsg(sk, msg);
+			if (err < 0)
+				goto out_err;
+
+			goto out;
+		}
+
+		err = -EINVAL;
 		if (sk->sk_state != TCP_ESTABLISHED)
 			goto out_err;
 
diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c
index eeaac39..352480c 100644
--- a/net/ipv4/tcp_input.c
+++ b/net/ipv4/tcp_input.c
@@ -3820,7 +3820,7 @@ void tcp_reset(struct sock *sk)
  *
  *	If we are in FINWAIT-2, a received FIN moves us to TIME-WAIT.
  */
-static void tcp_fin(struct sock *sk)
+void tcp_fin(struct sock *sk)
 {
 	struct tcp_sock *tp = tcp_sk(sk);
 	const struct dst_entry *dst;
-- 
1.8.5.3

Re: [CRIU] [PATCH 3/3] tcp: add ability to restore a fin packet

[Christopher Covington]

On 03/21/2014 09:33 AM, Andrey Vagin wrote:
> It's required for restoring sockets in closing states:
> TCP_FIN_WAIT{1,2}, TCP_WAIT_STOP, TCP_CLOSING, TCP_LAST_ACK.
> 
> A fin packet is restored by sending a control message (ancillary data).
> In which queue a packet is restored depends on a value of
> tp->repair_queue.
> 
> This interface is choosen, because we are goint to use sendmsg for

Nit: chosen, going

> --- a/include/uapi/linux/tcp.h
> +++ b/include/uapi/linux/tcp.h
> @@ -199,4 +199,7 @@ struct tcp_md5sig {
>  	__u8	tcpm_key[TCP_MD5SIG_MAXKEYLEN];		/* key (binary) */
>  };
>  
> +/* Conntroll message types to repair tcp connections */

Nit: Control

Christopher

-- 
Employee of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by the Linux Foundation.

Re: [PATCH RFC 0/3] tcp: allow to repair a tcp connections in closing states

[David Miller]

From: Andrey Vagin <avagin@openvz.org>
Date: Fri, 21 Mar 2014 17:32:58 +0400

> Currently connections only in the TCP_ESTABLISHED state can be dumped and
> restored. This series allows to restore connections in the FIN_WAIT_1,
> FIN_WAIT_2, LAST_ACK, CLOSE_WAIT or CLOSING states.
> 
> For restoring closing states we need an ability to restore a fin packet
> in a queue. In this series I suggest to use the interface of control
> messages for that.

I see no real value in this unless you tackle TIME_WAIT as well, and
that's going to add so much code.