[Xenomai] A fix to few generic issues in rttcp stack

[Xenomai] A fix to few generic issues in rttcp stack

[Matti Suominen]

Hi,

I have case where we use rtnet's tcp socket in server application to 
communicate with real time and non-real time tcp clients. To get 
communication work between client and server I had to do some changes to 
stack/ipv4/tcp/tcp.c file.

First. Original tcp flag enumeration didn't work at all in this case. This 
was verified by using Wireshark to track communication which didn't show 
me any flags in tcp packets. This was solved by replacing flags assigning 
TCP_FLAG_ prefixed enums with TCPHDR_ prefixed defines from <net/tcp.h> in 
stack/ipv4/tcp/tcp.c file. The original code only works in case of little 
endian machine, as rt_tcp_set_flags function places assumption that flags 
are located in the least-significant 8bits of the 32bit flag variable. 
This approach is similar to what regular TCP is using: 
https://github.com/torvalds/linux/blob/master/net/ipv4/tcp.c

Second. Server was able to receive messages from client but was unable to 
reply messages. I was able to trace function calls by adding debug prints 
to tcp.c functions and noticed that accepted socket wasn't signaled for 
"ready to send" at all and server was unable to reply messages. This seems 
to work properly if the send signal is posted in the end of the accept 
function, which also would seem the logical way to initialize it once 
connection has been established.

Third. In our case we have regular Linux tcp sockets and rtnet tcp sockets 
working parallel, serving different services. Referred to this I noticed 
that *rt_tcp_dest_socket() may cause problem when real time stack 
processes a message targeted to a regular Linux socket and does not find a 
rttcp socket for it. The rttcp stack then  tries to send RST|ACK to client 
to terminate the connection. This is temporarily disabled by setting 
"if (!th->rst)" clause into #ifdef YET_UNUSED. This should be corrected 
for example by implementing layer which is able to loop also non real time 
sockets. Or is it really a desired rttcp design principle to unsupport the 
usage of regular Linux tcp sockets in parallel?

---------------------------------------------------------------------------

We noticed also that Windows client causes assertion in pool balance at 
end of transmission.

Assertion failed! /src/stack/socket.c:rt_socket_cleanup:199 
(&sock->skb_pool)->pool_balance == 0

Call Trace:
[cbe97db0] [c00087e0] show_stack+0x48/0x150 (unreliable)
[cbe97df0] [d3830414] rt_socket_cleanup+0x13c/0x140 [rtnet]
[cbe97e10] [d38a1f00] rt_tcp_close+0x2b0/0x3a8 [rttcp]
[cbe97e50] [c009d3c0] __rt_dev_close+0x174/0x368
[cbe97e80] [c0071588] hisyscall_event+0x18c/0x35c
[cbe97ec0] [c00615c0] __ipipe_dispatch_event+0xac/0x1ec
[cbe97f20] [c000ea5c] __ipipe_syscall_root+0x5c/0x140
[cbe97f40] [c000fe60] DoSyscall+0x20/0x5c
--- Exception: c01 at 0x4899ba08

This is still unsolved but it seems not to cause harms.

---------------------------------------------------------------------------

Difference between Xenomai 3 tcp.c and our tcp.c additions at end of 
message. Our fixes is based to rtnet 0.9.13, but this is how we could do 
fix if we use Xenomai 3's rtnet.

Best regards,
Matti Suominen
Wapice Oy
Vaasa, Finland

---------------------------------------------------------------------------

PowerPC, CPU: MPC8309E
RTNet version 0.9.13
Xenomai 2.6.1
Linux version 3.1.10-ipipe

---------------------------------------------------------------------------
--- xeno3-tcp.c	2016-03-30 09:49:36.020400900 +0300
+++ xeno3-tcp-diff.c	2016-03-31 10:04:44.744351290 +0300
@@ -704,7 +704,7 @@
 
     /* add rtskb entry to the socket retransmission queue */
     if (ts->tcp_state != TCP_CLOSE &&
-	((flags & (TCP_FLAG_SYN|TCP_FLAG_FIN)) || data_len)) {
+	((flags & (TCPHDR_SYN|TCPHDR_FIN)) || data_len)) {
 	/* rtskb_clone below is called under lock, this is an admission,
 	   because for now there is no rtskb copy by reference */
 	cloned_skb = rtskb_clone(skb, &ts->sock.skb_pool);
@@ -720,7 +720,7 @@
 
     /* need to update sync here, because it is safe way in
        comparison with races on fast ACK response */
-    if (flags & (TCP_FLAG_FIN|TCP_FLAG_SYN))
+    if (flags & (TCPHDR_FIN|TCPHDR_SYN))
 	ts->sync.seq++;
 
     ts->sync.seq += data_len;
@@ -832,6 +832,11 @@
 	/*
 	  rtdm_printk("Not found addr:0x%08x, port: 0x%04x\n", daddr, dport);
 	*/
+#ifdef YET_UNUSED
+/* This ifdef is added because parallel use of linux and rt_dev sockets may
+ * produce unwanted connection close when application uses linux socket instead
+ * of rt_dev sockets and cannot find destination socket for package.
+ */
 	if (!th->rst) {
 	    /* No listening socket found, send RST|ACK */
 	    rst_socket.saddr = daddr;
@@ -846,10 +851,11 @@
 
 	    if (rt_ip_route_output(&rst_socket.rt, daddr, saddr) == 0) {
 		rt_socket_reference(&rst_socket.sock);
-		rt_tcp_send(&rst_socket, TCP_FLAG_ACK|TCP_FLAG_RST);
+		rt_tcp_send(&rst_socket, TCPHDR_ACK|TCPHDR_RST);
 		rtdev_dereference(rst_socket.rt.rtdev);
 	    }
 	}
+#endif
     }
 
     return skb->sk;
@@ -918,7 +924,7 @@
     if (ts->sync.ack_seq == (seq + 1) &&
 	ts->tcp_state == TCP_ESTABLISHED) {
 	rtdm_lock_put_irqrestore(&ts->socket_lock, context);
-	rt_tcp_send(ts, TCP_FLAG_ACK);
+	rt_tcp_send(ts, TCPHDR_ACK);
 	goto feed;
     }
 
@@ -930,7 +936,7 @@
 	    rtdm_lock_put_irqrestore(&ts->socket_lock, context);
 	    rtdm_event_signal(&ts->conn_evt);
 	    /* Send ACK */
-	    rt_tcp_send(ts, TCP_FLAG_ACK);
+	    rt_tcp_send(ts, TCPHDR_ACK);
 	    goto feed;
 	}
 
@@ -940,7 +946,7 @@
 
 	/* Send RST|ACK */
 	rtdm_event_signal(&ts->conn_evt);
-	rt_tcp_send(ts, TCP_FLAG_RST|TCP_FLAG_ACK);
+	rt_tcp_send(ts, TCPHDR_RST|TCPHDR_ACK);
 	goto drop;
     }
 
@@ -965,7 +971,7 @@
     } else if (rt_tcp_after(seq, ts->sync.ack_seq - data_len)) {
 	/* retransmission of data we already acked */
 	rtdm_lock_put_irqrestore(&ts->socket_lock, context);
-	rt_tcp_send(ts, TCP_FLAG_ACK);
+	rt_tcp_send(ts, TCPHDR_ACK);
 	goto drop;
     } else {
 	/* drop forward ack */
@@ -995,7 +1001,7 @@
 	if (rt_ip_route_output(&rst_socket.rt, rst_socket.daddr,
 			       rst_socket.saddr) == 0) {
 	    rt_socket_reference(&rst_socket.sock);
-	    rt_tcp_send(&rst_socket, TCP_FLAG_RST|TCP_FLAG_ACK);
+	    rt_tcp_send(&rst_socket, TCPHDR_RST|TCPHDR_ACK);
 	    rtdev_dereference(rst_socket.rt.rtdev);
 	}
 	goto drop;
@@ -1029,14 +1035,14 @@
 	    if (signal)
 		rt_tcp_socket_invalidate_signal(ts);
 
-	    rt_tcp_send(ts, TCP_FLAG_ACK);
+	    rt_tcp_send(ts, TCPHDR_ACK);
 	    goto feed;
 	} else if ((ts->tcp_state == TCP_FIN_WAIT1 && th->ack) ||
 		   ts->tcp_state == TCP_FIN_WAIT2) {
 	    /* Send ACK */
 	    ts->tcp_state = TCP_TIME_WAIT;
 	    rtdm_lock_put_irqrestore(&ts->socket_lock, context);
-	    rt_tcp_send(ts, TCP_FLAG_ACK);
+	    rt_tcp_send(ts, TCPHDR_ACK);
 	    /* data receiving is not possible anymore */
 	    rtdm_sem_destroy(&ts->sock.pending_sem);
 	    goto feed;
@@ -1044,7 +1050,7 @@
 	    /* Send ACK */
 	    ts->tcp_state = TCP_CLOSING;
 	    rtdm_lock_put_irqrestore(&ts->socket_lock, context);
-	    rt_tcp_send(ts, TCP_FLAG_ACK);
+	    rt_tcp_send(ts, TCPHDR_ACK);
 	    /* data receiving is not possible anymore */
 	    rtdm_sem_destroy(&ts->sock.pending_sem);
 	    goto feed;
@@ -1076,13 +1082,13 @@
 	    rtdm_lock_put_irqrestore(&ts->socket_lock, context);
 
 	    /* Send SYN|ACK */
-	    rt_tcp_send(ts, TCP_FLAG_SYN|TCP_FLAG_ACK);
+	    rt_tcp_send(ts, TCPHDR_SYN|TCPHDR_ACK);
 	    goto drop;
 	}
 
 	/* Send RST|ACK */
 	rtdm_lock_put_irqrestore(&ts->socket_lock, context);
-	rt_tcp_send(ts, TCP_FLAG_RST|TCP_FLAG_ACK);
+	rt_tcp_send(ts, TCPHDR_RST|TCPHDR_ACK);
 	goto drop;
     }
 
@@ -1134,7 +1140,7 @@
     /* Send ACK */
     ts->sync.window -= data_len;
     rtdm_lock_put_irqrestore(&ts->socket_lock, context);
-    rt_tcp_send(ts, TCP_FLAG_ACK);
+    rt_tcp_send(ts, TCPHDR_ACK);
 
     rtskb_queue_tail(&skb->sk->incoming, skb);
     rtdm_sem_up(&ts->sock.pending_sem);
@@ -1181,7 +1187,7 @@
     if (data_len > dst_window)
 	data_len = dst_window;
 
-    if ((ret = rt_tcp_segment(&ts->rt, ts, TCP_FLAG_ACK,
+    if ((ret = rt_tcp_segment(&ts->rt, ts, TCPHDR_ACK,
 			      data_len, data_ptr, 0)) < 0) {
 	rtdm_printk("rttcp: cann't send a packet: err %d\n", -ret);
 	return ret;
@@ -1372,7 +1378,7 @@
 	ts->tcp_state == TCP_SYN_RECV) {
 	/* close() from ESTABLISHED */
 	send_cmd.ts = ts;
-	send_cmd.flags = TCP_FLAG_FIN|TCP_FLAG_ACK;
+	send_cmd.flags = TCPHDR_FIN|TCPHDR_ACK;
 	signal = rt_tcp_socket_invalidate(ts, TCP_FIN_WAIT1);
 
 	rtdm_lock_put_irqrestore(&ts->socket_lock, context);
@@ -1386,7 +1392,7 @@
     } else if (ts->tcp_state == TCP_CLOSE_WAIT) {
 	/* Send FIN in CLOSE_WAIT */
 	send_cmd.ts = ts;
-	send_cmd.flags = TCP_FLAG_FIN|TCP_FLAG_ACK;
+	send_cmd.flags = TCPHDR_FIN|TCPHDR_ACK;
 	signal = rt_tcp_socket_invalidate(ts, TCP_LAST_ACK);
 
 	rtdm_lock_put_irqrestore(&ts->socket_lock, context);
@@ -1532,7 +1538,7 @@
     rtdm_lock_put_irqrestore(&ts->socket_lock, context);
 
     /* Complete three-way handshake */
-    ret = rt_tcp_send(ts, TCP_FLAG_SYN);
+    ret = rt_tcp_send(ts, TCPHDR_SYN);
     if (ret < 0) {
 	rtdm_printk("rttcp: cann't send SYN\n");
 	return ret;
@@ -1668,6 +1674,10 @@
     rtdm_lock_put_irqrestore(&ts->socket_lock, context);
 
     ret = rt_socket_fd(&ts->sock)->fd;
+/*
+ * Socket should be signaled ready to send
+ */
+    rtdm_event_signal(&ts->send_evt);
 
  err:
     /* it is not critical to leave this unlocked
@@ -1931,7 +1941,7 @@
 	    } else {
 		ts->sync.window = block_size;
 		rtdm_lock_put_irqrestore(&ts->socket_lock, context);
-		rt_tcp_send(ts, TCP_FLAG_ACK); /* window update */
+		rt_tcp_send(ts, TCPHDR_ACK); /* window update */
 	    }
 
 	    __rtskb_pull(skb, block_size);
@@ -1954,7 +1964,7 @@
 	} else {
 	    ts->sync.window = block_size;
 	    rtdm_lock_put_irqrestore(&ts->socket_lock, context);
-	    rt_tcp_send(ts, TCP_FLAG_ACK); /* window update */
+	    rt_tcp_send(ts, TCPHDR_ACK); /* window update */
 	}
 
 	if ((skb = skb->next) != NULL) {

Re: [Xenomai] A fix to few generic issues in rttcp stack

[Gilles Chanteperdrix]

On Thu, Mar 31, 2016 at 10:33:09AM +0000, Matti Suominen wrote:
> Hi,
> 
> I have case where we use rtnet's tcp socket in server application to 
> communicate with real time and non-real time tcp clients. To get 
> communication work between client and server I had to do some changes to 
> stack/ipv4/tcp/tcp.c file.
> 
> First. Original tcp flag enumeration didn't work at all in this case. This 
> was verified by using Wireshark to track communication which didn't show 
> me any flags in tcp packets. This was solved by replacing flags assigning 
> TCP_FLAG_ prefixed enums with TCPHDR_ prefixed defines from <net/tcp.h> in 
> stack/ipv4/tcp/tcp.c file. The original code only works in case of little 
> endian machine, as rt_tcp_set_flags function places assumption that flags 
> are located in the least-significant 8bits of the 32bit flag variable. 
> This approach is similar to what regular TCP is using: 
> https://github.com/torvalds/linux/blob/master/net/ipv4/tcp.c
> 
> Second. Server was able to receive messages from client but was unable to 
> reply messages. I was able to trace function calls by adding debug prints 
> to tcp.c functions and noticed that accepted socket wasn't signaled for 
> "ready to send" at all and server was unable to reply messages. This seems 
> to work properly if the send signal is posted in the end of the accept 
> function, which also would seem the logical way to initialize it once 
> connection has been established.
> 
> Third. In our case we have regular Linux tcp sockets and rtnet tcp sockets 
> working parallel, serving different services. Referred to this I noticed 
> that *rt_tcp_dest_socket() may cause problem when real time stack 
> processes a message targeted to a regular Linux socket and does not find a 
> rttcp socket for it. The rttcp stack then  tries to send RST|ACK to client 
> to terminate the connection. This is temporarily disabled by setting 
> "if (!th->rst)" clause into #ifdef YET_UNUSED. This should be corrected 
> for example by implementing layer which is able to loop also non real time 
> sockets. Or is it really a desired rttcp design principle to unsupport the 
> usage of regular Linux tcp sockets in parallel?
> 
> ---------------------------------------------------------------------------
> 
> We noticed also that Windows client causes assertion in pool balance at 
> end of transmission.
> 
> Assertion failed! /src/stack/socket.c:rt_socket_cleanup:199 
> (&sock->skb_pool)->pool_balance == 0

The pool_balance assertions do not work: they are updated outside of
critical sections, and thus are subject to race conditions.

> 
> Call Trace:
> [cbe97db0] [c00087e0] show_stack+0x48/0x150 (unreliable)
> [cbe97df0] [d3830414] rt_socket_cleanup+0x13c/0x140 [rtnet]
> [cbe97e10] [d38a1f00] rt_tcp_close+0x2b0/0x3a8 [rttcp]
> [cbe97e50] [c009d3c0] __rt_dev_close+0x174/0x368
> [cbe97e80] [c0071588] hisyscall_event+0x18c/0x35c
> [cbe97ec0] [c00615c0] __ipipe_dispatch_event+0xac/0x1ec
> [cbe97f20] [c000ea5c] __ipipe_syscall_root+0x5c/0x140
> [cbe97f40] [c000fe60] DoSyscall+0x20/0x5c
> --- Exception: c01 at 0x4899ba08
> 
> This is still unsolved but it seems not to cause harms.
> 
> ---------------------------------------------------------------------------
> 
> Difference between Xenomai 3 tcp.c and our tcp.c additions at end of 
> message. Our fixes is based to rtnet 0.9.13, but this is how we could do 
> fix if we use Xenomai 3's rtnet.

TCP is disabled in Xenomai 3 rtnet. The main problem is that
creating a socket in Xenomai 3 requires switching to secondary mode,
so the "accept" service seems hard to implement without leaving
primary mode. Given the fact that TCP can not be a deterministic
protocol, I planned to drop TCP support in the next RTnet iteration.
If you require to send messages over TCP from a thread running in
primary mode, you should rather pass the message using an XDDP
socket to a thread running in secondary mode and using plain linux
TCP sockets. You will not get more determinism with TCP in RTnet.

-- 
					    Gilles.
https://click-hack.org

Re: [Xenomai] A fix to few generic issues in rttcp stack

[Gilles Chanteperdrix]

On Thu, Mar 31, 2016 at 12:44:09PM +0200, Gilles Chanteperdrix wrote:
> TCP is disabled in Xenomai 3 rtnet. The main problem is that
> creating a socket in Xenomai 3 requires switching to secondary mode,
> so the "accept" service seems hard to implement without leaving
> primary mode. Given the fact that TCP can not be a deterministic
> protocol, I planned to drop TCP support in the next RTnet iteration.
> If you require to send messages over TCP from a thread running in
> primary mode, you should rather pass the message using an XDDP
> socket to a thread running in secondary mode and using plain linux
> TCP sockets. You will not get more determinism with TCP in RTnet.

The question I guess, is, why do you need rttcp ? If you really need
it, then we will have to find a solution for the "accept" problem.
Keeping a pool of sockets ready for instance.

-- 
					    Gilles.
https://click-hack.org

Re: [Xenomai] A fix to few generic issues in rttcp stack

[Matti Suominen]

From: Gilles Chanteperdrix [mailto:gilles.chanteperdrix@xenomai.org] 
Sent: 31. maaliskuuta 2016 19:23
To: Matti Suominen <matti.suominen@wapice.com>
Cc: Lassi Niemistö <lassi.niemisto@wapice.com>; xenomai@xenomai.org
>> TCP is disabled in Xenomai 3 rtnet. The main problem is that creating 
>> a socket in Xenomai 3 requires switching to secondary mode, so the 
>> "accept" service seems hard to implement without leaving primary mode. 
>> Given the fact that TCP can not be a deterministic protocol, I planned 
>> to drop TCP support in the next RTnet iteration.
>> If you require to send messages over TCP from a thread running in 
>> primary mode, you should rather pass the message using an XDDP socket 
>> to a thread running in secondary mode and using plain linux TCP 
>> sockets. You will not get more determinism with TCP in RTnet.
>
>The question I guess, is, why do you need rttcp ? If you really need it, 
>then we will have to find a solution for the "accept" problem.
>Keeping a pool of sockets ready for instance.
>

Sorry to hear that you are planning to drop support for rttcp. It's a 
useful feature. TCP can be used by deterministic way if your connection is 
point to point and thus retransmission is clearly a sign for error in  
connection. Rttcp is also significantly faster than regular tcp.

For demonstration, I ran some test in a setup with rttcp client and two 
server implementations; one rttcp and one with regular sockets. Our tests 
are showing that average round-trip-time for same server application is 
half when using rttcp instead of normal Linux sockets (470 µs vs. 960 µs). 
Maximum rtt in our case is 4 ms for rt and easily over 10 ms for normal 
Linux sockets. Each send/recv call with regular sockets drops to secondary 
mode so there is no way the rtt maximum could be deterministic. Dropping 
does not happen in rttcp.

In many applications it isn't a problem when accept drops to secondary 
mode, because creating a connection surely doesn't need to be a real time 
operation when connection serving is still real time. Of course accept 
which drops to secondary mode is kind a blemish when you think a big 
picture of real time kernel, but definitely isn't a reason to drop rttcp 
away.

>From my point of view it is clearly useful feature if same application 
(for example Modbus/TCP) can be used from rt and pc clients, and I thus do 
not need to maintain two protocols.

Matti

Re: [Xenomai] A fix to few generic issues in rttcp stack

[Gilles Chanteperdrix]

On Fri, Apr 01, 2016 at 09:36:26AM +0000, Matti Suominen wrote:
> From: Gilles Chanteperdrix [mailto:gilles.chanteperdrix@xenomai.org] 
> Sent: 31. maaliskuuta 2016 19:23
> To: Matti Suominen <matti.suominen@wapice.com>
> Cc: Lassi Niemistö <lassi.niemisto@wapice.com>; xenomai@xenomai.org
> >> TCP is disabled in Xenomai 3 rtnet. The main problem is that creating 
> >> a socket in Xenomai 3 requires switching to secondary mode, so the 
> >> "accept" service seems hard to implement without leaving primary mode. 
> >> Given the fact that TCP can not be a deterministic protocol, I planned 
> >> to drop TCP support in the next RTnet iteration.
> >> If you require to send messages over TCP from a thread running in 
> >> primary mode, you should rather pass the message using an XDDP socket 
> >> to a thread running in secondary mode and using plain linux TCP 
> >> sockets. You will not get more determinism with TCP in RTnet.
> >
> >The question I guess, is, why do you need rttcp ? If you really need it, 
> >then we will have to find a solution for the "accept" problem.
> >Keeping a pool of sockets ready for instance.
> >
> 
> Sorry to hear that you are planning to drop support for rttcp. It's a 
> useful feature. TCP can be used by deterministic way if your connection is 
> point to point and thus retransmission is clearly a sign for error in  
> connection. Rttcp is also significantly faster than regular tcp.
> 
> For demonstration, I ran some test in a setup with rttcp client and two 
> server implementations; one rttcp and one with regular sockets. Our tests 
> are showing that average round-trip-time for same server application is 
> half when using rttcp instead of normal Linux sockets (470 µs vs. 960 µs). 
> Maximum rtt in our case is 4 ms for rt and easily over 10 ms for normal 
> Linux sockets. Each send/recv call with regular sockets drops to secondary 
> mode so there is no way the rtt maximum could be deterministic. Dropping 
> does not happen in rttcp.

Ok, I understand your point of view. I would say in that case you
could use UDP instead of TCP, but I guess you do not get to choose
the underlying IP protocol of application level protocols.

> 
> In many applications it isn't a problem when accept drops to secondary 
> mode, because creating a connection surely doesn't need to be a real time 
> operation when connection serving is still real time. Of course accept 
> which drops to secondary mode is kind a blemish when you think a big 
> picture of real time kernel, but definitely isn't a reason to drop rttcp 
> away.

>From what I understood looking at TCP servers it is very common for
a server to be implemented with one big select loop which both
accepts new connections and sends/receives to existing ones. If
accept drops to secondary mode and you want to keep communication
with existing connections in primary mode, you will have to split
the loop in two threads and have a means of communication (message
queueus for instance, or XDDP sockets) between the two threads. This
makes the job of porting a TCP server to RTnet harder than it should
be. I wonder if we can not hide that second thread in Xenomai
libraries (like is done for printf) and have listen and accept
handle the synchronization with it.

> 
> From my point of view it is clearly useful feature if same application 
> (for example Modbus/TCP) can be used from rt and pc clients, and I thus do 
> not need to maintain two protocols.

Ok, I will try to think of something.

-- 
					    Gilles.
https://click-hack.org

Re: [Xenomai] A fix to few generic issues in rttcp stack

[Gilles Chanteperdrix]

On Thu, Mar 31, 2016 at 10:33:09AM +0000, Matti Suominen wrote:
> Hi,
> 
> I have case where we use rtnet's tcp socket in server application to 
> communicate with real time and non-real time tcp clients. To get 
> communication work between client and server I had to do some changes to 
> stack/ipv4/tcp/tcp.c file.
> 
> First. Original tcp flag enumeration didn't work at all in this case. This 
> was verified by using Wireshark to track communication which didn't show 
> me any flags in tcp packets. This was solved by replacing flags assigning 
> TCP_FLAG_ prefixed enums with TCPHDR_ prefixed defines from <net/tcp.h> in 
> stack/ipv4/tcp/tcp.c file. The original code only works in case of little 
> endian machine, as rt_tcp_set_flags function places assumption that flags 
> are located in the least-significant 8bits of the 32bit flag variable. 
> This approach is similar to what regular TCP is using: 
> https://github.com/torvalds/linux/blob/master/net/ipv4/tcp.c
> 
> Second. Server was able to receive messages from client but was unable to 
> reply messages. I was able to trace function calls by adding debug prints 
> to tcp.c functions and noticed that accepted socket wasn't signaled for 
> "ready to send" at all and server was unable to reply messages. This seems 
> to work properly if the send signal is posted in the end of the accept 
> function, which also would seem the logical way to initialize it once 
> connection has been established.
> 
> Third. In our case we have regular Linux tcp sockets and rtnet tcp sockets 
> working parallel, serving different services. Referred to this I noticed 
> that *rt_tcp_dest_socket() may cause problem when real time stack 
> processes a message targeted to a regular Linux socket and does not find a 
> rttcp socket for it. The rttcp stack then  tries to send RST|ACK to client 
> to terminate the connection. This is temporarily disabled by setting 
> "if (!th->rst)" clause into #ifdef YET_UNUSED. This should be corrected 
> for example by implementing layer which is able to loop also non real time 
> sockets. Or is it really a desired rttcp design principle to unsupport the 
> usage of regular Linux tcp sockets in parallel?

I am going to merge your patch for the time being. However, I have a
question about using rttcp in parallel with Linux tcp: are you using
the rtnetproxy module? If yes, then testing for
CONFIG_XENO_DRIVERS_NET_ADDON_PROXY and the rt_ip_fallback_handler
pointer would be the way to go.

-- 
					    Gilles.
https://click-hack.org

Re: [Xenomai] A fix to few generic issues in rttcp stack

[Gilles Chanteperdrix]

On Sun, Apr 03, 2016 at 09:09:48PM +0200, Gilles Chanteperdrix wrote:
> On Thu, Mar 31, 2016 at 10:33:09AM +0000, Matti Suominen wrote:
> > Hi,
> > 
> > I have case where we use rtnet's tcp socket in server application to 
> > communicate with real time and non-real time tcp clients. To get 
> > communication work between client and server I had to do some changes to 
> > stack/ipv4/tcp/tcp.c file.
> > 
> > First. Original tcp flag enumeration didn't work at all in this case. This 
> > was verified by using Wireshark to track communication which didn't show 
> > me any flags in tcp packets. This was solved by replacing flags assigning 
> > TCP_FLAG_ prefixed enums with TCPHDR_ prefixed defines from <net/tcp.h> in 
> > stack/ipv4/tcp/tcp.c file. The original code only works in case of little 
> > endian machine, as rt_tcp_set_flags function places assumption that flags 
> > are located in the least-significant 8bits of the 32bit flag variable. 
> > This approach is similar to what regular TCP is using: 
> > https://github.com/torvalds/linux/blob/master/net/ipv4/tcp.c
> > 
> > Second. Server was able to receive messages from client but was unable to 
> > reply messages. I was able to trace function calls by adding debug prints 
> > to tcp.c functions and noticed that accepted socket wasn't signaled for 
> > "ready to send" at all and server was unable to reply messages. This seems 
> > to work properly if the send signal is posted in the end of the accept 
> > function, which also would seem the logical way to initialize it once 
> > connection has been established.
> > 
> > Third. In our case we have regular Linux tcp sockets and rtnet tcp sockets 
> > working parallel, serving different services. Referred to this I noticed 
> > that *rt_tcp_dest_socket() may cause problem when real time stack 
> > processes a message targeted to a regular Linux socket and does not find a 
> > rttcp socket for it. The rttcp stack then  tries to send RST|ACK to client 
> > to terminate the connection. This is temporarily disabled by setting 
> > "if (!th->rst)" clause into #ifdef YET_UNUSED. This should be corrected 
> > for example by implementing layer which is able to loop also non real time 
> > sockets. Or is it really a desired rttcp design principle to unsupport the 
> > usage of regular Linux tcp sockets in parallel?
> 
> I am going to merge your patch for the time being. However, I have a
> question about using rttcp in parallel with Linux tcp: are you using
> the rtnetproxy module? If yes, then testing for
> CONFIG_XENO_DRIVERS_NET_ADDON_PROXY and the rt_ip_fallback_handler
> pointer would be the way to go.

Hi, ping ?

-- 
					    Gilles.
https://click-hack.org

Re: [Xenomai] A fix to few generic issues in rttcp stack

[Matti Suominen]

Hi and sorry for delay.

Yes, we are using rtnetproxy and I am now testing fallback handler. I'm 
not sure am I using it right way but it seems to work correctly. Code 
snippet at end of message.

Matti Suominen
Wapice Oy
www.wapice.com

On Sun, Apr 03, 2016 at 09:09:48PM +0200, Gilles Chanteperdrix wrote:

> I am going to merge your patch for the time being. However, I have a
> question about using rttcp in parallel with Linux tcp: are you using
> the rtnetproxy module? If yes, then testing for
> CONFIG_XENO_DRIVERS_NET_ADDON_PROXY and the rt_ip_fallback_handler
> pointer would be the way to go.


--- tcp.c ---
static struct rtsocket *rt_tcp_dest_socket(struct rtskb *skb)
{
    struct tcphdr    *th = skb->h.th;

    u32 saddr = skb->nh.iph->saddr;
    u32 daddr = skb->nh.iph->daddr;
    u32 sport = th->source;
    u32 dport = th->dest;

    u32 data_len;

    if (tcp_v4_check(skb->len, saddr, daddr,
		     csum_partial(skb->data, skb->len, 0))) {
	rtdm_printk("rttcp: invalid TCP packet checksum, dropped\n");
	return NULL; /* Invalid checksum, drop the packet */
    }

    /* find the destination socket */
    if ((skb->sk = rt_tcp_v4_lookup(daddr, dport)) == NULL) {
	/*
	  rtdm_printk("Not found addr:0x%08x, port: 0x%04x\n", daddr, dport);
	*/
#ifdef CONFIG_RTNET_ADDON_PROXY
		if (rt_ip_fallback_handler) {
			/* If a fallback handler for IP protocol has been installed,
			 * call it. */
			rt_ip_fallback_handler(skb);
		}
#endif
		if (skb && !th->rst) {
			/* No listening socket found, send RST|ACK */
			rst_socket.saddr = daddr;
			rst_socket.daddr = saddr;
			rst_socket.sport = dport;
			rst_socket.dport = sport;

			data_len = skb->len - (th->doff << 2);

			rst_socket.sync.seq = 0;
			rst_socket.sync.ack_seq = rt_tcp_compute_ack_seq(th, data_len);

			if (rt_ip_route_output(&rst_socket.rt, daddr, saddr) == 0) {
			rt_socket_reference(&rst_socket.sock);
			rt_tcp_send(&rst_socket, TCPHDR_ACK|TCPHDR_RST);
			rtdev_dereference(rst_socket.rt.rtdev);
			}
		}
    }

    return skb->sk;
}

Re: [Xenomai] A fix to few generic issues in rttcp stack

[Gilles Chanteperdrix]

On Mon, May 02, 2016 at 06:32:46AM +0000, Matti Suominen wrote:
> Hi and sorry for delay.
> 
> Yes, we are using rtnetproxy and I am now testing fallback handler. I'm 
> not sure am I using it right way but it seems to work correctly. Code 
> snippet at end of message.
> 
> Matti Suominen
> Wapice Oy
> www.wapice.com
> 
> On Sun, Apr 03, 2016 at 09:09:48PM +0200, Gilles Chanteperdrix wrote:
> 
> > I am going to merge your patch for the time being. However, I have a
> > question about using rttcp in parallel with Linux tcp: are you using
> > the rtnetproxy module? If yes, then testing for
> > CONFIG_XENO_DRIVERS_NET_ADDON_PROXY and the rt_ip_fallback_handler
> > pointer would be the way to go.
> 
> 
> --- tcp.c ---
> static struct rtsocket *rt_tcp_dest_socket(struct rtskb *skb)
> {
>     struct tcphdr    *th = skb->h.th;
> 
>     u32 saddr = skb->nh.iph->saddr;
>     u32 daddr = skb->nh.iph->daddr;
>     u32 sport = th->source;
>     u32 dport = th->dest;
> 
>     u32 data_len;
> 
>     if (tcp_v4_check(skb->len, saddr, daddr,
> 		     csum_partial(skb->data, skb->len, 0))) {
> 	rtdm_printk("rttcp: invalid TCP packet checksum, dropped\n");
> 	return NULL; /* Invalid checksum, drop the packet */
>     }
> 
>     /* find the destination socket */
>     if ((skb->sk = rt_tcp_v4_lookup(daddr, dport)) == NULL) {
> 	/*
> 	  rtdm_printk("Not found addr:0x%08x, port: 0x%04x\n", daddr, dport);
> 	*/
> #ifdef CONFIG_RTNET_ADDON_PROXY
> 		if (rt_ip_fallback_handler) {
> 			/* If a fallback handler for IP protocol has been installed,
> 			 * call it. */
> 			rt_ip_fallback_handler(skb);
> 		}
> #endif

You do not need to call rt_ip_fallback_handler, it will be called by
ip_input.c when dest_socket returns NULL. However, you need to
return, otherwise the RST|ACK is sent, which is what you wanted to
avoid.

-- 
					    Gilles.
https://click-hack.org

Re: [Xenomai] A fix to few generic issues in rttcp stack

[Matti Suominen]

From: Gilles Chanteperdrix [mailto:gilles.chanteperdrix@xenomai.org] 
Sent: 2. toukokuuta 2016 10:11
Subject: Re: [Xenomai] A fix to few generic issues in rttcp stack

>You do not need to call rt_ip_fallback_handler, it will be called by
>ip_input.c when dest_socket returns NULL. However, you need to
>return, otherwise the RST|ACK is sent, which is what you wanted to
>avoid.

Okay, there was a little misunderstanding. We tried now replacing (in 
our latest code snippet) the fallback_handler call with just retun NULL 
and it works as well. We agree that this is the correct solution.

Matti Suominen
Wapice Oy
www.wapice.com 

--- tcp.c ---
static struct rtsocket *rt_tcp_dest_socket(struct rtskb *skb)
{
    struct tcphdr    *th = skb->h.th;

    u32 saddr = skb->nh.iph->saddr;
    u32 daddr = skb->nh.iph->daddr;
    u32 sport = th->source;
    u32 dport = th->dest;

    u32 data_len;

    if (tcp_v4_check(skb->len, saddr, daddr,
		     csum_partial(skb->data, skb->len, 0))) {
	rtdm_printk("rttcp: invalid TCP packet checksum, dropped\n");
	return NULL; /* Invalid checksum, drop the packet */
    }

    /* find the destination socket */
    if ((skb->sk = rt_tcp_v4_lookup(daddr, dport)) == NULL) {
	/*
	  rtdm_printk("Not found addr:0x%08x, port: 0x%04x\n", daddr, dport);
	*/
#ifdef CONFIG_RTNET_ADDON_PROXY
		if (rt_ip_fallback_handler) {
			/* If a fallback handler for IP protocol has been installed,
			 * call it. */
			return NULL;
		}
#endif
		if (skb && !th->rst) {
			/* No listening socket found, send RST|ACK */
			rst_socket.saddr = daddr;
			rst_socket.daddr = saddr;
			rst_socket.sport = dport;
			rst_socket.dport = sport;

			data_len = skb->len - (th->doff << 2);

			rst_socket.sync.seq = 0;
			rst_socket.sync.ack_seq = rt_tcp_compute_ack_seq(th, data_len);

			if (rt_ip_route_output(&rst_socket.rt, daddr, saddr) == 0) {
			rt_socket_reference(&rst_socket.sock);
			rt_tcp_send(&rst_socket, TCPHDR_ACK|TCPHDR_RST);
			rtdev_dereference(rst_socket.rt.rtdev);
			}
		}
    }

    return skb->sk;
}

Re: [Xenomai] A fix to few generic issues in rttcp stack

[Gilles Chanteperdrix]

On Tue, May 03, 2016 at 04:54:30AM +0000, Matti Suominen wrote:
> From: Gilles Chanteperdrix [mailto:gilles.chanteperdrix@xenomai.org] 
> Sent: 2. toukokuuta 2016 10:11
> Subject: Re: [Xenomai] A fix to few generic issues in rttcp stack
> 
> >You do not need to call rt_ip_fallback_handler, it will be called by
> >ip_input.c when dest_socket returns NULL. However, you need to
> >return, otherwise the RST|ACK is sent, which is what you wanted to
> >avoid.
> 
> Okay, there was a little misunderstanding. We tried now replacing (in 
> our latest code snippet) the fallback_handler call with just retun NULL 
> and it works as well. We agree that this is the correct solution.

Will merge, thanks.

-- 
					    Gilles.
https://click-hack.org

[Xenomai] PowerPC hardware watchpoints unstable with GDB and Xenomai

[Lassi Niemistö]

Hello,

Note: please keep gdb@sourceware.org as a recipient - I tried to post on both but xenomai rejected due to missing membership and I do not want to re-post on gdb.. 

I am trying to debug (applications, not kernel) using hardware watchpoints (memory breakpoints) with 

GDB: 7.10.1 and 7.11 (local mode)
Processor: PowerPC e300c3 (MPC8309E)
Xenomai: 2.6.1
Linux: 3.1.10-ipipe

I have two issues:

1) If I set up a while(1) loop with (a global) counter and sleep, and place the watchpoint on the counter variable, I get GDB to break nicely to each increment if I use usleep() in my loop. Instead, if I use rt_task_sleep(), the watchpoint may go unnoticed for up to 100 loops, then trigger, then again go unnoticed for 5 loops and so on. Triggering happens on correct line, though. This is reproducable with a simple C main with only rt_task_shadow and the loop. 

2) The watchpoint seems to go broken (at least sometimes) when new (rt) threads are being created. The GDB still thinks the watchpoint exists, but it never gets triggered. If I remove the wp and create it again, it starts triggering. This finding I will investigate further and see if it is reproducable with simple test code.

Are these known issues? I found only little information about using watchpoints on xenomai, so is it something rarely used? Or is the PowerPC integration the broken part here? GDB, in turn, has had some patches on the hardware-watchpoint inheritance at thread creation, so I would assume that it at least "should work". Any hints for debugging the issue further?

BR,
Lassi Niemistö
Wapice Oy
Vaasa, Finland

Re: [Xenomai] PowerPC hardware watchpoints unstable with GDB and Xenomai

[Gilles Chanteperdrix]

On Mon, Jun 13, 2016 at 02:01:10PM +0000, Lassi Niemistö wrote:
> Hello,
> 
> Note: please keep gdb@sourceware.org as a recipient - I tried to post on both but xenomai rejected due to missing membership and I do not want to re-post on gdb.. 

Whether or not you should contact the gdb mailing list is simple: if
you get the issue without Xenomai, then it is a gdb issue, if you
get the issue only with Xenomai, then it is a Xenomai issue, and
there is no reason to bother the gdb mailing list.

> 
> I am trying to debug (applications, not kernel) using hardware watchpoints (memory breakpoints) with 
> 
> GDB: 7.10.1 and 7.11 (local mode)
> Processor: PowerPC e300c3 (MPC8309E)
> Xenomai: 2.6.1

Xenomai 2.6.1 is pretty old, a lot of things have been fixed since
then, including issues with gdb. So, do you get the same issue with
Xenomai 2.6.4?

-- 
					    Gilles.
https://click-hack.org

Re: [Xenomai] PowerPC hardware watchpoints unstable with GDB and Xenomai

[Lassi Niemistö]

> Whether or not you should contact the gdb mailing list is simple: if you get the issue without Xenomai, then it is a 
> gdb issue, if you get the issue only with Xenomai, then it is a Xenomai issue, and there is no reason to bother the 
> gdb mailing list.

Well, I had no idea which side contains the Xenomai support and I wished the GDB guys to provide some debugging hints. But if you see it most probably as a Xenomai issue, let's not cross-post it then.


> Xenomai 2.6.1 is pretty old, a lot of things have been fixed since then, including issues with gdb. So, do you get the same issue with Xenomai 2.6.4?

I will find out if we have the possibility of trying the version update with sensible effort.

A thought: I have been getting "warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available." at GDB startup, but ignored it this far as the threading has worked just fine.. Might this have anything to do with my issue, and does anyone know how to get rid of the message on Xenomai? I know how to specify the library for GDB, but which one should be the correct for Xenomai?

BR,
Lassi Niemistö
Wapice Oy
Vaasa, Finland

Re: [Xenomai] PowerPC hardware watchpoints unstable with GDB and Xenomai

[Gilles Chanteperdrix]

On Mon, Jun 13, 2016 at 03:14:05PM +0000, Lassi Niemistö wrote:
> > Whether or not you should contact the gdb mailing list is simple: if you get the issue without Xenomai, then it is a 
> > gdb issue, if you get the issue only with Xenomai, then it is a Xenomai issue, and there is no reason to bother the 
> > gdb mailing list.
> 
> Well, I had no idea which side contains the Xenomai support and I wished the GDB guys to provide some debugging hints. But if you see it most probably as a Xenomai issue, let's not cross-post it then.
> 
> 
> > Xenomai 2.6.1 is pretty old, a lot of things have been fixed since then, including issues with gdb. So, do you get the same issue with Xenomai 2.6.4?
> 
> I will find out if we have the possibility of trying the version
> update with sensible effort. 

The update should be painless: the two versions belong to the same
branch, they are ABI compatible. This means that you do not even
need to recompile your applications, you simply need to compile the
new xenomai version. Also, recompiling the kernel support should be
enough.

> 
> A thought: I have been getting "warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available." at GDB startup, but ignored it this far as the threading has worked just fine.. Might this have anything to do with my issue, and does anyone know how to get rid of the message on Xenomai? I know how to specify the library for GDB, but which one should be the correct for Xenomai?

No idea, I always debugged threaded programs with libthread_db.so,
and it has to come from the toolchain used to compile the target
gdb/gdbserver. And if using gdbserver, the cross-gdb running on host
and the gdbserver running on target should have the same version too.

-- 
					    Gilles.
https://click-hack.org