XTP for 2.6.25

XTP for 2.6.25

[Shigeo N]

I implemented Xpress Transport Protocol 4.0 for Linux kernel 2.6.25.
Does anyone test it and give me feedback to enhance together ?

I tested in the network where UDP throughput is 29Mbps, then TCP
throughput was 13Mbps, but XTP's reached to 25Mbps.

The source is here.
<http://downloads.sourceforge.net/linux-xtp/linux-xtp.0.3.1.tar.gz?use_mirror=osdn>

XTP specification is here.
<http://www.packeteer.com/resources/prod-sol/XTP.pdf>

Multicast function is not implemented yet.

Thanks,
Shigeo N

Re: XTP for 2.6.25

[David Miller]

From: "Shigeo N" <shigeonx@gmail.com>
Date: Thu, 24 Apr 2008 19:14:38 +0900

[ netdev added to CC: ]

> I implemented Xpress Transport Protocol 4.0 for Linux kernel 2.6.25.
> Does anyone test it and give me feedback to enhance together ?
> 
> I tested in the network where UDP throughput is 29Mbps, then TCP
> throughput was 13Mbps, but XTP's reached to 25Mbps.
> 
> The source is here.
> <http://downloads.sourceforge.net/linux-xtp/linux-xtp.0.3.1.tar.gz?use_mirror=osdn>
> 
> XTP specification is here.
> <http://www.packeteer.com/resources/prod-sol/XTP.pdf>
> 
> Multicast function is not implemented yet.

Please discuss your work on netdev@vger.kernel.org, which is where
the Linux networking developers are listening.

Thank you.

Re: XTP for 2.6.25

[Shigeo N]

My motivation to implement XTP for Linux kernel is that XTP can be
better protocol than TCP in broad-band network, because XTP is free
from BD power limitation by separation of rate and flow control.

As far as I know, Packeteer's SkyX is only the product that implements
XTP. SkyX is mainly used for Satellite network which delay is too
long, but nowadays SkyX is used for broad-band network because XTP is
of great advantege to network delay in broad-band network.

Linux based network product is popular now, so adding XTP to Linux
kernel is useful for someone who develops Linux router, server and
etc, I think.

Thanks
Shigeo

On Thu, Apr 24, 2008 at 7:30 PM, David Miller <davem@davemloft.net> wrote:
> From: "Shigeo N" <shigeonx@gmail.com>
>  Date: Thu, 24 Apr 2008 19:14:38 +0900
>
>  [ netdev added to CC: ]
>
>
>
>  > I implemented Xpress Transport Protocol 4.0 for Linux kernel 2.6.25.
>  > Does anyone test it and give me feedback to enhance together ?
>  >
>  > I tested in the network where UDP throughput is 29Mbps, then TCP
>  > throughput was 13Mbps, but XTP's reached to 25Mbps.
>  >
>  > The source is here.
>  > <http://downloads.sourceforge.net/linux-xtp/linux-xtp.0.3.1.tar.gz?use_mirror=osdn>
>  >
>  > XTP specification is here.
>  > <http://www.packeteer.com/resources/prod-sol/XTP.pdf>
>  >
>  > Multicast function is not implemented yet.
>
>  Please discuss your work on netdev@vger.kernel.org, which is where
>  the Linux networking developers are listening.
>
>  Thank you.
>

Re: XTP for 2.6.25

[Andi Kleen]

"Shigeo N" <shigeonx@gmail.com> writes:
>
> I tested in the network where UDP throughput is 29Mbps, then TCP
> throughput was 13Mbps, but XTP's reached to 25Mbps.

One interesting question is why TCP was so much slower than UDP
on your test. It shouldn't be on a fair test setup. 

Please post details. Was the network losing packets?

New protocols might be interesting, but even more interesting is to
fix any (real) problems in existing protocols.

-Andi

Re: XTP for 2.6.25

[Shigeo N]

Now I have connected 2 hosts directly, and evaluate the each throughput.
Then all the results of UDP, TCP and XTP are the same and 94Mbps. (My
netwrok is 100Base/TX).

In this case round-trip time between 2 hosts is less than 0.1ms
because they are directly connected. But my previouse case, round-trip
time between 2 hosts are 4ms. (I use IPSEC between the security
gateways to increase delay).
I think that's the reason TCP throughput is slow. If ACK packets are
delayed, sending window cannot slide and sending packets cannot be
fully bursted.

If I changes wmem size through /proc/sys/net/ipv4/tcp_wmem, TCP's
throughput may improve, but congestion control becomes more difficult
for TCP.

That is TCP's disadvantage to XTP.

Best
Shigeo

On 4/24/08, Andi Kleen <andi@firstfloor.org> wrote:
> "Shigeo N" <shigeonx@gmail.com> writes:
> >
> > I tested in the network where UDP throughput is 29Mbps, then TCP
> > throughput was 13Mbps, but XTP's reached to 25Mbps.
>
> One interesting question is why TCP was so much slower than UDP
> on your test. It shouldn't be on a fair test setup.
>
> Please post details. Was the network losing packets?
>
> New protocols might be interesting, but even more interesting is to
> fix any (real) problems in existing protocols.
>
> -Andi
>

Re: XTP for 2.6.25

[Shigeo N]

I emulated WAN delay by netem and compared the performance again.

case1) fixed delay

    <------------- 100Mbps -------------->
Host1 --------- Linux Router ---------- Host2

 In this case, just adds a fixed amount of delay to all packets going
out on Linux Router by command, "tc qdisc add dev eth0 root netem
delay 100ms".

 Here is the result.

    fixed delay    UDP/TCP/XTP throghput
  ---------------------------------------
      10ms         88/85/92 Mbps
      30ms         88/70/91 Mbps
      50ms         88/63/89 Mbps
      70ms         88/52/84 Mbps
     100ms         88/36/70 Mbps


case2) random delay

 In this case, add a fixed amount of delay + 10% random delay by
command, "tc change dev eth0 root netem delay 100ms 10ms 25%".

 Here is the result.

    fixed delay    UDP/TCP/XTP throghput
  ---------------------------------------
      10ms         88/80/92 Mbps
      30ms         88/51/91 Mbps
      50ms         88/41/89 Mbps
      70ms         88/28/80 Mbps
     100ms         88/19/66 Mbps

 TCP's perfomance looks very poor when delay is long and variable.

Thanks
Shigeo


On Thu, Apr 24, 2008 at 10:06 PM, Shigeo N <shigeonx@gmail.com> wrote:
> Now I have connected 2 hosts directly, and evaluate the each throughput.
>  Then all the results of UDP, TCP and XTP are the same and 94Mbps. (My
>  netwrok is 100Base/TX).
>
>  In this case round-trip time between 2 hosts is less than 0.1ms
>  because they are directly connected. But my previouse case, round-trip
>  time between 2 hosts are 4ms. (I use IPSEC between the security
>  gateways to increase delay).
>  I think that's the reason TCP throughput is slow. If ACK packets are
>  delayed, sending window cannot slide and sending packets cannot be
>  fully bursted.
>
>  If I changes wmem size through /proc/sys/net/ipv4/tcp_wmem, TCP's
>  throughput may improve, but congestion control becomes more difficult
>  for TCP.
>
>  That is TCP's disadvantage to XTP.
>
>  Best
>  Shigeo
>
>
>
>  On 4/24/08, Andi Kleen <andi@firstfloor.org> wrote:
>  > "Shigeo N" <shigeonx@gmail.com> writes:
>  > >
>  > > I tested in the network where UDP throughput is 29Mbps, then TCP
>  > > throughput was 13Mbps, but XTP's reached to 25Mbps.
>  >
>  > One interesting question is why TCP was so much slower than UDP
>  > on your test. It shouldn't be on a fair test setup.
>  >
>  > Please post details. Was the network losing packets?
>  >
>  > New protocols might be interesting, but even more interesting is to
>  > fix any (real) problems in existing protocols.
>  >
>  > -Andi
>  >
>

Re: XTP for 2.6.25

[Ilpo Järvinen]

On Fri, 25 Apr 2008, Shigeo N wrote:

> case2) random delay
> 
>  In this case, add a fixed amount of delay + 10% random delay by
> command, "tc change dev eth0 root netem delay 100ms 10ms 25%".
> 
>  Here is the result.
> 
>     fixed delay    UDP/TCP/XTP throghput
>   ---------------------------------------
>       10ms         88/80/92 Mbps
>       30ms         88/51/91 Mbps
>       50ms         88/41/89 Mbps
>       70ms         88/28/80 Mbps
>      100ms         88/19/66 Mbps
> 
>  TCP's perfomance looks very poor when delay is long and variable.

Where these results with 2.6.25 as well? ...John Heffner just couple of 
days ago fixed reordering related problem in TCP (in case that netem 
parametrization causes reordering), the fix is not even in 2.6.24.y, I 
think...

-- 
 i.

Re: XTP for 2.6.25

[David Miller]

From: "Ilpo_Järvinen" <ilpo.jarvinen@helsinki.fi>
Date: Fri, 25 Apr 2008 11:23:58 +0300 (EEST)

> Where these results with 2.6.25 as well? ...John Heffner just couple of 
> days ago fixed reordering related problem in TCP (in case that netem 
> parametrization causes reordering), the fix is not even in 2.6.24.y, I 
> think...

Right, I didn't submit it there yet but I will make sure it
hits -stable in the next round.

Re: XTP for 2.6.25

[Shigeo N]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=UTF-8, Size: 1192 bytes --]

One host(sender) is 2.6.25, but the other(receiver) is 2.6.18.I will evaluate again after I upgrade it to 2.6.25.
Thanks
On Fri, Apr 25, 2008 at 5:23 PM, Ilpo Järvinen<ilpo.jarvinen@helsinki.fi> wrote:> On Fri, 25 Apr 2008, Shigeo N wrote:>>  > case2) random delay>  >>  >  In this case, add a fixed amount of delay + 10% random delay by>  > command, "tc change dev eth0 root netem delay 100ms 10ms 25%".>  >>  >  Here is the result.>  >>  >     fixed delay    UDP/TCP/XTP throghput>  >   --------------------------------------->  >       10ms         88/80/92 Mbps>  >       30ms         88/51/91 Mbps>  >       50ms         88/41/89 Mbps>  >       70ms         88/28/80 Mbps>  >      100ms         88/19/66 Mbps>  >>  >  TCP's perfomance looks very poor when delay is long and variable.>>  Where these results with 2.6.25 as well? ...John Heffner just couple of>  days ago fixed reordering related problem in TCP (in case that netem>  parametrization causes reordering), the fix is not even in 2.6.24.y, I>  think...>>  -->   i.>ÿôèº{.nÇ+‰·Ÿ®‰­†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±þG«éÿŠ{ayº\x1dʇڙë,j\a­¢f£¢·hšïêÿ‘êçz_è®\x03(­éšŽŠÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?™¨è­Ú&£ø§~á¶iO•æ¬z·švØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?–I¥

Re: XTP for 2.6.25

[Shigeo N]

One host(sender) is 2.6.25, but the other(receiver) is 2.6.18.
I will evaluate again after I upgrade it to 2.6.25.

Thanks

On Fri, Apr 25, 2008 at 5:23 PM, Ilpo Järvinen
<ilpo.jarvinen@helsinki.fi> wrote:
> On Fri, 25 Apr 2008, Shigeo N wrote:
>
>  > case2) random delay
>  >
>  >  In this case, add a fixed amount of delay + 10% random delay by
>  > command, "tc change dev eth0 root netem delay 100ms 10ms 25%".
>  >
>  >  Here is the result.
>  >
>  >     fixed delay    UDP/TCP/XTP throghput
>  >   ---------------------------------------
>  >       10ms         88/80/92 Mbps
>  >       30ms         88/51/91 Mbps
>  >       50ms         88/41/89 Mbps
>  >       70ms         88/28/80 Mbps
>  >      100ms         88/19/66 Mbps
>  >
>  >  TCP's perfomance looks very poor when delay is long and variable.
>
>  Where these results with 2.6.25 as well? ...John Heffner just couple of
>  days ago fixed reordering related problem in TCP (in case that netem
>  parametrization causes reordering), the fix is not even in 2.6.24.y, I
>  think...
>
>  --
>   i.
>

Re: XTP for 2.6.25

[David Miller]

From: "Shigeo N" <shigeonx@gmail.com>
Date: Fri, 25 Apr 2008 17:51:49 +0900

> One host(sender) is 2.6.25, but the other(receiver) is 2.6.18.
> I will evaluate again after I upgrade it to 2.6.25.

The bug fix isn't in 2.6.25 either.

Here is the patch in question, please add it to the kernel
during your tests.

>From 3c0390e1a6d8a5d0cdae57959f0be37a236b474d Mon Sep 17 00:00:00 2001
From: John Heffner <jheffner@napa.(none)>
Date: Fri, 25 Apr 2008 01:43:57 -0700
Subject: [PATCH] [TCP]: Increase the max_burst threshold from 3 to tp->reordering.

[ Upstream commit: dd9e0dda66ba38a2ddd1405ac279894260dc5c36 ]

This change is necessary to allow cwnd to grow during persistent
reordering.  Cwnd moderation is applied when in the disorder state
and an ack that fills the hole comes in.  If the hole was greater
than 3 packets, but less than tp->reordering, cwnd will shrink when
it should not have.

Signed-off-by: John Heffner <jheffner@napa.(none)>
Signed-off-by: David S. Miller <davem@davemloft.net>
---
 include/net/tcp.h |    7 +++++--
 1 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/include/net/tcp.h b/include/net/tcp.h
index 4fd3eb2..d69d12e 100644
--- a/include/net/tcp.h
+++ b/include/net/tcp.h
@@ -776,11 +776,14 @@ extern void tcp_enter_cwr(struct sock *sk, const int set_ssthresh);
 extern __u32 tcp_init_cwnd(struct tcp_sock *tp, struct dst_entry *dst);
 
 /* Slow start with delack produces 3 packets of burst, so that
- * it is safe "de facto".
+ * it is safe "de facto".  This will be the default - same as
+ * the default reordering threshold - but if reordering increases,
+ * we must be able to allow cwnd to burst at least this much in order
+ * to not pull it back when holes are filled.
  */
 static __inline__ __u32 tcp_max_burst(const struct tcp_sock *tp)
 {
-	return 3;
+	return tp->reordering;
 }
 
 /* Returns end sequence number of the receiver's advertised window */
-- 
1.5.5.1.57.g5909c

Re: XTP for 2.6.25

[Ilpo Järvinen]

On Fri, 25 Apr 2008, David Miller wrote:

> From: "Shigeo N" <shigeonx@gmail.com>
> Date: Fri, 25 Apr 2008 17:51:49 +0900
> 
> > One host(sender) is 2.6.25, but the other(receiver) is 2.6.18.
> > I will evaluate again after I upgrade it to 2.6.25.
> 
> The bug fix isn't in 2.6.25 either.

Sure it is, or git is lying:

$ git-describe dd9e0dda66ba38a2ddd1405ac279894260dc5c36
v2.6.25-rc8-1190-gdd9e0dd

> [ Upstream commit: dd9e0dda66ba38a2ddd1405ac279894260dc5c36 ]


-- 
 i.

Re: XTP for 2.6.25

[David Miller]

From: "Ilpo_Järvinen" <ilpo.jarvinen@helsinki.fi>
Date: Fri, 25 Apr 2008 16:05:51 +0300 (EEST)

> On Fri, 25 Apr 2008, David Miller wrote:
> 
> > From: "Shigeo N" <shigeonx@gmail.com>
> > Date: Fri, 25 Apr 2008 17:51:49 +0900
> > 
> > > One host(sender) is 2.6.25, but the other(receiver) is 2.6.18.
> > > I will evaluate again after I upgrade it to 2.6.25.
> > 
> > The bug fix isn't in 2.6.25 either.
> 
> Sure it is, or git is lying:
> 
> $ git-describe dd9e0dda66ba38a2ddd1405ac279894260dc5c36
> v2.6.25-rc8-1190-gdd9e0dd

I just submitted the TCP reorder patch for 2.6.25-stable, it applied
on top of a tree for which I did:

	git checkout -b linux25 v2.6.25

so something is up :-)

Re: XTP for 2.6.25

[Ilpo Järvinen]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1092 bytes --]

On Fri, 25 Apr 2008, David Miller wrote:

> From: "Ilpo_Järvinen" <ilpo.jarvinen@helsinki.fi>
> Date: Fri, 25 Apr 2008 16:05:51 +0300 (EEST)
> 
> > On Fri, 25 Apr 2008, David Miller wrote:
> > 
> > > From: "Shigeo N" <shigeonx@gmail.com>
> > > Date: Fri, 25 Apr 2008 17:51:49 +0900
> > > 
> > > > One host(sender) is 2.6.25, but the other(receiver) is 2.6.18.
> > > > I will evaluate again after I upgrade it to 2.6.25.
> > > 
> > > The bug fix isn't in 2.6.25 either.
> > 
> > Sure it is, or git is lying:
> > 
> > $ git-describe dd9e0dda66ba38a2ddd1405ac279894260dc5c36
> > v2.6.25-rc8-1190-gdd9e0dd
> 
> I just submitted the TCP reorder patch for 2.6.25-stable, it applied
> on top of a tree for which I did:
> 
> 	git checkout -b linux25 v2.6.25
> 
> so something is up :-)

Of course I was wrong :-), misintepreted git-describe's result as it 
cannot prove what I was trying to make it to prove (basically it was
reporting just the latest net-2.6 -> net-2.6.26 merge of yours, while 
that doesn't tell anything about entry of a commit to Linus' tree).

...Sorry about the noise.

-- 
 i.

Re: XTP for 2.6.25

[Shigeo N]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=UTF-8, Size: 2407 bytes --]

I tested again on Linux 2.6.25 with TCP reordering patch.Of course, TCP performance was improved.
Here is the result.
 case2-2) random delay (with TCP reordering patch)
  In this case, add a fixed amount of delay + 10% random delay by command,  "tc change dev eth0 root netem delay 100ms 10ms 25%".
    fixed delay    UDP/TCP/XTP throghput  ---------------------------------------      10ms         88/85/92 Mbps      30ms         88/67/91 Mbps      50ms         88/58/89 Mbps      70ms         88/43/80 Mbps     100ms         88/26/66 Mbps
 case2) random delay (without TCP reordering patch)
  In this case, add a fixed amount of delay + 10% random delay by command,  "tc change dev eth0 root netem delay 100ms 10ms 25%".
    fixed delay    UDP/TCP/XTP throghput  ---------------------------------------      10ms         88/80/92 Mbps      30ms         88/51/91 Mbps      50ms         88/41/89 Mbps      70ms         88/28/80 Mbps     100ms         88/19/66 Mbps
  case1) fixed delay
    In this case, just adds a fixed amount of delay to all packets going out by  command, "tc qdisc add dev eth0 root netem delay 100ms".
    fixed delay    UDP/TCP/XTP throghput  ---------------------------------------      10ms         88/85/92 Mbps      30ms         88/70/91 Mbps      50ms         88/63/89 Mbps      70ms         88/52/84 Mbps     100ms         88/36/70 Mbps
Thanks,
On Fri, Apr 25, 2008 at 5:23 PM, Ilpo Järvinen<ilpo.jarvinen@helsinki.fi> wrote:> On Fri, 25 Apr 2008, Shigeo N wrote:>>> case2) random delay>>>>  In this case, add a fixed amount of delay + 10% random delay by>> command, "tc change dev eth0 root netem delay 100ms 10ms 25%".>>>>  Here is the result.>>>>     fixed delay    UDP/TCP/XTP throghput>>   --------------------------------------->>       10ms         88/80/92 Mbps>>       30ms         88/51/91 Mbps>>       50ms         88/41/89 Mbps>>       70ms         88/28/80 Mbps>>      100ms         88/19/66 Mbps>>>>  TCP's perfomance looks very poor when delay is long and variable.>> Where these results with 2.6.25 as well? ...John Heffner just couple of> days ago fixed reordering related problem in TCP (in case that netem> parametrization causes reordering), the fix is not even in 2.6.24.y, I> think...>> -->  i.>ÿôèº{.nÇ+‰·Ÿ®‰­†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±þG«éÿŠ{ayº\x1dʇڙë,j\a­¢f£¢·hšïêÿ‘êçz_è®\x03(­éšŽŠÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?™¨è­Ú&£ø§~á¶iO•æ¬z·švØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?–I¥

Re: XTP for 2.6.25

[Shigeo N]

I tested again on Linux 2.6.25 with TCP reordering patch.
Of course, TCP performance was improved.

Here is the result.

 case2-2) random delay (with TCP reordering patch)

  In this case, add a fixed amount of delay + 10% random delay by command,
  "tc change dev eth0 root netem delay 100ms 10ms 25%".

    fixed delay    UDP/TCP/XTP throghput
  ---------------------------------------
      10ms         88/85/92 Mbps
      30ms         88/67/91 Mbps
      50ms         88/58/89 Mbps
      70ms         88/43/80 Mbps
     100ms         88/26/66 Mbps

 case2) random delay (without TCP reordering patch)

  In this case, add a fixed amount of delay + 10% random delay by command,
  "tc change dev eth0 root netem delay 100ms 10ms 25%".

    fixed delay    UDP/TCP/XTP throghput
  ---------------------------------------
      10ms         88/80/92 Mbps
      30ms         88/51/91 Mbps
      50ms         88/41/89 Mbps
      70ms         88/28/80 Mbps
     100ms         88/19/66 Mbps

  case1) fixed delay

    In this case, just adds a fixed amount of delay to all packets going out by
  command, "tc qdisc add dev eth0 root netem delay 100ms".

    fixed delay    UDP/TCP/XTP throghput
  ---------------------------------------
      10ms         88/85/92 Mbps
      30ms         88/70/91 Mbps
      50ms         88/63/89 Mbps
      70ms         88/52/84 Mbps
     100ms         88/36/70 Mbps

Thanks,

On Fri, Apr 25, 2008 at 5:23 PM, Ilpo Järvinen
<ilpo.jarvinen@helsinki.fi> wrote:
> On Fri, 25 Apr 2008, Shigeo N wrote:
>
>> case2) random delay
>>
>>  In this case, add a fixed amount of delay + 10% random delay by
>> command, "tc change dev eth0 root netem delay 100ms 10ms 25%".
>>
>>  Here is the result.
>>
>>     fixed delay    UDP/TCP/XTP throghput
>>   ---------------------------------------
>>       10ms         88/80/92 Mbps
>>       30ms         88/51/91 Mbps
>>       50ms         88/41/89 Mbps
>>       70ms         88/28/80 Mbps
>>      100ms         88/19/66 Mbps
>>
>>  TCP's perfomance looks very poor when delay is long and variable.
>
> Where these results with 2.6.25 as well? ...John Heffner just couple of
> days ago fixed reordering related problem in TCP (in case that netem
> parametrization causes reordering), the fix is not even in 2.6.24.y, I
> think...
>
> --
>  i.
>