Problems with idle DCCP connections

Problems with idle DCCP connections

[Samuel Jero]

Hello All,

I am a researcher at Ohio University looking to encapsulate a special
point-to-point Delay Tolerant Networking protocol in DCCP instead of UDP
to take advantage of DCCP's congestion control.

In attempting to set this up I have come across some very odd behavior
from DCCP. My goal is to open one DCCP connection and then keep it open
even when the encapsulated protocol is idle (this avoids having a
three-way handshake for every packet sent). However, in my tests, DCCP
clamps the sending rate down to three packets per second after a short
idle period (a few seconds) and then never attempts to increase that
rate. This seems to be a very unusual and counter-intuitive behavior.
Has anyone else seen similar behavior? This seems to contradict RFC4341
section 5.1 which says that CCID2 should slow start after an idle
period. Is this not implemented yet or is it a bug?

My test machines are running Ubuntu 10.04. Uname -a
"2.6.32-24-generic-pae #38-Ubuntu SMP Mon Jul 5 10:54:21 UTC 2010 i686
GNU/Linux". I have seen identical behavior with Ubuntu 9.10 (Linux
2.6.31). I'm using CCID2 "TCP-like Congestion control" since I want as
much bandwidth as possible, even if that means rapid fluctuations.

Thanks.

Sincerely,
Samuel Jero

Re: Problems with idle DCCP connections

[Ian McDonald]

Hi Samuel,

My main comment about DCCP in Linux is that it is still very
experimental. There is a lot of good work gone into it, but there are
also still some problems.

I used it for my research also (see my website and click on research)
and did quite a bit to get it working for me but I largely used CCID3
and fixed up aspects of that and got code merged for that. I would
encourage you to have a go at seeing if you can fix it (provided
no-one else does first!) as this will help you in your future research
if you are able to work on this aspect.

Best regards

Ian
--
twitter imcdnzl
web http://www.next-genit.co.uk



On 30 July 2010 22:07, Samuel Jero <sj323707@ohio.edu> wrote:
>
> Hello All,
>
> I am a researcher at Ohio University looking to encapsulate a special
> point-to-point Delay Tolerant Networking protocol in DCCP instead of UDP
> to take advantage of DCCP's congestion control.
>
> In attempting to set this up I have come across some very odd behavior
> from DCCP. My goal is to open one DCCP connection and then keep it open
> even when the encapsulated protocol is idle (this avoids having a
> three-way handshake for every packet sent). However, in my tests, DCCP
> clamps the sending rate down to three packets per second after a short
> idle period (a few seconds) and then never attempts to increase that
> rate. This seems to be a very unusual and counter-intuitive behavior.
> Has anyone else seen similar behavior? This seems to contradict RFC4341
> section 5.1 which says that CCID2 should slow start after an idle
> period. Is this not implemented yet or is it a bug?
>
> My test machines are running Ubuntu 10.04. Uname -a
> "2.6.32-24-generic-pae #38-Ubuntu SMP Mon Jul 5 10:54:21 UTC 2010 i686
> GNU/Linux". I have seen identical behavior with Ubuntu 9.10 (Linux
> 2.6.31). I'm using CCID2 "TCP-like Congestion control" since I want as
> much bandwidth as possible, even if that means rapid fluctuations.
>
> Thanks.
>
> Sincerely,
> Samuel Jero
>
> --
> To unsubscribe from this list: send the line "unsubscribe dccp" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Problems with idle DCCP connections

[Arnaldo Carvalho de Melo]

Em Mon, Aug 02, 2010 at 10:19:15AM +0100, Ian McDonald escreveu:
> Hi Samuel,
> 
> My main comment about DCCP in Linux is that it is still very
> experimental. There is a lot of good work gone into it, but there are
> also still some problems.
> 
> I used it for my research also (see my website and click on research)
> and did quite a bit to get it working for me but I largely used CCID3
> and fixed up aspects of that and got code merged for that. I would
> encourage you to have a go at seeing if you can fix it (provided
> no-one else does first!) as this will help you in your future research
> if you are able to work on this aspect.

Exactly, it will get as much traction and improve as people like Ian,
Andrea, Gerrit, Leandro, etc, did and you are encouraged to do: while
doing some research, fix bugs in the current implementation.
 
- Arnaldo

Re: Problems with idle DCCP connections

[Gerrit Renker]

| In attempting to set this up I have come across some very odd behavior
| from DCCP. My goal is to open one DCCP connection and then keep it open
| even when the encapsulated protocol is idle (this avoids having a
| three-way handshake for every packet sent). However, in my tests, DCCP
| clamps the sending rate down to three packets per second after a short
| idle period (a few seconds) and then never attempts to increase that
| rate. This seems to be a very unusual and counter-intuitive behavior.
| Has anyone else seen similar behavior? This seems to contradict RFC4341
| section 5.1 which says that CCID2 should slow start after an idle
| period. Is this not implemented yet or is it a bug?
| 
It is probably a bug, the state of the CCID-2 code is not as good as the
CCID-3 code, the latter has received more attention.

But you will find several bug CCID-2 bug fixes already contained in the
DCCP test tree available at
     git://eden-feed.erg.abdn.ac.uk/dccp_exp   [subtree 'dccp' or 'ccid4']
     http://[same-address]

and I would like to ask you to use this tree for testing and bug-fixing, 
since it is ahead of the mainline tree. I know from testing that the 
mainline CCID-2 code does not perform very well.

In addition, there is a CCID-248 ("UDP-like") test-CCID available in the
test tree, which does not perform congestion control. This might be useful
for your tests which need as much bandwidth as possible. 

Please let us know any bugs or problems encountered/fixed.

Re: Problems with idle DCCP connections

[Samuel Jero]

> It is probably a bug, the state of the CCID-2 code is not as good as the
> CCID-3 code, the latter has received more attention.
> 
> But you will find several bug CCID-2 bug fixes already contained in the
> DCCP test tree available at
>      git://eden-feed.erg.abdn.ac.uk/dccp_exp   [subtree 'dccp' or 'ccid4']
>      http://[same-address]
> 
> and I would like to ask you to use this tree for testing and bug-fixing, 
> since it is ahead of the mainline tree. I know from testing that the 
> mainline CCID-2 code does not perform very well.

Thanks for the information. I'm compiling the test tree right now. 

I have also done some testing with CCID-3 (Linux 2.6.35 kernel.org tree)
and the same symptoms are present there, but much, much less often.
Maybe once every for every 2 hours the dccp connection is transferring
data, as apposed to once for every 2 minutes in CCID-2.

Samuel Jero

Re: Problems with idle DCCP connections

[gerrit]

> I have also done some testing with CCID-3 (Linux 2.6.35 kernel.org tree)
> and the same symptoms are present there, but much, much less often.
> Maybe once every for every 2 hours the dccp connection is transferring
> data, as apposed to once for every 2 minutes in CCID-2.
>
Looking back at your other post, it is likely that you will see the
same behaviour also with CCID-3 in the test tree.

If I understand your setting of Delay Tolerant Networking correctly, then
this is a connection which has long idle phases.

The old TFRC specification (RFC 3448) penalized such connections (see
RFC 5348, 9.1), the new specification has support for data-limited
intervals (RFC 5348, 4.3), but the code is still at the in-between state
of rfc3448bis which preceded RFC 5348.

Hence it is a good time to add this missing support.

In the meantime, for testing CCID-248 might be of interest
    CONFIG_IP_DCCP_CCID0=y
and then, after modprobe -v dccp,
    echo  248 > /proc/sys/net/dccp/default/tx_ccid
    echo  248 > /proc/sys/net/dccp/default/rx_ccid

Re: Problems with idle DCCP connections

[Gerrit Renker]

Samuel, -
| However, in my tests, DCCP clamps the sending rate down to three packets
| per second after a short idle period (a few seconds) and then never
| attempts to increase that rate. This seems to be a very unusual and
| counter-intuitive behavior.
| Has anyone else seen similar behavior? This seems to contradict RFC4341
| section 5.1 which says that CCID2 should slow start after an idle
| period. Is this not implemented yet or is it a bug?

I took another look through the patches and found an implementation of
TCP Congestion Window Validation (CWV, RFC 2861) at home. This implements
slow-start after idle as a byproduct. 

When I originally tested this I found that CWV proved very worthwhile to
have in CCID-2.

Could I thus ask you to pull 
      git://eden-feed.erg.abdn.ac.uk/dccp_exp  [subtree 'dccp']

once again and retry with CWV (it is the ccid2_do_cwv module parameter,
per default it is on)? To me it seems likely that this could be helpful
for what you are trying to do.

The patch is at 
    http://eden-feed.erg.abdn.ac.uk/cgi-bin/gitweb.cgi?p‹cp_exp.git;a=commitdiff;hû67a3025b17d07d72d30bc48d08accb9a6487f3

and will also, per forma, be submitted to the list just now.
--
To unsubscribe from this list: send the line "unsubscribe dccp" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Problems with idle DCCP connections

[Samuel Jero]

I finally got the testing tree build and running yesterday. In running
my tests on it today, I discovered, to my surprise, that the problem I
was having with DCCP clamping down on the sending rate after idle
periods is not present in either CCID-2 or CCID-3 of the testing tree
from 8/11/2010.

I do, however, see a bunch of messages like the following
in /var/log/messages when using CCID-3: "dccp_sane_rtt: RTT sample
8644424 out of bounds!"

> Could I thus ask you to pull 
>       git://eden-feed.erg.abdn.ac.uk/dccp_exp  [subtree 'dccp']
> 
> once again and retry with CWV (it is the ccid2_do_cwv module parameter,
> per default it is on)? To me it seems likely that this could be helpful
> for what you are trying to do.

I will pull the current tree and double check it; however, my problem
(or at least the symptoms) appears to be fixed without CWV. I'll let you
know how the new tree works by Monday or Tuesday.

Thank you for your help.

Samuel Jero

Re: Problems with idle DCCP connections

[Gerrit Renker]

Hi Samuel,

thank you very much for testing.

| I do, however, see a bunch of messages like the following
| in /var/log/messages when using CCID-3: "dccp_sane_rtt: RTT sample
| 8644424 out of bounds!"
| 
This message warns that the network has an unusually high RTT (8.64 seconds).

The warning is issued whenever the RTT exceeds 3 seconds, to avoid that the
sending rate of CCID-3 is reduced too much (since X is proportional to s/RTT).

 #define DCCP_SANE_RTT_MAX       (3 * USEC_PER_SEC)

| I will pull the current tree and double check it; however, my problem
| (or at least the symptoms) appears to be fixed without CWV. I'll let you
| know how the new tree works by Monday or Tuesday.
I would be very interested in any performance reports and comments.

Re: Problems with idle DCCP connections

[Samuel Jero]

> | I do, however, see a bunch of messages like the following
> | in /var/log/messages when using CCID-3: "dccp_sane_rtt: RTT sample
> | 8644424 out of bounds!"
> | 
> This message warns that the network has an unusually high RTT (8.64 seconds).
> 
> The warning is issued whenever the RTT exceeds 3 seconds, to avoid that the
> sending rate of CCID-3 is reduced too much (since X is proportional to s/RTT).
> 
>  #define DCCP_SANE_RTT_MAX       (3 * USEC_PER_SEC)

My network RTT is nowhere near 8 seconds; it should be ~1ms... I'm
running my tests between VMs on the same machine... This must be a
result of the TFRC algorithm you mentioned that penalizes connections
with idle periods?

> 
> | I will pull the current tree and double check it; however, my problem
> | (or at least the symptoms) appears to be fixed without CWV. I'll let you
> | know how the new tree works by Monday or Tuesday.

The current tree is also working perfectly. I ran several tests with
CCID-2 that had 5 second idle periods every few seconds and DCCP didn't
clamp down the sending rate. The transfer rate during the connected
periods was ~400Kbps, but that's a result of my research software, not
DCCP.

Samuel Jero

Re: Problems with idle DCCP connections

[gerrit]

>> | I do, however, see a bunch of messages like the following
>> | in /var/log/messages when using CCID-3: "dccp_sane_rtt: RTT sample
>> | 8644424 out of bounds!"
>> |
>> This message warns that the network has an unusually high RTT (8.64
>> seconds).
>>
>> The warning is issued whenever the RTT exceeds 3 seconds, to avoid that
>> the
>> sending rate of CCID-3 is reduced too much (since X is proportional to
>> s/RTT).
>>
>>  #define DCCP_SANE_RTT_MAX       (3 * USEC_PER_SEC)
>
> My network RTT is nowhere near 8 seconds; it should be ~1ms... I'm
> running my tests between VMs on the same machine... This must be a
> result of the TFRC algorithm you mentioned that penalizes connections
> with idle periods?
>
I am very interested in further details -- in particular, was it at the
receiver where this happened? If yes, then it is a bug in the way RTTs
are measured, and I would like to isolate its causes.

I think it very likely that this happened at the receiver: the sender
uses proper timestamps, which on (virtual) LANs with small RTTs are
rounded up to 0.1 millisecond.

What I think the bug is:
 * receiver uses CCVal window counter to estimate RTT
 * sender idles between sends
 * nofeedback timer triggers at sender, reduces rate, increases packet gap
 * receiver sees the inter-packet gap doubling and wrongly interprets this
   as inflated RTT

I'd be grateful for any more information to verify this.