[Announce]: Test results with latest CCID3 patch set

[Announce]: Test results with latest CCID3 patch set

[Gerrit Renker]

As promised, here are some test results using the latest (2.6.25-backported)
version of the test tree.

These are sanity checks and by no means "statistically significant". For
that it would necessary to run more tests, take averages and calculate
standard deviation / error bars.

Longer test runs (> 1/3 Min) would also be good.

Please therefore feel encouraged to run similar tests against this; on
possibly different pieces of hardware.

All runs were for 20 seconds. The figures in square brackets are the
theoretical values using the throughput equation from RFC 3448, 3.1, which
ignores IP/DCCP header sizes.


I) First run: laptop to double-CPU server via NetEm router
----------------------------------------------------------
This run was done before the latest revision of patches today, but
already including the patch set as per Friday last week.
Setup for these tests:
	* client: Pentium M 1.7 GHz laptop
	* server: 2 x Pentium III 700 MHz SMP server
	* router: 2.4 GHz PIV desktop with 2.6.24 NetEm

RTT 40ms, loss 1%
	* DCCPv6, s\x1420:        3.3  Mbps   [3.3 Mbps]
	*         s%6:         658  kbps   [602 kbps]
	*         s2:           63  kbps   [ 75 kbps]
	* DCCPv4, s\x1440:        3.5  Mbps   [3.4 Mbps]
	*         s%6:         638  kbps   [602 kbps]
	*         s2:           73  kbps   [ 75 kbps]

RTT 150ms, loss 1%
	* DCCPv6, s\x1420:         826 kbps   [891 kbps]
	*         s%6:          120 kbps   [161 kbps]
	*         s2:            18 kbps   [ 20 kbps]
	* DCCPv4, s\x1440:         714 kbps   [904 kbps]
	*         s%6:          230 kbps   [161 kbps]
	*         s2:            20 kbps   [ 20 kbps]

RTT 150ms, loss 10%
	* DCCPv6, s\x1420:        130  kbps   [132 kbps]
	*         s%6:          23  kbps   [ 24 kbps]
	*         s2:           3.8 kbps   [  3 kbps]
	* DCCPv4, s\x1440:        221  kbps   [134 kbps]
	*         s%6:          25  kbps   [ 24 kbps]
	*         s2:            3  kbps   [  3 kbps]



II) Second run: desktop-to-desktop via NetEm bridge
---------------------------------------------------
This run was done with the latest revision of patches today.
Setup for these tests:
	* client: 2.4 GHz PIV desktop
	* server: 2.8 GHz Pentium dual-core desktop
	* bridge: 2.4 GHz PIV desktop with 2.6.18-4 NetEm

RTT 40ms, loss 1%
	* DCCPv6, s\x1420:        3.07 Mbps   [3.3 Mbps]
	*         s%6:          644 kbps   [602 kbps]
	*         s2:            63 kbps   [ 75 kbps]
	* DCCPv4, s\x1440:        3.26 Mbps   [3.4 Mbps]
	*         s%6:          600 kbps   [602 kbps]
	*         s2:            67 kbps   [ 75 kbps]

RTT 150ms, loss 1%
	* DCCPv6, s\x1420:         756 kbps   [891 kbps]
	*         s%6:          148 kbps   [161 kbps]
	*         s2:            16 kbps   [ 20 kbps]
	* DCCPv4, s\x1440:         947 kbps   [904 kbps]
	*         s%6:          131 kbps   [161 kbps]
	*         s2:            23 kbps   [ 20 kbps]

RTT 150ms, loss 10%
	* DCCPv6, s\x1420:         117 kbps   [132 kbps]
	*         s%6:           28 kbps   [ 24 kbps]
	*         s2:           3.6 kbps   [  3 kbps]
	* DCCPv4, s\x1440:         187 kbps   [134 kbps]
	*         s%6:           50 kbps   [ 24 kbps]
	*         s2:             4 kbps   [  3 kbps]


I think this encouraging and asks for more challenging tests.

Re: [Announce]: Test results with latest CCID3 patch set

[Alessio Botta]

Hi all,

I am new of this mailing list and I am really interested in the
measurements you are performing with DCCP.
Which tool are you using ? Are you using Iperf for such measurements ?
In the past I used it, currently I'm working on D-ITG.
Have you ever heard about D-ITG ?
It is both a flexible traffic generator and a measurement platform which
also supports DCCP.
More precisely, as regards transport protocols, it
supports TCP, UDP, SCTP, and DCCP. It is also possible
to generate ICMP traffic.
You can find more information here:
http://www.grid.unina.it/software/ITG

I am one of the authors of such platform and I have also
performed some very preliminary tests with DCCP.

I would be very glad to have your opinion on that and I'm very interested
in improving its features, also with specific regard to the support of
transport protocols.

Best Regards.

AB


Gerrit Renker ha scritto:
> As promised, here are some test results using the latest (2.6.25-backported)
> version of the test tree.
> 
> These are sanity checks and by no means "statistically significant". For
> that it would necessary to run more tests, take averages and calculate
> standard deviation / error bars.
> 
> Longer test runs (> 1/3 Min) would also be good.
> 
> Please therefore feel encouraged to run similar tests against this; on
> possibly different pieces of hardware.
> 
> All runs were for 20 seconds. The figures in square brackets are the
> theoretical values using the throughput equation from RFC 3448, 3.1, which
> ignores IP/DCCP header sizes.
> 
> 
> I) First run: laptop to double-CPU server via NetEm router
> ----------------------------------------------------------
> This run was done before the latest revision of patches today, but
> already including the patch set as per Friday last week.
> Setup for these tests:
> 	* client: Pentium M 1.7 GHz laptop
> 	* server: 2 x Pentium III 700 MHz SMP server
> 	* router: 2.4 GHz PIV desktop with 2.6.24 NetEm
> 
> RTT 40ms, loss 1%
> 	* DCCPv6, s\x1420:        3.3  Mbps   [3.3 Mbps]
> 	*         s%6:         658  kbps   [602 kbps]
> 	*         s2:           63  kbps   [ 75 kbps]
> 	* DCCPv4, s\x1440:        3.5  Mbps   [3.4 Mbps]
> 	*         s%6:         638  kbps   [602 kbps]
> 	*         s2:           73  kbps   [ 75 kbps]
> 
> RTT 150ms, loss 1%
> 	* DCCPv6, s\x1420:         826 kbps   [891 kbps]
> 	*         s%6:          120 kbps   [161 kbps]
> 	*         s2:            18 kbps   [ 20 kbps]
> 	* DCCPv4, s\x1440:         714 kbps   [904 kbps]
> 	*         s%6:          230 kbps   [161 kbps]
> 	*         s2:            20 kbps   [ 20 kbps]
> 
> RTT 150ms, loss 10%
> 	* DCCPv6, s\x1420:        130  kbps   [132 kbps]
> 	*         s%6:          23  kbps   [ 24 kbps]
> 	*         s2:           3.8 kbps   [  3 kbps]
> 	* DCCPv4, s\x1440:        221  kbps   [134 kbps]
> 	*         s%6:          25  kbps   [ 24 kbps]
> 	*         s2:            3  kbps   [  3 kbps]
> 
> 
> 
> II) Second run: desktop-to-desktop via NetEm bridge
> ---------------------------------------------------
> This run was done with the latest revision of patches today.
> Setup for these tests:
> 	* client: 2.4 GHz PIV desktop
> 	* server: 2.8 GHz Pentium dual-core desktop
> 	* bridge: 2.4 GHz PIV desktop with 2.6.18-4 NetEm
> 
> RTT 40ms, loss 1%
> 	* DCCPv6, s\x1420:        3.07 Mbps   [3.3 Mbps]
> 	*         s%6:          644 kbps   [602 kbps]
> 	*         s2:            63 kbps   [ 75 kbps]
> 	* DCCPv4, s\x1440:        3.26 Mbps   [3.4 Mbps]
> 	*         s%6:          600 kbps   [602 kbps]
> 	*         s2:            67 kbps   [ 75 kbps]
> 
> RTT 150ms, loss 1%
> 	* DCCPv6, s\x1420:         756 kbps   [891 kbps]
> 	*         s%6:          148 kbps   [161 kbps]
> 	*         s2:            16 kbps   [ 20 kbps]
> 	* DCCPv4, s\x1440:         947 kbps   [904 kbps]
> 	*         s%6:          131 kbps   [161 kbps]
> 	*         s2:            23 kbps   [ 20 kbps]
> 
> RTT 150ms, loss 10%
> 	* DCCPv6, s\x1420:         117 kbps   [132 kbps]
> 	*         s%6:           28 kbps   [ 24 kbps]
> 	*         s2:           3.6 kbps   [  3 kbps]
> 	* DCCPv4, s\x1440:         187 kbps   [134 kbps]
> 	*         s%6:           50 kbps   [ 24 kbps]
> 	*         s2:             4 kbps   [  3 kbps]
> 
> 
> I think this encouraging and asks for more challenging tests.
> -
> To unsubscribe from this list: send the line "unsubscribe dccp" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Alessio Botta, PhD Student
Dipartimento di Informatica e Sistemistica
Università degli Studi di Napoli Federico II
Via Claudio 21 -- 80125 Napoli (Italy)
Phone: +390817683821 - Fax: +390817683816
Email: a.botta@unina.it
         abotta@napoli.consorzio-cini.it
WWW: http://wpage.unina.it/a.botta

Re: [Announce]: Test results with latest CCID3 patch set

[Gerrit Renker]

| I am new of this mailing list and I am really interested in the
| measurements you are performing with DCCP.
This was more of a regression test, as there had been recent changes in
the test tree, to see that the kernel (not userspace) still performs in
a predictable way.

| Which tool are you using ? Are you using Iperf for such measurements ?
The setup is the one from
	http://www.linux-foundation.org/en/Net:DCCP_Testing#Regression_testing
and, yes, it uses iperf.	

| Have you ever heard about D-ITG ?

| You can find more information here:
| http://www.grid.unina.it/software/ITG
| 
| I am one of the authors of such platform and I have also
| performed some very preliminary tests with DCCP.
| 
| I would be very glad to have your opinion on that and I'm very interested
| in improving its features, also with specific regard to the support of
| transport protocols.
| 
It is a very nice tool with many features. I only ran simple tests with it (version 2.6),
again only as basic sanity tests -- the throughput result was similar to the one tested with
iperf.

I think that the tool has more to offer and can help improve/extend DCCP testing.
Here is my list of points, hoping that the others will add theirs, too:

 * would be good to have a standardised set of scripts, for comparison/benchmarking

 * the built-in VoIP module only works for UDP -- is it possible to port this to DCCP?
 
 * as per previous email, more complex traffic scenarios would be good, in particular
   - switching on/off background traffic at times to observe TCP/flow-friendliness
   - running multiple DCCP flows in parallel and at overlapping times 


Gerrit   

Re: [Announce]: Test results with latest CCID3 patch set

[Arnaldo Carvalho de Melo]

Em Tue, Dec 11, 2007 at 02:21:28PM +0000, Gerrit Renker escreveu:
> | I am new of this mailing list and I am really interested in the
> | measurements you are performing with DCCP.
> This was more of a regression test, as there had been recent changes in
> the test tree, to see that the kernel (not userspace) still performs in
> a predictable way.
> 
> | Which tool are you using ? Are you using Iperf for such measurements ?
> The setup is the one from
> 	http://www.linux-foundation.org/en/Net:DCCP_Testing#Regression_testing
> and, yes, it uses iperf.	
> 
> | Have you ever heard about D-ITG ?
> 
> | You can find more information here:
> | http://www.grid.unina.it/software/ITG
> | 
> | I am one of the authors of such platform and I have also
> | performed some very preliminary tests with DCCP.
> | 
> | I would be very glad to have your opinion on that and I'm very interested
> | in improving its features, also with specific regard to the support of
> | transport protocols.
> | 
> It is a very nice tool with many features. I only ran simple tests with it (version 2.6),
> again only as basic sanity tests -- the throughput result was similar to the one tested with
> iperf.
> 
> I think that the tool has more to offer and can help improve/extend DCCP testing.
> Here is my list of points, hoping that the others will add theirs, too:
> 
>  * would be good to have a standardised set of scripts, for comparison/benchmarking
> 
>  * the built-in VoIP module only works for UDP -- is it possible to port this to DCCP?
>  
>  * as per previous email, more complex traffic scenarios would be good, in particular
>    - switching on/off background traffic at times to observe TCP/flow-friendliness
>    - running multiple DCCP flows in parallel and at overlapping times 

Does this tool records results in a database keyed by kernel
version/buildid for us to use it as a regression tool?

Something that would produce results around these lines:

"WARNING: test #23 counter #3 variance bigger than specified since the
last kernel tested (git cset 55ed793afb4a8025d33a8e6a5f2f89d5ac4d8432)!"

- Arnaldo

Re: [Announce]: Test results with latest CCID3 patch set

[Burak Gorkemli]

>
> Here is my list of points, hoping that the others will add theirs, too:
> 
>  * would be good to have a standardised set of scripts,
> for
> 
 comparison/benchmarking
> 
>  * the built-in VoIP module only works for UDP -- is it possible
> to
> 
 port this to DCCP?
>  
>  * as per previous email, more complex traffic scenarios would be
> good,
> 
 in particular
>    - switching on/off background traffic at times to
> observe
> 
 TCP/flow-friendliness
>    - running multiple DCCP flows in parallel and at overlapping times 
>

The scenario that I mostly use is limiting the bandwidth with a middlebox running TBF. However, all of the recent trees except 2.6.20final_dccp (2.6.20 patched with Ian's modifications) that I have tested fail to achieve acceptable transfer rates. Test setup is as follows, unless otherwise mentioned:

- Two Linux boxes are connected through another Linux box (middlebox) running netem. The boxes at the edge are identical machines with P4 2.4GHz and 512MB memory. Middlebox is a P4 2.6GHz with 512MB memory.
- TBF is used to limit the bandwidth. TBF buffer is set as 10000 bytes and the limit is 30000 bytes.
- 20ms constant delay is present in both sender->receiver and receiver->sender directions.
- iperf in bytestream mode is used in these tests, and the streaming duration is set as 60 seconds.
- DCCPv4 is used in the tests.


Below are the results of the two different trees:

Kernel: 2.6.20final_dccp (2.6.20 davem tree with Ian's patches - except for the best_packet_next patch. The patches applied are not the latest ones which are updated at 2nd of December, but the ones before the latest.)
Results:
Bottleneck\x1000Kbps, tx_qlen=5:    0.0-60.3 sec    6.86 MBytes   955 Kbits/sec



Bottleneck 00Kbps, tx_qlen=5:    0.0-60.1 sec    13.3 MBytes   1.86 Kbits/sec




BottleneckP00Kbps, tx_qlen=5:    0.0-60.0 sec    27.2 MBytes   3.80 Kbits/sec




Bottleneck\x10000Kbps, limitP000bytes, tx_qlen=5:    0.0-60.0 sec    43.6 MBytes   6.09 Kbits/sec






Kernel: 2.6.24-rc4 (Gerrit's latest tree):
Results (Bottleneck bandwidth, tx_qlen:    iperf server output):
Bottleneck\x1000Kbps, tx_qlen=5:    0.0-168.2 sec    107 KBytes   5.21 Kbits/sec


Bottleneck\x1000Kbps, tx_qlen=0:    0.0-147.8 sec    157 KBytes   8.71 Kbits/sec


Bottleneck 00Kbps, tx_qlen=5:    0.0-89.0 sec    138 KBytes   12.7 Kbits/sec


Bottleneck 00Kbps, tx_qlen=0:    0.0-103.9 sec    203 KBytes   16.0 Kbits/sec


BottleneckP00Kbps, tx_qlen=5:    0.0-66.6 sec    222 KBytes   27.3 Kbits/sec


BottleneckP00Kbps, tx_qlen=0:    0.0-100.2 sec    314 KBytes   25.7 Kbits/sec


Bottleneck\x10000Kbps, tx_qlen=5:    0.0-60.5 sec    1.30 MBytes   181 Kbits/sec


Bottleneck\x10000Kbps, tx_qlen=0:    0.0-65.3 sec    922 KBytes   116 Kbits/sec

While writing this mail, I noticed that Ian has updated his patch set for 2.6.20. I will use this set and repeat the tests this week, hopefully. Moreover, I also tested Ian's patches for other trees (2.6.22 and 2.6.24) but the results were not as good as 2.6.20final_dccp results, if I remember correctly. I can go over them again, if necessary.


> 
> Gerrit   
> -

Burak

Re: [Announce]: Test results with latest CCID3 patch set

[Gerrit Renker]

| The scenario that I mostly use is limiting the bandwidth with a middlebox running TBF. However, all of the recent trees 
| except 2.6.20final_dccp (2.6.20 patched with Ian's modifications) that I have tested fail to achieve acceptable transfer rates. 
Thank you for the report, but the material as-is does not help us very much. What is missing is the link speed of the
ethernet cards that lead to the middlebox, and have you monitored the loss rate p? 
Using a TBF in this setting means slowing down the link speed by a factor of 10 .. 20, and unless you are using large
FIFOs in addition to the TBF, the loss rate will very soon reach high values.

Therefore, can you please clarify what you mean by "acceptable transfer rates": maybe the scenario is not supposed to
warrant any high transfer rates at all. Which would mean expecting something where not much can be expected - as said,
without more detailed knowledge about how p reacts, these figures don't tell us very much.

Didn't you have a web page with further information?

Furthermore, I don't think that this is a question of "Ian's patches" or "Gerrit's patches": between 2.6.20 and 2.6.25-rc4
there is a huge number of changes, among these the conversion to the ktime_t interface.


| Test setup is as follows, unless otherwise mentioned:
| 

| - Two Linux boxes are connected through another Linux box (middlebox) running netem. 
|   The boxes at the edge are identical machines with P4 2.4GHz and 512MB memory. Middlebox is a P4 2.6GHz with 512MB memory.
| - TBF is used to limit the bandwidth. TBF buffer is set as 10000 bytes and the limit is 30000 bytes.
| - 20ms constant delay is present in both sender->receiver and receiver->sender directions.
| - iperf in bytestream mode is used in these tests, and the streaming duration is set as 60 seconds.
| - DCCPv4 is used in the tests.
| 
| 
| Below are the results of the two different trees:
| 
| Kernel: 2.6.20final_dccp (2.6.20 davem tree with Ian's patches - except for the best_packet_next patch. The patches applied are not the latest ones which are updated at 2nd of December, but the ones before the latest.)
<snip>
| 
| While writing this mail, I noticed that Ian has updated his patch set for 2.6.20. I will use this set and repeat the tests this week, hopefully.i
| Moreover, I also tested Ian's patches for other trees (2.6.22 and 2.6.24) but the results were not as good as 2.6.20final_dccp results, if I remember correctly.
| I can go over them again, if necessary.
| 
Yes, please: the cleanest comparison would be to take a 2.6.24 tree, and compare the patch sets on the same basis. I almost expected that the
results are not as good as the 2.6.20final_dccp -- it is an almost sure indication that the difference is not due to the patches, and that
there are other factors at work. By carefully looking at these differences, we will be able to see clearer what is happening in the above.

Again thanks a lot for posting results, hope you will be back with
further information soon,

Gerrit

Re: [Announce]: Test results with latest CCID3 patch set

[Burak Gorkemli]

> 
> | The scenario that I mostly use is limiting the bandwidth with a
> 
 middlebox running TBF. However, all of the recent trees 
> | except 2.6.20final_dccp (2.6.20 patched with Ian's
> modifications)
> 
 that I have tested fail to achieve acceptable transfer rates. 
> Thank you for the report, but the material as-is does not help us
> very much. What is missing is the link speed of the
> ethernet cards that lead to the middlebox, and have you monitored
> the 
 loss rate p? 
> Using a TBF in this setting means slowing down the link speed by
> a 
 factor of 10 .. 20, and unless you are using large
> FIFOs in addition to the TBF, the loss rate will very soon reach
> high 
 values.
> 

Link speed is 100 Mbps, and I tested DCCP under various bottleneck
bandwidths, like 1000, 2000, 5000 and 10000 Mbps. I will repeat the
tests with dccp_probe enabled, and show the results in a website,
as soon as I have time.

> Therefore, can you please clarify what you mean by "acceptable
> transfer 
 rates": maybe the scenario is not supposed to
> warrant any high transfer rates at all. Which would mean
> expecting 
 something where not much can be expected - as said,
> without more detailed knowledge about how p reacts, these figures
> don't 
 tell us very much.
> 

By acceptable transfer rates I was referring to the rates achieved
by 2.6.20final_dccp. But you are right, we cannot be sure of
the goodness of the results by comparing two DCCP stacks,
so I am giving TCP-Reno streaming results below under same 
conditions, which should be a solid benchmark:

Bottleneck\x1000Kbps:    0.0-60.6 sec    6.91 MBytes   958
 Kbits/sec
Bottleneck 00Kbps:    0.0-60.1 sec    13.7 MBytes  1.91 Mbits/sec

BottleneckP00Kbps:    0.0-60.1 sec    33.6 MBytes  4.69 Mbits/sec

Bottleneck\x10000Kbps,limitP000bytes:    0.0-60.1 sec    65.4 MBytes   9.13 Mbits/sec

The average transfer rates in the rightmost column show that
the configured bottleneck rates are achievable, hence I think
the transfer rates that 2.6.20final_dccp reaches seem acceptable,
whereas the rates of other trees are not.



> Didn't you have a web page with further information?

I will have soon, hopefully.

>
> Yes, please: the cleanest comparison would be to take a 2.6.24
> tree, 
 and compare the patch sets on the same basis. I almost expected that the
> results are not as good as the 2.6.20final_dccp -- it is an almost
> sure 
 indication that the difference is not due to the patches, and that
> there are other factors at work. By carefully looking at
> these 
 differences, we will be able to see clearer what is happening in the above.
> 

I agree with you. 2 or 3 weeks ago I applied Ian's patches to

more recent trees (2.6.22 and 2.6.24) and the results were

not as good as 2.6.20.


> Again thanks a lot for posting results, hope you will be back with
> further information soon,
> 
> Gerrit
> 

I am planning to repeat the tests focusing on 2.6.24 and
post the results - with dccp_probe figures in my website.

Burak

Re: [Announce]: Test results with latest CCID3 patch set

[Alessio Botta]

Gerrit Renker ha scritto:
> | I am new of this mailing list and I am really interested in the
> | measurements you are performing with DCCP.
> This was more of a regression test, as there had been recent changes in
> the test tree, to see that the kernel (not userspace) still performs in
> a predictable way.
> 
> | Which tool are you using ? Are you using Iperf for such measurements ?
> The setup is the one from
> 	http://www.linux-foundation.org/en/Net:DCCP_Testing#Regression_testing
> and, yes, it uses iperf.	
> 
> | Have you ever heard about D-ITG ?
> 
> | You can find more information here:
> | http://www.grid.unina.it/software/ITG
> | 
> | I am one of the authors of such platform and I have also
> | performed some very preliminary tests with DCCP.
> | 
> | I would be very glad to have your opinion on that and I'm very interested
> | in improving its features, also with specific regard to the support of
> | transport protocols.
> | 
> It is a very nice tool with many features. I only ran simple tests with it (version 2.6),
> again only as basic sanity tests -- the throughput result was similar to the one tested with
> iperf.
> 
> I think that the tool has more to offer and can help improve/extend DCCP testing.
> Here is my list of points, hoping that the others will add theirs, too:
> 
>  * would be good to have a standardised set of scripts, for comparison/benchmarking

I can provide you the necessary support to verify the scripts and also 
to start to
set up them. However, I need to know what you would like to generate.

I have seen on the web page reported above the kind of tests you are 
performing
with iperf, and I would like to provide you some some pieces of useful 
information
regarding D-ITG. This tool allows to set two random variables that control
the characteristics of traffic you generate. One of these variables 
models the Inter
Departure Time (IDT) and the other one models the Size (PS) of the 
Packets. As of now,
we support 6 random variables that are exponential, gamma, normal, 
pareto, cauchy,
and poisson. Clearly also constant IDT and PS can be set.

 From the web page reported above I have seen that with iperf you 
generate CBR
traffic but I could not find the rate.

Moreover, in order to standardize the tests I need to know how you 
measure the
obtained bit-rate. D-ITG can log at both sender and receiver sides 
different information
for each sent and received packet respectively. Therefore after the 
tests you can
obtain different information by analyzing the log files (with ITGDec). 
Regarding the
bit-rate, you can have an average value related to the complete 
generation period as
well as related to smaller time intervals.

> 
>  * the built-in VoIP module only works for UDP -- is it possible to port this to DCCP?

The VoIP traffic is produced by setting an appropriate IDT and PS which 
are basically
constant and printed out on standard output when a VoIP traffic flows is 
about to be
generated. Therefore, you can already perform VoIP tests with DCCP.
For more information you can have a look at D-ITG manual:
http://www.grid.unina.it/software/ITG/codice/D-ITG2.6-manual.pdf

If you can not find all the necessary information, just ask me.

>  
>  * as per previous email, more complex traffic scenarios would be good, in particular
>    - switching on/off background traffic at times to observe TCP/flow-friendliness

D-ITG supports multi-flow operation mode. In this case it reads the 
input information
from a script file instead of the standard input. It uses a thread for 
each requested flow
therefore the flows can have completely different characteristics in 
terms of kind of
traffic (IDT and PS models), duration, start time, and transport protocols.

>    - running multiple DCCP flows in parallel and at overlapping times 

This is already possible thanks to the script mode.

> 
> 
> Gerrit   

Alessio

Re: [Announce]: Test results with latest CCID3 patch set

[Alessio Botta]

Arnaldo Carvalho de Melo ha scritto:
> Em Tue, Dec 11, 2007 at 02:21:28PM +0000, Gerrit Renker escreveu:
>> | I am new of this mailing list and I am really interested in the
>> | measurements you are performing with DCCP.
>> This was more of a regression test, as there had been recent changes in
>> the test tree, to see that the kernel (not userspace) still performs in
>> a predictable way.
>>
>> | Which tool are you using ? Are you using Iperf for such measurements ?
>> The setup is the one from
>> 	http://www.linux-foundation.org/en/Net:DCCP_Testing#Regression_testing
>> and, yes, it uses iperf.	
>>
>> | Have you ever heard about D-ITG ?
>>
>> | You can find more information here:
>> | http://www.grid.unina.it/software/ITG
>> | 
>> | I am one of the authors of such platform and I have also
>> | performed some very preliminary tests with DCCP.
>> | 
>> | I would be very glad to have your opinion on that and I'm very interested
>> | in improving its features, also with specific regard to the support of
>> | transport protocols.
>> | 
>> It is a very nice tool with many features. I only ran simple tests with it (version 2.6),
>> again only as basic sanity tests -- the throughput result was similar to the one tested with
>> iperf.
>>
>> I think that the tool has more to offer and can help improve/extend DCCP testing.
>> Here is my list of points, hoping that the others will add theirs, too:
>>
>>  * would be good to have a standardised set of scripts, for comparison/benchmarking
>>
>>  * the built-in VoIP module only works for UDP -- is it possible to port this to DCCP?
>>  
>>  * as per previous email, more complex traffic scenarios would be good, in particular
>>    - switching on/off background traffic at times to observe TCP/flow-friendliness
>>    - running multiple DCCP flows in parallel and at overlapping times 
> 
> Does this tool records results in a database keyed by kernel
> version/buildid for us to use it as a regression tool?

No, it does not record the results in a database. This feature has to be 
added by using
some perl-like scripts.

> 
> Something that would produce results around these lines:
> 
> "WARNING: test #23 counter #3 variance bigger than specified since the
> last kernel tested (git cset 55ed793afb4a8025d33a8e6a5f2f89d5ac4d8432)!"

Yes, we can work on building some scripts for this.

> 
> - Arnaldo


Alessio

Re: [Announce]: Test results with latest CCID3 patch set

[Arnaldo Carvalho de Melo]

Em Thu, Dec 13, 2007 at 12:21:20AM +0100, Alessio Botta escreveu:
>
>
> Arnaldo Carvalho de Melo ha scritto:
>> Em Tue, Dec 11, 2007 at 02:21:28PM +0000, Gerrit Renker escreveu:
>>> | I am new of this mailing list and I am really interested in the
>>> | measurements you are performing with DCCP.
>>> This was more of a regression test, as there had been recent changes in
>>> the test tree, to see that the kernel (not userspace) still performs in
>>> a predictable way.
>>>
>>> | Which tool are you using ? Are you using Iperf for such measurements ?
>>> The setup is the one from
>>> 	http://www.linux-foundation.org/en/Net:DCCP_Testing#Regression_testing
>>> and, yes, it uses iperf.	
>>>
>>> | Have you ever heard about D-ITG ?
>>>
>>> | You can find more information here:
>>> | http://www.grid.unina.it/software/ITG
>>> | | I am one of the authors of such platform and I have also
>>> | performed some very preliminary tests with DCCP.
>>> | | I would be very glad to have your opinion on that and I'm very 
>>> interested
>>> | in improving its features, also with specific regard to the support of
>>> | transport protocols.
>>> | It is a very nice tool with many features. I only ran simple tests with 
>>> it (version 2.6),
>>> again only as basic sanity tests -- the throughput result was similar to the one tested with
>>> iperf.
>>>
>>> I think that the tool has more to offer and can help improve/extend DCCP testing.
>>> Here is my list of points, hoping that the others will add theirs, too:
>>>
>>>  * would be good to have a standardised set of scripts, for comparison/benchmarking
>>>
>>>  * the built-in VoIP module only works for UDP -- is it possible to port this to DCCP?
>>>   * as per previous email, more complex traffic scenarios would be good, 
>>> in particular
>>>    - switching on/off background traffic at times to observe TCP/flow-friendliness
>>>    - running multiple DCCP flows in parallel and at overlapping times 
>>
>> Does this tool records results in a database keyed by kernel
>> version/buildid for us to use it as a regression tool?
>
> No, it does not record the results in a database. This feature has to be 
> added by using
> some perl-like scripts.
>
>>
>> Something that would produce results around these lines:
>>
>> "WARNING: test #23 counter #3 variance bigger than specified since the
>> last kernel tested (git cset 55ed793afb4a8025d33a8e6a5f2f89d5ac4d8432)!"
>
> Yes, we can work on building some scripts for this.

That would be very nice of you and of great help to the DCCP effort.
Being able to compare metrics automatically as we go evolving the
codebase would be very, very interesting.

- Arnaldo

Re: [Announce]: Test results with latest CCID3 patch set

[Gerrit Renker]

Burak

I have tried to replay your results but can not find anything wrong with the current test kernel.
Please have a look at the results below, I repeated the same experiment on two different testbeds.

Incidentally, these results agree with the values you observed for TCP Reno. It seems that the
setup is not right, I am happy to help debugging this but I don't think that there is a problem
with the test kernel.

Can you please describe your setup - maybe something went wrong.

The first thing to check is whether you are really using the test tree - it is necessary to check
out the `dccp' subtree, if you are on the `master' instead of the `dccp' subtree, then you are
using a vanilla 2.6.24-rc5.  A detailed HowTo is on
         http://www.linux-foundation.org/en/Net:DCCP_Testing#Experimental_DCCP_source_tree
The difference can also easily be told in the Request/Response handshake, the CCIDs are now
fully negotiated; the first value of the ConfirmL/R in the DCCP-Response says which CCIDs are
used (but they are still nonetheless set via net.dccp.default.{rx,tx}_ccid).

With regard to Ian's patches: quite a few of these are earlier patches which are either already in
the kernel or part of the test tree. The test tree in addition fixes several bugs/omissions in the
loss detection / loss computation code, so in a way this means comparing apples with pears - it will
not give you a clear result. What it can help with, and I am keen to investigate that, is to make
sure that the test tree does not trade old bugs for new ones.

Similar results had been posted by Patrick Andrieux, who used TBF+delay and TBF+RED (posted this
summer). I tried similar experiments on the test tree, and they produced sane results; but this
was done earlier and not posted.


1) Two hosts via NetEm router
-----------------------------
All runs for 60 seconds from 1.7 Ghz Pentium M laptop onto 2x700Mhz Pentium-III Xeon
SMP server, connected via 2.4GHz PIV acting as router with TBF script as below.
The laptop had a e1000 interface, the PIV 2x100Mbps via crossover (one e100, one sc92301
cheapo RTL8139 clone), the server had an e100, also connected via crossover.
All kernels were 2.6.24-rc5.

tx_qlen = 5:
 1000kbps -  951 kbps   (loss initially up to 5%, going down to ~0.8% after 10 seconds)
 2000kbps - 1.90 Mbps   (loss initially up to 3%, going down to ~0.6% after circa 5 seconds)
 5000kbps - 4.72 Mbps   (loss initially peaking at 1%, converging to ~0.15% after 10 seconds)
10000kbps - 7.73 Mbps   (loss initially at 1%, quickly going down to ~0.2%)

For comparison, TCP throughput with TBF bottleneck of 10Mbps was 7.95 Mbps.

tx_qlen = 0, same setup:
 1000kbps -  950 kbps   (loss initially up to ~6%, then going down to ~0.8% after 10 seconds)
 2000kbps - 1.91 Mbps   (similar loss pattern)
 5000kbps - 4.75 Mbps   (loss initially up to 1%, then going down to ~0.3% after 2 seconds)
10000kbps - 8.90 Mbps   (loss initially up to 1%, quickly going down to ~0.1% after 1..2 seconds)


2) Two hosts via NetEm bridge
-----------------------------
  Sender:   PIV 2.4GHz with e100 NIC
  Receiver: Pentium D 2.8 Ghz with 3com 3c905 (Typhoon)
  Bridge:   PIV 2.4GHz using e100 and 3c905, using NetEm based on 2.6.18-4

All tests likewise for 60 seconds, detailed analysis of p omitted (likely very similar to the above).

tx_qlen = 5:
 1000kbps -  908 kbps
 2000kbps - 1.93 Mbps
 5000kbps - 4.72 Mbps
10000kbps - 8.68 Mbps

For comparison, TCP (cubic) throughput with TBF bottleneck of 10Mbps was 7.85 Mbps. 

tx_qlen = 0, same setup:
 1000kbps -  951 kbps
 2000kbps - 1.88 Mbps
 5000kbps - 4.75 Mbps
10000kbps - 9.15 Mbps


3) Script I used (with RTT@ms, delay=RTT/2):
-------------------------------------------
#!/bin/sh
#
# tc script for rate-control plus delay
#
# Usage: $0 <RTT in ms> <TBF-rate in kbps>
#
set -e

case "$1" in
  start)
        RTT=${2:?}; rate=${3:?}; delay=$(($RTT / 2))
	echo "Using ${delay}ms one-way delay each and a TBF rate of ${rate} Kbps "
	set -x
        tc qdisc add dev eth0 root handle 1: netem delay ${delay}ms
        tc qdisc add dev eth1 root handle 1: netem delay ${delay}ms
	# rate:   rate of incoming tokens, in bytes
	# buffer: size of the bucket, in bytes
	# limit:  number of bytes that can be queued waiting for tokens to become available
        tc qdisc add dev eth0 parent 1:1 tbf rate ${rate}kbit buffer 10000 limit 30000
        tc qdisc add dev eth1 parent 1:1 tbf rate ${rate}kbit buffer 10000 limit 30000
        ;;
  stop)
        tc qdisc del dev eth0 root
        tc qdisc del dev eth1 root
        ;;
  restart) $0 stop; shift; $0 start $@
  	;;
  show|stat*)
        tc -s qdisc ls dev eth0
        tc -s qdisc ls dev eth1
	;;
  *)    echo "Usage: $0 {start|stop|restart|show} <RTT in ms>  <TBF-rate in kbps>"
  	exit 1;;
esac

Re: [Announce]: Test results with latest CCID3 patch set

[Burak Gorkemli]

Gerrit,

I am cloning your tree with the following command:

    git-clone git://eden-feed.erg.abdn.ac.uk/dccp_exp my_dccp


But I did not check out the DCCP sub-branch, by:

    git-checkout --track -b dccp origin/dccp


However, as I type this command above, git complains saying:

    git checkout: branch dccp already exists


It this OK to say so?
 
--
Burak

----- Original Message ----
> 
> Burak
> 
> I have tried to replay your results but can not find anything
> wrong
> 
 with the current test kernel.
> Please have a look at the results below, I repeated the same
> experiment
> 
 on two different testbeds.
> 
> Incidentally, these results agree with the values you observed for
> TCP
> 
 Reno. It seems that the
> setup is not right, I am happy to help debugging this but I don't
> think
> 
 that there is a problem
> with the test kernel.
> 
> Can you please describe your setup - maybe something went wrong.
> 
> The first thing to check is whether you are really using the test
> tree
> 
 - it is necessary to check
> out the `dccp' subtree, if you are on the `master' instead of
> the
> 
 `dccp' subtree, then you are
> using a vanilla 2.6.24-rc5.  A detailed HowTo is on
>        
> 
 http://www.linux-foundation.org/en/Net:DCCP_Testing#Experimental_DCCP_source_tr
> ee
> The difference can also easily be told in the
> Request/Response
> 
 handshake, the CCIDs are now
> fully negotiated; the first value of the ConfirmL/R in
> the
> 
 DCCP-Response says which CCIDs are
> used (but they are still nonetheless set
> via
> 
 net.dccp.default.{rx,tx}_ccid).
> 

<snip>

Re: [Announce]: Test results with latest CCID3 patch set

[Gerrit Renker]

|     git-clone git://eden-feed.erg.abdn.ac.uk/dccp_exp my_dccp
| 
| 
| But I did not check out the DCCP sub-branch, by:
| 
|     git-checkout --track -b dccp origin/dccp
| 
| 
| However, as I type this command above, git complains saying:
| 
|     git checkout: branch dccp already exists
| 
| 
| It this OK to say so?
The `--track' command is only necessary to instruct git to always pull
from the given remote destination.

Apparently the dccp subtree got cloned as well. If this succeeded, you
can check out the sub branch via "git-checkout dccp" (no -b); check
with "git-branch" on which branch you are on.

I mostly use the other method, i.e. for my work I pull the test tree
after creating a branch master/dccp and then clone:
  git checkout -b dccp master
  git pull git://eden-feed.erg.abdn.ac.uk/dccp_exp dccp
 

Sorry in the previous reply I forgot to mention that there is a much
easier way to doi all this - the entire test tree is available as a
single patch, for most recent releases on
 http://www.erg.abdn.ac.uk/users/gerrit/dccp/testing_dccp/test-tree/
This can be applied via "zcat test-tree_*.diff.gz| patch -p1 [--dry-run]"

Re: [Announce]: Test results with latest CCID3 patch set

[Ian McDonald]

On 12/13/07, Gerrit Renker <gerrit@erg.abdn.ac.uk> wrote:
> Burak
>
> I have tried to replay your results but can not find anything wrong with the current test kernel.
> Please have a look at the results below, I repeated the same experiment on two different testbeds.
>
Coming in very late in this thread I'd just make a couple of observations:
- I'm still using 2.6.20 for myself but only because for my PhD I need
one baseline and if I were to upgrade I'd have to rerun every single
test for my PhD. A daunting task. I would recommend anyone else use
more modern code. In particular the 2.6.20 code often crashes when
running at full speed and also is missing a whole lot of
fixes/improvements
- I know for a while there were issues like Burak experienced but I
have no idea if there still is. It might be as Gerrit says because he
is not using latest code which has now been explained
- when I did have issues like Burak I tracked it down to the send
credit side of things. I could never quite prove exactly what was
going on though. If problems are still there that is where I would
look first
- I don't have time to really help :-( Burak - if you want to help
hack DCCP code as well that would be awesome.

Ian