[Benchmark] Chat results

[Benchmark] Chat results

[Paolo Ciarrocchi]

		--- What is the chat benchmark ---
Usage: chat_s ip_addr [port]

       ip_addr:	ip address that the server binds to listening socket.
       [port]:	port number server binds to the listening socket.
		default is 9999.

Usage: chat_c ip_addr [num_rooms] [num_messages] [server_port]

       ip_addr:		server ip address that the client connects to.
       [num_rooms]:	the number of chat rooms.
			default is 10.
       [num_messages]:	the number of messages sent by each chat room member.
			default is 100.	
       [port]:		server port number the client connects to.
			default is 9999.

This benchmark includes both a client and server.  The benchmark is
modeled after a chat room.  This benchmark will create a lot of threads,
tcp connections, and send and receive a lot of messages.  The client side
of the benchmark will report the number of messages sent per second.

The number of chat rooms and messages will control the workload.
The default number of chat rooms is 10.
The default number of messages is 100.  The size of each message is 100 bytes.
Both of these parameters are specified on the client side.

(1) Each chat room has 20 users.
(2) 10 chat rooms represents 20*10 or 200 users.
(3) For each user in the chat room, the client will make a connection
    to the server.
(4) 10 chat rooms will create 10*20 or 200 connections between the client
    and server.
(5) For each user (or connection) in the chat room, 1 'send' thread is created
    and 1 'receive' thread is created.
(6) 10 chat rooms will create 10*20*2 or 400 client threads and 400 server
    threads for a total of 800 threads.
(7) Each client 'send' thread will send the specified number of messages
    to the server.
(8) For 10 chat rooms and 100 messages, the client will send 10*20*100 or
    20,000 messages.  The server 'receive' thread will receive the
    corresponding number of messages.
(9) The chat room server will echo each of the messages back to the other
    users in the chat room.
(10) For 10 chat rooms and 100 messages, the server 'send' thread will send
     10*20*100*19 or 380,000 messages.  The client 'receive' thread will
     receive the corresponding number of messages.
(11) The client will report the message throughput at the end of the test.
(12) The server loops and accepts another start message from the client.

		--- The scripts I used ---
#!/bin/sh
/etc/init.d/network stop
/etc/init.d/network start
./chat_s 127.0.0.1 &
> `uname -r`_total.results
for i in `seq 1 1 10`
do
	./chat_c 127.0.0.1 30 1000 9999 >>`uname -r`_total.results
done
grep Average `uname -r`_total.results | awk '{tot+=$4}; END {print "Average throughput: " tot/NR " messages per second"}' > `uname -r`.average
grep Average `uname -r`_total.results|sort >`uname -r`.sort
echo -n "Min throughput:" >> `uname -r`.average
head -1 `uname -r`.sort|cut -d ":" -f 2 >> `uname -r`.average
echo -n "Max throughput:" >> `uname -r`.average
tail -1 `uname -r`.sort|cut -d ":" -f 2 >> `uname -r`.average


#!/bin/bash
out=(`ls *.average|sort`)
total=`echo ${out[@]}|wc -w`
# echo $total 
> summary.txt
for i in `seq 0 1 $[total-1]`
do
kernel_version=`echo ${out[i]}|cut -d "." --fields=1-3`
	echo -e "\t\tKernel version: $kernel_version" >> summary.txt
	cat ${out[i]} >> summary.txt
	echo >> summary.txt
done;


		--- Results --- 
		
		Kernel version: 2.4.19-ck7
Average throughput: 57210.1 messages per second
Min throughput: 55007 messages per second
Max throughput: 61988 messages per second

		Kernel version: 2.4.19
Average throughput: 47250.9 messages per second
Min throughput: 45634 messages per second
Max throughput: 50940 messages per second

		Kernel version: 2.5.38
Average throughput: 62543.3 messages per second
Min throughput: 58416 messages per second
Max throughput: 64196 messages per second

		Kernel version: 2.5.40
Average throughput: 60115.4 messages per second
Min throughput: 52443 messages per second
Max throughput: 63264 messages per second

		Kernel version: 2.5.43
Average throughput: 58807.5 messages per second
Min throughput: 55376 messages per second
Max throughput: 60617 messages per second

		Kernel version: 2.5.44-mm1
Average throughput: 56060.7 messages per second
Min throughput: 53250 messages per second
Max throughput: 58617 messages per second

		Kernel version: 2.5.44-mm5
Average throughput: 56778.8 messages per second
Min throughput: 54685 messages per second
Max throughput: 59737 messages per second

		Kernel version: 2.5.44
Average throughput: 57906.2 messages per second
Min throughput: 49808 messages per second
Max throughput: 60197 messages per second



Comments/suggestions ?


		Paolo


-- 

Powered by Outblaze

[BENCHMARK] chat results

[Paolo Ciarrocchi]

Hi all,
has been a long time since the last time I posted the results of the chat benchmark.

Something strange happens in 2.5.63*
10 runs for each kernel
Version		Throughput	Min	Max	Ratio
2.4.19-ck7	57210.1 	55007	61988	1.000
2.4.19		47250.9 	45634	50940	0.826
2.5.38		62543.3 	58416	64196	1.093
2.5.40		60115.4 	52443	63264	1.051
2.5.43		58807.5 	55376	60617	1.028
2.5.44-mm1	56060.7 	53250	58617	0.980
2.5.44-mm5	56778.8 	54685	59737	0.992
2.5.44-mm6	57106 		53157	59385	0.998
2.5.44		57906.2 	49808	60197	1.012
2.5.45		56822.7 	55267	59651	0.993
2.5.46		57106.9 	54388	59432	0.998
2.5.49		56188.8 	53938	57825	0.982
2.5.50		53191 		26636	58603	0.930
2.5.53		55815.7 	53807	58414	0.976
2.5.54bk4	56988.5 	55644	58530	0.996
2.5.59		56962.8 	54391	58416	0.996
2.5.60-cfq2	55966.8 	54391	57546	0.978
2.5.61		55592 		53932	57043	0.972
2.5.63-mjb2	54064.7 	27396	59334	0.945
2.5.63		54304.2 	27175	59169	0.949

-- 2.5.63 -- 
Average throughput : 58682 messages per second
Average throughput : 56287 messages per second
Average throughput : 56127 messages per second
Average throughput : 27175 messages per second
Average throughput : 57696 messages per second
Average throughput : 57918 messages per second
Average throughput : 57793 messages per second
Average throughput : 56328 messages per second
Average throughput : 55867 messages per second
Average throughput : 59169 messages per second

-- 2.5.63-mjb2 --
Average throughput : 59334 messages per second
Average throughput : 57104 messages per second
Average throughput : 54823 messages per second
Average throughput : 55014 messages per second
Average throughput : 59313 messages per second
Average throughput : 58056 messages per second
Average throughput : 57461 messages per second
Average throughput : 55884 messages per second
Average throughput : 27396 messages per second
Average throughput : 56262 messages per second

I still don't know why 2.5.38 is the winner...

Ciao,
           Paolo

-- 
______________________________________________
http://www.linuxmail.org/
Now with e-mail forwarding for only US$5.95/yr

Powered by Outblaze

Re: [BENCHMARK] chat results

[Martin J. Bligh]

> Average throughput : 56127 messages per second
> Average throughput : 27175 messages per second

Any idea why it's doing that sort of thing randomly?

M.