multi-threaded chunkd, design and code

[Jeff Garzik <jeff@garzik.org>]

A multi-threaded version of chunkd has been checked into the 
repository[1], on a branch aptly named "thread."

The design:
- uses GLib's thread pools[2]
- for each poll(2)
	for each active fd
		push fd to worker thread via thread pool
- worker thread
	handle one unit of activity (tcp_cli_event func call)
	wake up main thread, to requeue fd for further polling
	return to thread pool, to seek more work

Concurrent performance (multiple TCP streams) is definitely improved, 
but single stream suffers a bit, probably due to excessive wakeups.

Both single-thread and MT performance can definitely be improved with 
epoll, but I wanted to get the working-across-all-platforms poll(2) code 
solid first.

One possible alternative model is to leave chunkd single-threaded, yet 
use aio_read() and aio_write() for file I/O.  That would eliminate the 
current bottleneck common to single-threaded designs:  the serialization 
of file input (READs) requested by multiple parallel TCP clients.

If you have multiple TCP clients issuing multiple reads at the same time 
-- a common case -- multiple threads are the only way to submit I/Os to 
the kernel in parallel, thereby permitting the kernel elevator and 
hardware (SCSI/ATA) elevator opportunity to optimize for seek performance.

In most applications, reads are synchronous, which in the current 
single-threaded chunkd becomes a visible performance wart.

Regardless...  I do not currently plan on pushing the "thread" branch 
onto git upstream until I explore which epoll-based design is best (ST 
or MT), and perhaps tune this poll(2) design a bit more.

Feedback and contributions always welcome, of course...

	Jeff


[1] git://git.kernel.org/pub/scm/daemon/distsrv/chunkd.git
[2] http://library.gnome.org/devel/glib/stable/glib-Thread-Pools.html