From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jeff Garzik Subject: multi-threaded chunkd, design and code Date: Sat, 07 Nov 2009 01:05:39 -0500 Message-ID: <4AF50E33.4030709@garzik.org> Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Return-path: Sender: hail-devel-owner@vger.kernel.org List-ID: Content-Type: text/plain; charset="us-ascii"; format="flowed" To: Project Hail A multi-threaded version of chunkd has been checked into the repository[1], on a branch aptly named "thread." The design: - uses GLib's thread pools[2] - for each poll(2) for each active fd push fd to worker thread via thread pool - worker thread handle one unit of activity (tcp_cli_event func call) wake up main thread, to requeue fd for further polling return to thread pool, to seek more work Concurrent performance (multiple TCP streams) is definitely improved, but single stream suffers a bit, probably due to excessive wakeups. Both single-thread and MT performance can definitely be improved with epoll, but I wanted to get the working-across-all-platforms poll(2) code solid first. One possible alternative model is to leave chunkd single-threaded, yet use aio_read() and aio_write() for file I/O. That would eliminate the current bottleneck common to single-threaded designs: the serialization of file input (READs) requested by multiple parallel TCP clients. If you have multiple TCP clients issuing multiple reads at the same time -- a common case -- multiple threads are the only way to submit I/Os to the kernel in parallel, thereby permitting the kernel elevator and hardware (SCSI/ATA) elevator opportunity to optimize for seek performance. In most applications, reads are synchronous, which in the current single-threaded chunkd becomes a visible performance wart. Regardless... I do not currently plan on pushing the "thread" branch onto git upstream until I explore which epoll-based design is best (ST or MT), and perhaps tune this poll(2) design a bit more. Feedback and contributions always welcome, of course... Jeff [1] git://git.kernel.org/pub/scm/daemon/distsrv/chunkd.git [2] http://library.gnome.org/devel/glib/stable/glib-Thread-Pools.html