From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jeff Garzik Subject: Re: HAIL volunteer Rick Peralta Date: Wed, 29 Jul 2009 10:31:56 -0400 Message-ID: <4A705D5C.9050909@garzik.org> References: <29025029.1248785350151.JavaMail.root@mswamui-andean.atl.sa.earthlink.net> <4A6F5A3E.1070907@garzik.org> <4A704376.6000303@tiac.net> Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <4A704376.6000303@tiac.net> Sender: hail-devel-owner@vger.kernel.org List-ID: Content-Type: text/plain; charset="us-ascii"; format="flowed" To: Rick Peralta Cc: Project Hail (added hail-devel to CC, with permission) Rick Peralta wrote: > Hi Jeff, > > Is there someone taking the point for the chunkd development? That's me, for the moment. :) In general, since Pete and I are Linux kernel developers -- one of the largest open source projects in the world -- I think importing the tried-and-true kernel requirements for code maintainership makes sense: Once you are intimately familiar with a codebase, and can answers others' questions about design and code, you have reached the level where you could be a maintainer. But more importantly -- that is not important! Just contribute code, because that is the why open source projects advance in a particular direction anyway. Linus Torvalds does not take point of the Linux kernel -- the people who actually write code do that. Linus writes maybe 0.01% of the code these days. The people who code, make the roadmap. > To get to 10 Gbe there may be a variety of problems: > 1) If there are data copies, the multiplicative effect can saturate the > memory system. > 2) If the primary storage is rotating media (disk), it wold take 40+ > devices to keep up (assuming large stripes). > 3) There are artifacts of the VM system that show up at high bandwidth, > especially if there is a lot of RAM. > 4) The transaction frequency can become problematic, depending on the > application. > 5) Running a single network transport at 10 Gbe can be challenging. Indeed, scaling up the networking and storage can have plenty of implications. We saw some of this when I was first working with NIC hardware manufacturers to add the first 10 Gbe NIC drivers to the Linux kernel. chunkd is intentionally message-based, which implies that non-TCP protocols could readily be bolted on, for use in data centers with 10 Gbe networks (AMQP? RDMA?). > Is there an application profile that might be used as a performance > metric? Something like total volume size (all media), total number of > chunks, distribution of chunk sizes, aggregate bandwidth, et cetera. This is unfortunately going to vary wildly depending on the application using chunkd (and the application using that, in turn). To take tabled as an example, and assuming a "standard cloud node" hardware setup, * total single-node volume size: one cheap SATA hard drive * total number of chunks: == total number of tabled objects / number of storage nodes * distribution of chunk sizes: dependent upon the application using tabled * aggregate bandwidth: dependent upon the application using tabled > If it matters I investigated building something very much like chunkd a > while back and has some stringent performance criteria. It was not > clear what general application demand there is. Is there a resource to > get a sense of where there is real need? You should read the GoogleFS paper referenced on the chunkd wiki page: http://labs.google.com/papers/gfs-sosp2003.pdf It describes the purpose and use of a chunk server, in the context of distributed cloud storage. The demand is largely internal -- other Project Hail projects and outside distributed storage application should use chunkd in the creation of their own cloud-based service. The intent is for tabled, nfs4d, and other distributed-storage projects to communicate with multiple chunkd's on multiple nodes, to accomplish replicated, highly available distributed storage. If there is some specialized use of chunkd that you have in mind, we're interested in hearing that, too... I certainly want to enable as many applications as possible with these projects. Regards, Jeff