From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jeff Garzik Subject: Re: Hail usable as a "limited local AWS S3" and doc contribution? Date: Wed, 14 Oct 2009 09:01:09 -0400 Message-ID: <4AD5CB95.9000701@garzik.org> References: <117304.55473.qm@web82303.mail.mud.yahoo.com> Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <117304.55473.qm@web82303.mail.mud.yahoo.com> Sender: hail-devel-owner@vger.kernel.org List-ID: Content-Type: text/plain; charset="us-ascii"; format="flowed" To: Zack Perry Cc: hail-devel@vger.kernel.org On 10/13/2009 01:27 PM, Zack Perry wrote: > Hi, > > While trying to find a less expensive way to validate my application targeting AWS S3, I stumbled across Project Hail. > > After reading the wiki, I downloaded cld, chunkd, and tabled tar balls, built them on my Ubuntu 9.04 based VPS successfully. Both cld and chunkd passed the built-in tests too (make check). Not having cld up, I haven't tested tabled yet. > > My intention is to set up all 3 components on my Xen based VPS (Ubuntu 9.04, 2.6.24-19-xen kernel, 512Mb RAM, 1G swap/16G disk space, 1 vcpu) as a "local AWS S3". > > Looking at the bundled docs, it seems to me that only chunkd and tabled has a setup.txt in their respective doc file. For cld, I don't see a corresponding file, although I suppose I can consult scripts/c codes in the test subdirectory to get some ideas. Yep. cld setup is intentionally simple, designed so the daemon may be started from the command line or a simple script, rather than requiring a ton of preparation. Basically you need to set the data directory, and then run the daemon. The daemon will create a new filesystem database if none exists already. Run "cld --help" for command line info (ditto for the other daemons), and ask plenty of questions here on the list ;) > Before diving in further, two questions: > > 0. Can either Jeff or Pete tell me whether the project is already usable for what I would like to do with it? i.e. a setup that one can use something like S3fox Organizer to interact with? Useable, yes. Production ready, not yet. > 1. Am I correct in assuming that I can run all three components on a single VPS with the resources outlined above? Assuming your disk space requirements work, yes, that should be fine. The daemons themselves do not require a ton of memory; we prefer that the OS disk cache be an integral part of our caching scheme. > If yes to both, can you outline the steps that you would setup the three components for such a purpose? It's pretty much just creating data directories (or pointing tabled to a chunkd instance, in tabled's case) and other small details like PID file location. Ideally a chunkd or tabled configuration is stored inside a CLD filesystem. > I am happy to do a full setup and share my experience and details here for others' future reference. If you were motivated to contribute information to the wiki, that would be quite useful. http://hail.wiki.kernel.org/ Jeff