Re: Hail usable as a "limited local AWS S3" and doc contribution?

[Jeff Garzik <jeff@garzik.org>]

On 10/13/2009 01:27 PM, Zack Perry wrote:
> Hi,
>
> While trying to find a less expensive way to validate my application targeting AWS S3, I stumbled across Project Hail.
>
> After reading the wiki, I downloaded cld, chunkd, and tabled tar balls, built them on my Ubuntu 9.04 based VPS successfully. Both cld and chunkd passed the built-in tests too (make check). Not having cld up, I haven't tested tabled yet.
>
> My intention is to set up all 3 components on my Xen based VPS (Ubuntu 9.04, 2.6.24-19-xen kernel, 512Mb RAM, 1G swap/16G disk space, 1 vcpu) as a "local AWS S3".
>
> Looking at the bundled docs, it seems to me that only chunkd and tabled has a setup.txt in their respective doc file. For cld, I don't see a corresponding file, although I suppose I can consult scripts/c codes in the test subdirectory to get some ideas.

Yep.  cld setup is intentionally simple, designed so the daemon may be 
started from the command line or a simple script, rather than requiring 
a ton of preparation.

Basically you need to set the data directory, and then run the daemon. 
The daemon will create a new filesystem database if none exists already.

Run "cld --help" for command line info (ditto for the other daemons), 
and ask plenty of questions here on the list ;)


> Before diving in further, two questions:
>
> 0. Can either Jeff or Pete tell me whether the project is already usable for what I would like to do with it? i.e. a setup that one can use something like S3fox Organizer to interact with?

Useable, yes.

Production ready, not yet.


> 1. Am I correct in assuming that I can run all three components on a single VPS with the resources outlined above?

Assuming your disk space requirements work, yes, that should be fine. 
The daemons themselves do not require a ton of memory; we prefer that 
the OS disk cache be an integral part of our caching scheme.


> If yes to both, can you outline the steps that you would setup the three components for such a purpose?

It's pretty much just creating data directories (or pointing tabled to a 
chunkd instance, in tabled's case) and other small details like PID file 
location.

Ideally a chunkd or tabled configuration is stored inside a CLD filesystem.


> I am happy to do a full setup and share my experience and details here for others' future reference.

If you were motivated to contribute information to the wiki, that would 
be quite useful.  http://hail.wiki.kernel.org/

	Jeff