Re: Zookeeper instead of CLD in Hail

[Jeff Garzik <jeff@garzik.org>]

On 06/07/2010 09:21 AM, Jeff Darcy wrote:
> On 06/04/2010 11:27 PM, Pete Zaitcev wrote:
>> About the focus, ZK is just like CLD from a certain angle (it has
>> the good old files and provides a set of un-posixy operations
>> on them: watches, uniques, "ephemerals"), but it's also entirely
>> unlike CLD (e.g. no locks in the protocol). CLD's model is that
>> clients are daemons, each of which reads a few of its files, maybe
>> locks one or two at boot, and then nothing happens except keepalives.
>> Zookeeper's model... honestly I don't know what it is because it's
>> never explained concisely, but the docs that I saw seem to imply
>> huge numbers of clients all doing random ops all the time on the
>> same files, enough to cause a herd concerns. It looks like Yahoo
>> may be using Zookeeper as a lease manager or something. Crazy.
>
> According to the Chubby paper (available at e.g.
> http://labs.google.com/papers/chubby.html) Chubby itself is mostly used
> as a naming service.  They even say explicitly that few clients hold
> locks (page 11, left column, under "use and behavior").
>
>> I heard people say they cribbed from the same Chubby paper, but
>> it's bollocks. It's absolutely nothing like what Chubby implies.
>> No locks for one thing.
>
> I guess it could be argued that locks are the essential feature of
> Chubby - both the name of the service and the title of the paper refer
> to it - but I think your dismissal of ZK and Chubby as unrelated is
> itself bollocks.  Watches and ephemerals within a hierarchical namespace
> of small pseudo-files are also central to how Chubby is actually used
> within Google, even without locking.  ZK copied those features, along
> with many of the underlying algorithms and protocols.  CLD copied a
> slightly different set.  There's still a lot of overlap, even if the one
> feature that has been overused in Hail is absent (by design) in ZK.

I think you're overselling that angle a bit.  Google discourages use of 
Chubby as a strict publish-subscribe mechanism, so watches and 
ephemerals aren't the bread-and-butter of Chubby necessarily.  A lot of 
the supposed commonality comes simply from the attribute of being a 
centralized respository of data for autonomous cloud systems -- a 
shared, highly reliable filesystem -- not any particular attribute 
related to watches or ephemerals.


>> So I do not see
>> an upside for us to switch at this point, and I have better things
>> to do than learning Zookeeper ropes for weeks.
>
> That's a fair enough evaluation.  I think I'm the one who might have
> suggested investigating ZK, but the reasons for that are largely
> community-related and not technical.  If the technical barriers are too
> high, then it's probably not worth the bother.  That does, however,
> leave the door open for someone else to provide more scalable solutions
> for the same problems by using more appropriate primitives.  If the goal
> of Hail is to provide building blocks for people to use in constructing
> their own higher-level service, then it seems to me that the building
> blocks should be of a familiar shape and the adoption of ZK is about
> 100x that of CLD for similar purposes.

Not sure what you mean by familiar shape?  ZK design is too haphazard, 
and developers IMO have an easier time grasping CLD's fundamental 
FS-like API.  The implementation is also 3x or more in terms of size, 
compared to CLD.

I'd rather get the basics right first, then pursue a popularity contest :)

	Jeff