[RFC] Switching store to use domain id's for keys

[RFC] Switching store to use domain id's for keys

[Anthony Liguori]

Hey guys,

As I go through the console code, I notice that the domain console 
information is being stored in /console/%d/domain instead of within the 
/domain tree as one would expect.

The reasoning seems obvious to me, there's no easy way to get the UUID 
for a domain so constructing a UUID based path outside of Xend is very 
difficult.

Perhaps now is a good time to reconsider just using domain ids instead 
of UUIDs for the paths?  In a cluster we could just use 
<nodeid>/domain/<domid> for unique identification.

Or, if we really want to use UUIDs in the node path, perhaps we can add 
UUIDs to the domain structure in the hypervisor so that we can actually 
query it through getdomaininfo?

Regards,

Anthony Liguori

Re: [RFC] Switching store to use domain id's for keys

[Christian Limpach]

On Tue, Aug 30, 2005 at 03:32:46PM -0500, Anthony Liguori wrote:
> As I go through the console code, I notice that the domain console 
> information is being stored in /console/%d/domain instead of within the 
> /domain tree as one would expect.

We were wondering about this yesterday.

> The reasoning seems obvious to me, there's no easy way to get the UUID 
> for a domain so constructing a UUID based path outside of Xend is very 
> difficult.

Hmm, it's less than 40 lines of C code to iterate over the /domain entries
and find a domain with a matching name or id.  It would be convenient
to have library functions which does this...

> Perhaps now is a good time to reconsider just using domain ids instead 
> of UUIDs for the paths?  In a cluster we could just use 
> <nodeid>/domain/<domid> for unique identification.

We'd like the identifier for a domain to remain the same after
relocating the domain to a different physical machine.[1]

If we consider changing this, I'd go for /domain/<nodeid>-<domid>.  It
would make it easy to find the path for a domain on its home node but
it wouldn't work anymore once you move the domain to a new host.

> Or, if we really want to use UUIDs in the node path, perhaps we can add 
> UUIDs to the domain structure in the hypervisor so that we can actually 
> query it through getdomaininfo?

We've considered this but everytime we've found that the reason why
we wanted it was invalid.

In any case, it seems reasonable to move the console entries to the
domain tree then...

    christian

[1] this is currently broken since we create a new uuid on restore.
The issue is that device configuration is local to the host where the
domain is running and there's a short period of time where we want
device information for two hosts present at the same time.

Re: [RFC] Switching store to use domain id's for keys

[Anthony Liguori]

Christian Limpach wrote:

>>The reasoning seems obvious to me, there's no easy way to get the UUID 
>>for a domain so constructing a UUID based path outside of Xend is very 
>>difficult.
>>    
>>
>
>Hmm, it's less than 40 lines of C code to iterate over the /domain entries
>and find a domain with a matching name or id.  It would be convenient
>to have library functions which does this...
>  
>
The linear search just makes me sad :-)  There can still be a UUID key 
within the /domain tree.  One can just as easily code up a function to 
resolve UUIDs to domain IDs by doing a linear search.

I think the key is to optimize for the common case here.  Since, today 
and I reckon for the foreseeable future, the management functions take 
domain IDs, it seems like the right thing to optimize for.

>We'd like the identifier for a domain to remain the same after
>relocating the domain to a different physical machine.[1]
>  
>
Yes, this is a tough thing to get right.  It straight forward for 
migration, but what about save/restore, cloning, etc.  It seems almost 
better to let UUIDs preservation to be a higher level management task.

>If we consider changing this, I'd go for /domain/<nodeid>-<domid>.  It
>would make it easy to find the path for a domain on its home node but
>it wouldn't work anymore once you move the domain to a new host.
>  
>
I'm partial to a more hierarchical model.  Something like 
<nodeid>/domain/<domid>.  It lets <nodeid> become part of an implicit 
path...

Regards,

Anthony Liguori

>>Or, if we really want to use UUIDs in the node path, perhaps we can add 
>>UUIDs to the domain structure in the hypervisor so that we can actually 
>>query it through getdomaininfo?
>>    
>>
>
>We've considered this but everytime we've found that the reason why
>we wanted it was invalid.
>
>In any case, it seems reasonable to move the console entries to the
>domain tree then...
>
>    christian
>
>[1] this is currently broken since we create a new uuid on restore.
>The issue is that device configuration is local to the host where the
>domain is running and there's a short period of time where we want
>device information for two hosts present at the same time.
>
>
>  
>

Re: [RFC] Switching store to use domain id's for keys

[Rusty Russell]

On Wed, 2005-08-31 at 10:28 +0100, Christian Limpach wrote:
> On Tue, Aug 30, 2005 at 03:32:46PM -0500, Anthony Liguori wrote:
> > Perhaps now is a good time to reconsider just using domain ids instead 
> > of UUIDs for the paths?  In a cluster we could just use 
> > <nodeid>/domain/<domid> for unique identification.
> 
> We'd like the identifier for a domain to remain the same after
> relocating the domain to a different physical machine.[1]

You're never going to have the property that a uuid always maps to the
same domain, or a domain always has the same uuid, if we ever introduce
forking of domains by any method.  So, I think this is a losing battle,
but it's causing a mess of the store as a side effect.

As far as I can tell, UUIDs are a third identifier of domains, which buy
nothing over the existing two: names (cluster-wide unique, human
readable, slow), and domids (locally unique, fast).

> If we consider changing this, I'd go for /domain/<nodeid>-<domid>.  It
> would make it easy to find the path for a domain on its home node but
> it wouldn't work anymore once you move the domain to a new host.

That only makes sense if a single store is shared across the entire
cluster.  Otherwise the <node-id> is completely redundant, since it will
be the same for all visible domains.

A better solution might be to federate at the top level, so it would be
"/<nodeid>/domain..." in a cluster environment.  Then such a federated
store does not need to effect today's plans.

Rusty.
-- 
A bad analogy is like a leaky screwdriver -- Richard Braakman

Re: Re: [RFC] Switching store to use domain id's for keys

[Steven Hand]

>On Wed, 2005-08-31 at 10:28 +0100, Christian Limpach wrote:
>> On Tue, Aug 30, 2005 at 03:32:46PM -0500, Anthony Liguori wrote:
>> > Perhaps now is a good time to reconsider just using domain ids instead 
>> > of UUIDs for the paths?  In a cluster we could just use 
>> > <nodeid>/domain/<domid> for unique identification.
>> 
>> We'd like the identifier for a domain to remain the same after
>> relocating the domain to a different physical machine.[1]
>
>You're never going to have the property that a uuid always maps to the
>same domain, or a domain always has the same uuid, if we ever introduce
>forking of domains by any method.  So, I think this is a losing battle,
>but it's causing a mess of the store as a side effect.
>
>As far as I can tell, UUIDs are a third identifier of domains, which buy
>nothing over the existing two: names (cluster-wide unique, human
>readable, slow), and domids (locally unique, fast).

Well one issue is that cluster-wide unique human readable names are 
tricky to enforce. Right now what we need is something which refers
to the same "virtual machine" regardless of which domain it currently
inhabits. I.e. across save/restore, across migrate, etc. If this is 
unique to a "virtual machine", then a 'fork' (when we get it) is going 
to cause a new one of these to be created. I guess we could try to use
human readable stuff for this, but I think having the extra level of 
indirection makes it easier. 

Of course this means that per-domain (rather than per virtual machine) 
state such as event channels and mfns of shared memory should be indexed
by domain id in the store, not by uuid. The uuid is an important part 
of a domain's identity but it's effectively a 'foreign key' rather than
the primary key of the domain information. 


cheers,

S.

Re: Re: [RFC] Switching store to use domain id's for keys

[Rusty Russell]

On Mon, 2005-09-05 at 01:26 +0100, Steven Hand wrote:
> Rusty wrote:
> >As far as I can tell, UUIDs are a third identifier of domains, which buy
> >nothing over the existing two: names (cluster-wide unique, human
> >readable, slow), and domids (locally unique, fast).
> 
> Well one issue is that cluster-wide unique human readable names are 
> tricky to enforce.

A system with duplicate names is not really sane.  All user tools are
going to use names, so differentiation by uuids doesn't help.  Whether
name uniqueness is enforced or not I don't really mind: people are
creative, they can generate unique names all kinds off ways (even uuids
if that's what floats your boat).

> Right now what we need is something which refers
> to the same "virtual machine" regardless of which domain it currently
> inhabits. I.e. across save/restore, across migrate, etc. If this is 
> unique to a "virtual machine", then a 'fork' (when we get it) is going 
> to cause a new one of these to be created.

Sure, and the name fits these as well as UUID.  You cannot, in general,
meet your requirements, UUID or no, because you can fork and destroy the
original, etc.

> I guess we could try to use
> human readable stuff for this, but I think having the extra level of 
> indirection makes it easier. 

I disagree; more indirection, more concepts to master, more room for
confusion, more code, worse store layout, with no more features.

Seems all bad from where I'm sitting 8(
Rusty.
-- 
A bad analogy is like a leaky screwdriver -- Richard Braakman

Re: Re: [RFC] Switching store to use domain id's for keys

[Steven Hand]

>On Mon, 2005-09-05 at 01:26 +0100, Steven Hand wrote:
>> Rusty wrote:
>> >As far as I can tell, UUIDs are a third identifier of domains, which buy
>> >nothing over the existing two: names (cluster-wide unique, human
>> >readable, slow), and domids (locally unique, fast).
>> 
>> Well one issue is that cluster-wide unique human readable names are 
>> tricky to enforce.
>
>A system with duplicate names is not really sane.  All user tools are
>going to use names, so differentiation by uuids doesn't help.  Whether
>name uniqueness is enforced or not I don't really mind: people are
>creative, they can generate unique names all kinds off ways (even uuids
>if that's what floats your boat).
>
>> Right now what we need is something which refers
>> to the same "virtual machine" regardless of which domain it currently
>> inhabits. I.e. across save/restore, across migrate, etc. If this is 
>> unique to a "virtual machine", then a 'fork' (when we get it) is going 
>> to cause a new one of these to be created.
>
>Sure, and the name fits these as well as UUID.  

Well UUIDs can be regular making code simpler; they can be transparently
to the tools treated as structured allowing independent allocation and 
verification; they provide a simple unambiguous ordering; they sidestep
issues of internationalization; and they firmly place xenstore out of 
the view of the end-user which I think is quite a good thing. 

>You cannot, in general, meet your requirements, UUID or no, because you 
>can fork and destroy the original, etc.

I don't understand this - can you explain? 

(or is this just some kind of general impossibility statement regarding 
asynchronous messages and byazantine failures??) 


>> I guess we could try to use
>> human readable stuff for this, but I think having the extra level of 
>> indirection makes it easier. 
>
>I disagree; more indirection, more concepts to master, more room for
>confusion, more code, worse store layout, with no more features.
>
>Seems all bad from where I'm sitting 8(

Well I certainly wouldn't want to confuse you by requiring you to master 
more concepts... :-) 

However: I think we both /agree/ on the fact that domain ids should be 
used to name things which are about domains (e.g. event channel ids 
and backend domain ids, etc, etc). 

And i think we both /agree/ that within a domain's piece of xenstore, 
that there's at least one 'name' which is a cluster-wide unique string
and which persists across save/restore/migrate. 

Yes? 


cheers,

S.

Re: Re: [RFC] Switching store to use domain id's for keys

[Rusty Russell]

On Mon, 2005-09-05 at 02:43 +0100, Steven Hand wrote:
> Well I certainly wouldn't want to confuse you by requiring you to master 
> more concepts... :-) 

You have a unique identifier which is already used by the tools and is
simple to understand.  You want to add another one, related in some way
to the first one, because it's *easy to generate*.

I'm speechless at this logic.

Your other benefits include: easier to canonically order (memcmp, of
course, being too hard), easier to deal with (memcpy too hard as well I
guess) and, my favorite benefit of all those you came up with: it makes
the store *harder* to read.

Must be that famous dry British sense of humor 8)

> However: I think we both /agree/ on the fact that domain ids should be 
> used to name things which are about domains (e.g. event channel ids 
> and backend domain ids, etc, etc). 
> 
> And i think we both /agree/ that within a domain's piece of xenstore, 
> that there's at least one 'name' which is a cluster-wide unique string
> and which persists across save/restore/migrate. 

Indeed, these were never in doubt.  You're arguing that we need *two*
names, and sorry, I just can't see it.

Rusty.
-- 
A bad analogy is like a leaky screwdriver -- Richard Braakman

Re: Re: [RFC] Switching store to use domain id's for keys

[Steven Hand]

>On Mon, 2005-09-05 at 02:43 +0100, Steven Hand wrote:
>> Well I certainly wouldn't want to confuse you by requiring you to master 
>> more concepts... :-) 
>
>You have a unique identifier which is already used by the tools and is
>simple to understand.  

Well actually I currently have two; user-defined 'names' and UUIDs. I'm 
arguing that we retain these two (along with the third sort of 'name' 
in the form of domain ids) but use them more appropriately. 

>You want to add another one, related in some way to the first one, because 
>it's *easy to generate*.

Amongst other things, yes. Generating / managing unique names across a 
cluster or federated system is tricky. Expecting a user's chosen name 
to happen to be unique on its own doesn't work;  prefixing with e.g. 
a node id or whatever is ambiguous in our world of live migration; 
prefixing with a user name or role name or whatever is clunky. 

>I'm speechless at this logic.

Well obviously not... 

( <duck> :^)

>Your other benefits include: easier to canonically order (memcmp, of
>course, being too hard), easier to deal with (memcpy too hard as well I
>guess) and, my favorite benefit of all those you came up with: it makes
>the store *harder* to read.

Fixed sized strings are simpler to handle than variable sized ones. To 
take your example, if you use memcmp() what's n? What happens when two 
strings are different lengths (i.e. what do you explicitly or implicitly 
pad the shorted with)? 

Fixed sized strings are simpler to handle. 

W.r.t the store being harder to read - sure. I'm ok with this, although
I can see that people currently writing and debugging store stuff would 
find it nicer to have 'friendly' names (the cruft left behind in the 
store right now after you've been running for a while is pretty nasty). 

But longer term I don't envison direct or 'raw' human access to the store 
as a common operation. Instead a piece of software like 'xm' or 'vm' or a 
script or whatever will read + extract the relevant parts and format them 
for output. 

>Must be that famous dry British sense of humor 8)

Maybe, although I'm not British. 

>> However: I think we both /agree/ on the fact that domain ids should be 
>> used to name things which are about domains (e.g. event channel ids 
>> and backend domain ids, etc, etc). 
>> 
>> And i think we both /agree/ that within a domain's piece of xenstore, 
>> that there's at least one 'name' which is a cluster-wide unique string
>> and which persists across save/restore/migrate. 
>
>Indeed, these were never in doubt.  You're arguing that we need *two*
>names, and sorry, I just can't see it.

That's a shame. 

Anyway let's work on the above right now since that requires a restructure
of the store which is common whether or not we assume the unique cluster
wide things are UUIDs or random user strings. 

cheers,


S.

Re: Re: [RFC] Switching store to use domain id's for keys

[Keir Fraser]

On 5 Sep 2005, at 09:35, Steven Hand wrote:

>> You have a unique identifier which is already used by the tools and is
>> simple to understand.
>
> Well actually I currently have two; user-defined 'names' and UUIDs. I'm
> arguing that we retain these two (along with the third sort of 'name'
> in the form of domain ids) but use them more appropriately.

Can we not just sidestep this issue by saying that UUIDs are arbitrary 
strings? Then people who want to use names at that level can do so, and 
we can use random strings or DCE UUIDs or whatever else when we want to 
auto-generate a UUID (and I can think of scenarios where this will be 
the case, and the user will not be specifying the name nor care what 
that name is).

I think pretty much every user of the store treats UUIDs as opaque, so 
this seems a good solution to me. The argument that UUIDs are regular 
and easy to generate is not a winner for me.

  -- Keir

Re: Re: [RFC] Switching store to use domain id's for keys

[Rusty Russell]

On Mon, 2005-09-05 at 09:35 +0100, Steven Hand wrote:
> >On Mon, 2005-09-05 at 02:43 +0100, Steven Hand wrote:
> >> Well I certainly wouldn't want to confuse you by requiring you to master 
> >> more concepts... :-) 
> >
> >You have a unique identifier which is already used by the tools and is
> >simple to understand.  
> 
> Well actually I currently have two; user-defined 'names' and UUIDs. I'm 
> arguing that we retain these two (along with the third sort of 'name' 
> in the form of domain ids) but use them more appropriately. 

If we end up always mapping name => UUID => nodeid + domid then UUID is
not buying us anything but extra code, confusion and pain.  The
alternative is exposing the UUID to the admins, in addition to the name,
and that's a poor bad user-interface decision IMHO for a case which
shouldn't happen.

> >You want to add another one, related in some way to the first one, because 
> >it's *easy to generate*.
> 
> Amongst other things, yes. Generating / managing unique names across a 
> cluster or federated system is tricky. Expecting a user's chosen name 
> to happen to be unique on its own doesn't work;  prefixing with e.g. 
> a node id or whatever is ambiguous in our world of live migration; 
> prefixing with a user name or role name or whatever is clunky. 

(1) We already need to handle duplicate UUID detection on restore and
fork, so we already have this problem 8(
(2) An admin who creates two domains of the same name will have trouble
controlling them, but I think that's OK, and implied.

> Anyway let's work on the above right now since that requires a restructure
> of the store which is common whether or not we assume the unique cluster
> wide things are UUIDs or random user strings.

So in all the arguing, did we decide on /domain/<domid>/ which
becomes /<nodeid>/domain/<domid> in a cluster environment, and a "name"
or "uuid" entry within it identifying the particular domain?

Cheers,
Rusty.
-- 
A bad analogy is like a leaky screwdriver -- Richard Braakman