[RFC] tmem ABI change... backwards compatibility unnecessary?

[RFC] tmem ABI change... backwards compatibility unnecessary?

[Dan Magenheimer]

Tmem users and Xen developers/distros --

(Please forward/repost as you see fit.)

After a great deal of discussion and review with linux
kernel developers, it appears there are "next-generation"
filesystems (such as btrfs, xfs, Lustre) that will not
be able to use tmem due to an ABI limitation... a field
that represents a unique file identifier is 64-bits in
the tmem ABI and may need to be as large as 192-bits.
So to support these guest filesystems, the tmem ABI must be
revised, from "v0" to "v1".

I *think* it is still the case that tmem is experimental
and is not used anywhere yet in production.  If I am
wrong, PLEASE LET ME KNOW ASAP.

The tmem ABI is designed to support multiple revisions,
so the Xen tmem implementation could be updated to
handle both v0 and v1.  However this is a bit
messy and would require data structures for both v0
and v1 to appear in public Xen header files.

I am inclined to update the Xen tmem implementation
to only support v1 and gracefully fail v0.  This would
result in only a performance loss (as if tmem were
disabled) for newly launched tmem-v0-enabled guests,
but live-migration between old tmem-v0 Xen and new
tmem-v1 Xen machines would fail, and saved tmem-v0
guests will not be able to be restored on a tmem-v1
Xen machine.  I would plan to update both pre-4.0.2
and unstable (future 4.1) to only support v1.

I believe these restrictions are reasonable at this
point in the tmem lifecycle, though they may not
be reasonable in the near future; should the tmem
ABI need to be revised from v1 to v2, I understand
backwards compatibility will be required.

Comments or questions?  (If agreeable, positive public
acks appreciated.)

Thanks,
Dan

P.S. For general information on tmem, see:
http://oss.oracle.com/projects/tmem 

Re: [RFC] tmem ABI change... backwards compatibility unnecessary?

[Vasiliy G Tolstov]

В Срд, 01/09/2010 в 07:36 -0700, Dan Magenheimer пишет:
> Tmem users and Xen developers/distros --
> 
> (Please forward/repost as you see fit.)
> 
> After a great deal of discussion and review with linux
> kernel developers, it appears there are "next-generation"
> filesystems (such as btrfs, xfs, Lustre) that will not
> be able to use tmem due to an ABI limitation... a field
> that represents a unique file identifier is 64-bits in
> the tmem ABI and may need to be as large as 192-bits.
> So to support these guest filesystems, the tmem ABI must be
> revised, from "v0" to "v1".
> 
> I *think* it is still the case that tmem is experimental
> and is not used anywhere yet in production.  If I am
> wrong, PLEASE LET ME KNOW ASAP.
> 
> The tmem ABI is designed to support multiple revisions,
> so the Xen tmem implementation could be updated to
> handle both v0 and v1.  However this is a bit
> messy and would require data structures for both v0
> and v1 to appear in public Xen header files.
> 
> I am inclined to update the Xen tmem implementation
> to only support v1 and gracefully fail v0.  This would
> result in only a performance loss (as if tmem were
> disabled) for newly launched tmem-v0-enabled guests,
> but live-migration between old tmem-v0 Xen and new
> tmem-v1 Xen machines would fail, and saved tmem-v0
> guests will not be able to be restored on a tmem-v1
> Xen machine.  I would plan to update both pre-4.0.2
> and unstable (future 4.1) to only support v1.
> 
> I believe these restrictions are reasonable at this
> point in the tmem lifecycle, though they may not
> be reasonable in the near future; should the tmem
> ABI need to be revised from v1 to v2, I understand
> backwards compatibility will be required.
> 
> Comments or questions?  (If agreeable, positive public
> acks appreciated.)

Thank You, Dan. What needed to do, to use tmem v1? Does it needed to
recompile dom0 kernel, or only domU kernel affected by change?

-- 
Vasiliy G Tolstov <v.tolstov@selfip.ru>
Selfip.Ru

Re: [RFC] tmem ABI change... backwards compatibility unnecessary?

[Jan Beulich]

>>> On 01.09.10 at 16:36, Dan Magenheimer <dan.magenheimer@oracle.com> wrote:
> I *think* it is still the case that tmem is experimental
> and is not used anywhere yet in production.  If I am
> wrong, PLEASE LET ME KNOW ASAP.

Well, if you call us shipping it (default disabled) in a couple of
releases "not used in production"...

> I am inclined to update the Xen tmem implementation
> to only support v1 and gracefully fail v0.

If "graceful" really means what it says, this would appear to be
acceptable irrespective of my note above.

Jan

RE: [RFC] tmem ABI change... backwards compatibility unnecessary?

[Dan Magenheimer]

> > Comments or questions?  (If agreeable, positive public
> > acks appreciated.)
> 
> Thank You, Dan. What needed to do, to use tmem v1? Does it needed to
> recompile dom0 kernel, or only domU kernel affected by change?

I am working on a patch now for tmem v1 which will change
the hypervisor and xend. (I don't think the xl tools fully
support tmem yet, but will look at that later.)  

The dom0 kernel will not need to change unless you enable tmem
in your dom0 kernel.  I do not normally recommend enabling
tmem in dom0, though some do.

Any domU kernel that enables tmem also must move to v1.
I will provide kernel patches when I complete the hypervisor
and tools patch.

Dan

Re: [RFC] tmem ABI change... backwards compatibility unnecessary?

[Jeremy Fitzhardinge]

 On 09/01/2010 07:36 AM, Dan Magenheimer wrote:
> I am inclined to update the Xen tmem implementation
> to only support v1 and gracefully fail v0.  This would
> result in only a performance loss (as if tmem were
> disabled) for newly launched tmem-v0-enabled guests,
> but live-migration between old tmem-v0 Xen and new
> tmem-v1 Xen machines would fail, and saved tmem-v0
> guests will not be able to be restored on a tmem-v1
> Xen machine.  I would plan to update both pre-4.0.2
> and unstable (future 4.1) to only support v1.
>
> I believe these restrictions are reasonable at this
> point in the tmem lifecycle, though they may not
> be reasonable in the near future; should the tmem
> ABI need to be revised from v1 to v2, I understand
> backwards compatibility will be required.
>
> Comments or questions?  (If agreeable, positive public
> acks appreciated.)

Sounds fine to me.  Given that pvops doesn't have tmem support in it at
all yet, I don't think the tmem userbase can be all that big.  Once its
in pvops/upstream, this kind of decision will take a lot more care...

    J

RE: [RFC] tmem ABI change... backwards compatibility unnecessary?

[Dan Magenheimer]

> From: Jan Beulich [mailto:JBeulich@novell.com]
> Sent: Wednesday, September 01, 2010 9:04 AM
> To: Dan Magenheimer
> Cc: stephen.spector@citrix.com; Keir Fraser; JeremyFitzhardinge; Xen-
> Devel (xen-devel@lists.xensource.com); Chris Mason; Kurt Hackel; tmem-
> devel@oss.oracle.com; Vasiliy G Tolstov
> Subject: Re: [RFC] tmem ABI change... backwards compatibility
> unnecessary?
> 
> >>> On 01.09.10 at 16:36, Dan Magenheimer <dan.magenheimer@oracle.com>
> wrote:
> > I *think* it is still the case that tmem is experimental
> > and is not used anywhere yet in production.  If I am
> > wrong, PLEASE LET ME KNOW ASAP.
> 
> Well, if you call us shipping it (default disabled) in a couple of
> releases "not used in production"...
> 
> > I am inclined to update the Xen tmem implementation
> > to only support v1 and gracefully fail v0.
> 
> If "graceful" really means what it says, this would appear to be
> acceptable irrespective of my note above.

OK, I will submit a patch tomorrow with the following characteristics:

v0 (current) hypervisor + v0 guest: succeeds
v1 (patched) hypervisor + v1 guest: succeeds
v0 (current) hypervisor + v1 guest: fails
v1 (patched) hypervisor + v0 guest: fails

where fails is an xm dmesg message that says "unsupported
spec version" when the guest attempts to create a pool.
And pool creation failure ensures that all further tmem
operations also fail (indeed never even result in a
hypercall for most tmem-enabled kernels).

Thank goodness ABI versioning was built into tmem from
the beginning!

Dan

Re: [RFC] tmem ABI change... backwards compatibility unnecessary?

[Jeremy Fitzhardinge]

 On 09/02/2010 04:19 PM, Dan Magenheimer wrote:
> OK, I will submit a patch tomorrow with the following characteristics:
>
> v0 (current) hypervisor + v0 guest: succeeds
> v1 (patched) hypervisor + v1 guest: succeeds
> v0 (current) hypervisor + v1 guest: fails
> v1 (patched) hypervisor + v0 guest: fails
>
> where fails is an xm dmesg message that says "unsupported
> spec version" when the guest attempts to create a pool.
> And pool creation failure ensures that all further tmem
> operations also fail (indeed never even result in a
> hypercall for most tmem-enabled kernels).
>
> Thank goodness ABI versioning was built into tmem from
> the beginning!

Hm, I'm not really a big fan of having a single "ABI version".  It
always seems better to have individual calls which can be
augmented/replaced by new calls, and/or have capability flags to extend
the ABI.  Versions mean you end up being stuck doing updates in a very
coarse-grained way, and the long-term support gets very onerous.
(Microsoft ABIs are a good antipattern to avoid, especially DirectX.)

    J

RE: [RFC] tmem ABI change... backwards compatibility unnecessary?

[Dan Magenheimer]

> > Thank goodness ABI versioning was built into tmem from
> > the beginning!
> 
> Hm, I'm not really a big fan of having a single "ABI version".  It
> always seems better to have individual calls which can be
> augmented/replaced by new calls, and/or have capability flags to extend
> the ABI.  Versions mean you end up being stuck doing updates in a very
> coarse-grained way, and the long-term support gets very onerous.
> (Microsoft ABIs are a good antipattern to avoid, especially DirectX.)

While I agree in general, and tmem can cleanly handle other forms
of minor updates as well, this ABI version change was very nice
for this case where the ops data structure (passed to the hypercall
in memory) changed in size, thus changing the interface for most of
the critical tmem operations.