Re: A proposal - binary

Re: A proposal - binary

[Alan Cox]

Ar Gwe, 2006-08-04 am 13:41 -0700, ysgrifennodd Zachary Amsden:
> committed to working on it.  Which is why I wanted feedback on what we 
> have to do to make sure our ESX implementation is done in a way that is 
> acceptable to the community.  I too would like to push for an interface 
> in 2.6.19, and we can't have confusion on this issue be a last minute 
> stopper.

In part thats a legal question so only a lawyer can really tell you what
is and isn't the line for derivative works. 

Philosophically I can see the argument that the moment you hit a
hypervisor trap its akin to running another app (and an app which
communicates via that interface with many othr apps) so your Linux
kernel side code would be GPL and whatever it fires up which handles the
trap come syscall probably isn't. But I'm not a lawyer and neither you
nor anyone else, nor a court reviewing a case should consider the
statement above a guideline of intent.

Alan

Re: A proposal - binary

[Andi Kleen]

> In the Xen case, 
> they may want to run a dom-0 hypervisor which is compiled for an actual 
> hardware sub-arch, such as Summit or ES7000. 

There is no reason Summit or es7000 or any other subarchitecture 
would need to do different  virtualization. In fact these subarchitectures 
are pretty much obsolete by the generic subarchitecture and could be fully
done by runtime switching.

I don't disagree with your general point that some kind of PAL
code between kernels and hypervisors might be a good idea
(in fact I think Xen already uses vsyscall pages in some cases for this),
but this particular example is no good.

> I would expect to see these new sub-architectures 
> begin to grow like a rash. 

I hope not. The i386 subarchitecture setup is pretty bad already
and mostly obsolete for modern systems.

> The same approach can be used on x86_64 for paravirtualization, but also 
> to abstract out vendor differences between platforms.  Opteron and EMT64 
> hardware are quite different, and the plethora of non-standard 
> motherboards and uses have already intruded into the kernel.  Having a 
> clean interface to encapsulate these changes is also desirable here, and 
> once we've nailed down a final approach to achieving this for i386, it 
> makes sense to do x86_64 as well.

Possible.

> 
> I'm now talking lightyears into the future

tststs - please watch your units.

>, but when the i386 and x86_64  
> trees merge together,

I don't think that will happen in the way you imagine. I certainly
don't plan to ever merge legacy stuff like Voyager or Visual Workstation
or even 586 multiprocessor support.

It might be that x86-64 grows 32bit support at some point, but certainly
only for modern systems and without the heavyweight subarchitecture setup
that i386 uses.

> this layer will be almost identical for the two,  
> allowing sharing of tricky pieces of code, like the APIC and IO-APIC, 

No, one of the strong points of the x86-64 port is that APIC/IO-APIC support
doesn't carry all the legacy i386 has to carry.

> NMI handling, system profiling, and power management.  It the interface 
> evolves in a nicely packaged and compartmentalized way from that, then 
> perhaps someday it can grow to become a true cross-architecture way to 
> handle machine abstraction and virtualization. 

I don't fully agree to move everything into paravirt ops. IMHO
it should be only done for stuff which is performance critical
or cannot be virtualized.

For most other stuff a Hypervisor can always trap or not bother.

> (N-tiered cache coloring, 
> multiway hardware page tables, hypercubic interrupt routing, dynamically 
> morphed GPUs, quantum hypervisor isolation).  

I have my doubts paravirt ops will ever support any of this @)
If we tried that then it would be so messy that it would turn into
a bad idea.

> Of course, it will still  
> require a PCI bus.

And it's unlikely PCI will be ever a good fit for a Quantum computer @)

>  I too would like to push for an interface 
> in 2.6.19, and we can't have confusion on this issue be a last minute 
> stopper.

For 2.6.19 it's too late already. Freeze for its merge 
window has already nearly begun and this stuff is not ready yet.
 
> Maybe someday Xen and VMware can share the same ABI interface and both 
> use a VMI like layer. 

The problem with VMI is that while it allows hypervisor side evolution
it doesn't really allow Linux side evolution with its fixed spec.

But having it a bit isolated is probably ok.

-Andi

Re: A proposal - binary

[Zachary Amsden]

Andi Kleen wrote:
>> In the Xen case, 
>> they may want to run a dom-0 hypervisor which is compiled for an actual 
>> hardware sub-arch, such as Summit or ES7000. 
>>     
>
> There is no reason Summit or es7000 or any other subarchitecture 
> would need to do different  virtualization. In fact these subarchitectures 
> are pretty much obsolete by the generic subarchitecture and could be fully
> done by runtime switching.
>   

For privileged domains that have hardware privileges and need to send 
IPIs or something it might make sense.  Othewsie, there is no issue.

>> I would expect to see these new sub-architectures 
>> begin to grow like a rash. 
>>     
>
> I hope not. The i386 subarchitecture setup is pretty bad already
> and mostly obsolete for modern systems.
>   

Yes, I hope not too.

>   
>> I'm now talking lightyears into the future
>>     
>
> tststs - please watch your units.
>   

I realized after I wrote it ;)

> I don't fully agree to move everything into paravirt ops. IMHO
> it should be only done for stuff which is performance critical
> or cannot be virtualized.

Yes, this is all just a crazy idea, not an actual proposal.

> And it's unlikely PCI will be ever a good fit for a Quantum computer @)
>   

Hmm, a quantum bus would only allow one reader of each quantum bit.  So 
you couldn't broadcast without daisy chaining everything.  Could be an 
issue.

>> Maybe someday Xen and VMware can share the same ABI interface and both 
>> use a VMI like layer. 
>>     
>
> The problem with VMI is that while it allows hypervisor side evolution
> it doesn't really allow Linux side evolution with its fixed spec.
>   

It doesn't stop Linux from using the provided primitives in any way is 
sees fit.  So it doesn't top evolution in that sense.  What it does stop 
is having the Linux hypervisor interface grow antlers and have new 
hooves grafted onto it.  What it sorely needed in the interface is a way 
to probe and detect optional features that allow it to grow independent 
of one particular hypervisor vendor.

Zach

Re: A proposal - binary

[David Lang]

On Sat, 5 Aug 2006, Andi Kleen wrote:

> The problem with VMI is that while it allows hypervisor side evolution
> it doesn't really allow Linux side evolution with its fixed spec.
>
> But having it a bit isolated is probably ok.

actually, wouldn't something like this allow for a one-way evolution (the spec 
can be changed, but the hypervisor side needs to support clients what only talk 
older versions. i.e. the new spec is a superset of the old one (barring major 
security-type problems that require exeptions to the rules))?

David Lang

Re: A proposal - binary

[Andi Kleen]

> For privileged domains that have hardware privileges and need to send 
> IPIs or something it might make sense. 

Any SMP guest needs IPI support of some sort.

But it is hopefully independent of subarchitectures in the paravirtualized
case.


> doesn't stop Linux from using the provided primitives in any way is 
> sees fit.  So it doesn't top evolution in that sense.  What it does stop 
> is having the Linux hypervisor interface grow antlers and have new 
> hooves grafted onto it.  What it sorely needed in the interface is a way 
> to probe 

That's the direction the interface is evolving I think (see multiple
entry point discussion) 

> and detect optional features that allow it to grow independent 
> of one particular hypervisor vendor.

Ok maybe not with options and subsets so far, but one has to
start somewhere.

-Andi

Re: A proposal - binary

[James Bottomley]

On Fri, 2006-08-04 at 22:26 +0100, Alan Cox wrote:
> In part thats a legal question so only a lawyer can really tell you
> what
> is and isn't the line for derivative works. 

Actually, this isn't quite true.  In any licensing agreement between two
parties, what each thinks is an important consideration in the
enforcement of the agreement.  This is how we got binary modules in the
first place, and so it also follows that what kernel developers think
about this proposal is an important influence on the eventual legal
opinon.

My take is that the VMI proposal breaks down into two pieces:

1) A hypervisor ABI.  This is easy: we maintain ABIs today between libc
and the kernel, so nothing about an ABI is inherantly GPL violating.

2) A gateway page or vDSO provided by the hypervisor to the kernel.
This is the problematic piece, because the vDSO is de-facto linked into
the kernel and as such becomes subject to the prevailing developer
interpretation as being a derivative work by being linked in.  As Arjan
pointed out, this can be avoided as long as the gateway page itself is
GPL ... we could even create mechanisms like we use today for module
licensing by having a tag in the VMI describing the licensing of the
gateway page, so the kernel could be made only to load gateway pages
that promise they're available under the GPL.

I think that if we do this tagging to load the VMI vDSO interface, then
I'm happy that all of the legal niceties are safely taken care of.
(Although the onus is now back on VMware to establish if they can GPL
their VMI blob).

James

Re: A proposal - binary

[James Bottomley]

On Fri, 2006-08-04 at 13:41 -0700, Zachary Amsden wrote:
> Instead, what paravirt-ops promises long 
> term is a way to get rid of the sub-architecture layer altogether.  
> Sub-arches like Voyager and Visual workstation have some strange 
> initialization requirements, interrupt controllers, and SMP handling.  
> Exactly the kind of thing which paravirt_ops is being designed to 
> indirect for hypervisors. 

Well ... I agree that in principle it's possible to have a kernel that
would run on both voyager and a generic x86 system and, I'll admit, I
tried to go that route before creating the subarchitectures.  However,
in practice, I think the cost really becomes too high ... for voyager,
it becomes necessary really to intercept almost the entirety of the the
SMP API.  The purpose of the subarchitecture interface wasn't to
eventually have some API description that would allow voyager to
co-exist with more normal x86 systems.  It was to allow voyager to make
use of generic x86 while being completely different at the x86 SMP
level.  I really don't think there'll ever be another x86 machine that's
as different from the APIC approach as the voyager VIC/QIC is.  thus, I
think the actual x86 interface is much better described by mach-generic,
which abstracts out the interfaces necessary to the more standard APIC
based SMP systems.

James

Re: A proposal - binary

[Zachary Amsden]

James Bottomley wrote:
> Well ... I agree that in principle it's possible to have a kernel that
> would run on both voyager and a generic x86 system and, I'll admit, I
> tried to go that route before creating the subarchitectures.  However,
> in practice, I think the cost really becomes too high ... for voyager,
> it becomes necessary really to intercept almost the entirety of the the
> SMP API.  The purpose of the subarchitecture interface wasn't to
> eventually have some API description that would allow voyager to
> co-exist with more normal x86 systems.  It was to allow voyager to make
> use of generic x86 while being completely different at the x86 SMP
> level.  I really don't think there'll ever be another x86 machine that's
> as different from the APIC approach as the voyager VIC/QIC is.  thus, I
> think the actual x86 interface is much better described by mach-generic,
> which abstracts out the interfaces necessary to the more standard APIC
> based SMP systems.
>   

This is quite true today.  But it is entirely possible that support in 
Linux for Xen may want to rip out the APIC / IO-APIC entirely, replace 
that with event channels, and use different SMP shootdown mechanisms, as 
well as having their own special NMI delivery hook.  We're also going to 
have to make certain parts of the interface extremely efficient, and 
we've already got several schemes to remove the penalty of indirection 
by being rid of indirect branches - which could be a more broadly used 
technique if it proves unintrusive and reliable enough.  In that case, 
you could basically support Voyager without a subarch, plus or minus one 
special hook or two ;)

Zach

Re: A proposal - binary

[Zachary Amsden]

James Bottomley wrote:
> My take is that the VMI proposal breaks down into two pieces:
>   

This is a very accurate description of our interface.

> 1) A hypervisor ABI.  This is easy: we maintain ABIs today between libc
> and the kernel, so nothing about an ABI is inherantly GPL violating.
>   

This I think is an absolute must for any sane interpretation of the 
GPL.  Otherwise, running GPL apps on any proprietary operating system 
falls into the same situation, and you wouldn't be able to run them 
without violating the GPL.  Nor would you be able to run non-GPL 
applications on a GPL kernel.  It's really a matter of whether you 
interpret the intent of the GPL to prevent someone deriving work from 
your open source software and distributing that in binary form without 
providing the dervied work - or if you interpret the GPL as saying that 
all code must be open sourced.  The latter is a very extreme position, 
and I don't believe it is even correct with the current wording of GPL 
v2 (IANAL).

> 2) A gateway page or vDSO provided by the hypervisor to the kernel.
> This is the problematic piece, because the vDSO is de-facto linked into
> the kernel and as such becomes subject to the prevailing developer
> interpretation as being a derivative work by being linked in.  As Arjan
> pointed out, this can be avoided as long as the gateway page itself is
> GPL ... we could even create mechanisms like we use today for module
> licensing by having a tag in the VMI describing the licensing of the
> gateway page, so the kernel could be made only to load gateway pages
> that promise they're available under the GPL.
>   

Yes, this is what prompted my whole module rant.  The interesting thing 
is - Linux may link to the hypervisor vDSO.  But it may not link back 
into Linux.  This is where the line becomes very gray, as Theodore 
mentioned earlier.  Is it a license violation for a GPL app to link 
against a non-GPL library?  Surely, the other way around is a problem, 
unless the library has been made explicitly LGPL.  But if GPL apps can 
link to non-GPL libraries, what stops GPL kernels from linking to 
non-GPL modules?  This is where I think things become more interpretive 
than well defined.  And that is why it is important for us to get kernel 
developers feedback on exactly what that definition is.

> I think that if we do this tagging to load the VMI vDSO interface, then
> I'm happy that all of the legal niceties are safely taken care of.
> (Although the onus is now back on VMware to establish if they can GPL
> their VMI blob).
>   

Tagging is interesting.  You can tag modules by license.  I can't say 
today what license we will be able to use for it - it could be 
completely proprietary, some new license, BSD, or GPL.  This is the 
essence of my original rant - it would be nice to have a way to tag the 
license in the "blob" so the kernel can choose the appropriate course of 
action.  In that case, the pressure is off me to specify what the 
license is - it's there for everyone to see, and then it is just a 
matter of coming to a consensus as to what an acceptable license is for 
Linux to link to it.  What license(s) we provide is really not up to me, 
although I personally would very much like to see an open source license 
that allows everyone to see the code, fix any problems they have with 
it, and distribute those fixes (purely my own personal opinion, and in 
no way a statement, promise, or supposition in any legal or corporate 
sense for any past, present, or future work by VMware, EMC, or any other 
entity, wholly or partially owned by said corporations, and in no way 
should this be interpreted as constituting a legal opinion for the 
purposes of advice or rendering of any court decision, now, in the 
future, or in the past for legal arbiters with access to time travel 
equipment). <Now I'm covered better than Alan>.

Binary blob has been a PR disaster.  I don't know if I first said it 
unprompted, or if Cristoph cleverly baited me into using the phrase ;)  
But lets be clear on one thing - blob implies some kind of shapeless, 
fat thing.  The VMI fits in two pages of memory, and has a well defined 
interface, which gives it shape.  So I prefer binary redirection 
interface, or vDSO, or anything without the disparaged word "blob" in it.

Zach

Re: A proposal - binary

[Adrian Bunk]

On Fri, Aug 04, 2006 at 10:37:01PM -0700, Zachary Amsden wrote:
> James Bottomley wrote:
>...
> >2) A gateway page or vDSO provided by the hypervisor to the kernel.
> >This is the problematic piece, because the vDSO is de-facto linked into
> >the kernel and as such becomes subject to the prevailing developer
> >interpretation as being a derivative work by being linked in.  As Arjan
> >pointed out, this can be avoided as long as the gateway page itself is
> >GPL ... we could even create mechanisms like we use today for module
> >licensing by having a tag in the VMI describing the licensing of the
> >gateway page, so the kernel could be made only to load gateway pages
> >that promise they're available under the GPL.
> 
> Yes, this is what prompted my whole module rant.  The interesting thing 
> is - Linux may link to the hypervisor vDSO.  But it may not link back 
> into Linux.  This is where the line becomes very gray, as Theodore 
> mentioned earlier.  Is it a license violation for a GPL app to link 
> against a non-GPL library?  Surely, the other way around is a problem, 

I don't see the grey area.

Assuming non-GPL and not GPL compatible (e.g. 3 clause BSD is non-GPL 
but compatible):

Unless all people holding a copyright on the GPL app agreed that this 
linking is OK, it is considered a licence violation.

That's why you often see licence statements like the following:

"In addition, as a special exception, the Free Software Foundation
gives permission to link the code of its release of Wget with the
OpenSSL project's "OpenSSL" library (or with modified versions of it
that use the same license as the "OpenSSL" library), and distribute
the linked executables.  You must obey the GNU General Public License
in all respects for all of the code used other than "OpenSSL".  If you
modify this file, you may extend this exception to your version of the
file, but you are not obligated to do so.  If you do not wish to do
so, delete this exception statement from your version."

> unless the library has been made explicitly LGPL.  But if GPL apps can 
> link to non-GPL libraries, what stops GPL kernels from linking to 
> non-GPL modules?  This is where I think things become more interpretive 
> than well defined.  And that is why it is important for us to get kernel 
> developers feedback on exactly what that definition is.
>...

Some kernel developers (and some lawyers) consider all kernel modules 
with not GPL compatible licences illegal - similar to the case of 
linking a GPL app with a non-GPL library.

Quoting Novell [1]:

"Most developers of the kernel community consider non-GPL kernel
modules to be infringing on their copyright. Novell does respect this
position, and will no longer distribute non-GPL kernel modules as part 
of future products. Novell is working with vendors to find alternative
ways to provide the functionality that was previously only available
with non-GPL kernel modules."

And considering the number of people having a copyright on parts of the 
kernel, there's noone except a court who can tell what is OK and what is 
not (and even a court decision is not binding for courts in other 
countries).

> Zach

cu
Adrian

[1] http://lists.opensuse.org/archive/opensuse-announce/2006-Feb/0004.html

-- 

    Gentoo kernels are 42 times more popular than SUSE kernels among
    KLive users  (a service by SUSE contractor Andrea Arcangeli that
    gathers data about kernels from many users worldwide).

       There are three kinds of lies: Lies, Damn Lies, and Statistics.
                                                    Benjamin Disraeli

Re: A proposal - binary

[Adrian Bunk]

On Sat, Aug 05, 2006 at 12:01:52AM +0200, Andi Kleen wrote:
> 
> There is no reason Summit or es7000 or any other subarchitecture 
> would need to do different  virtualization. In fact these subarchitectures 
> are pretty much obsolete by the generic subarchitecture and could be fully
> done by runtime switching.
>...

Has anyone measured the performance impact of rutime CLOCK_TICK_RATE 
switching (since this will no longer allow some compile time 
optimizations in jiffies.h)?

> -Andi

cu
Adrian

-- 

    Gentoo kernels are 42 times more popular than SUSE kernels among
    KLive users  (a service by SUSE contractor Andrea Arcangeli that
    gathers data about kernels from many users worldwide).

       There are three kinds of lies: Lies, Damn Lies, and Statistics.
                                                    Benjamin Disraeli

Re: A proposal - binary

[Alan Cox]

Ar Gwe, 2006-08-04 am 22:37 -0700, ysgrifennodd Zachary Amsden:
> mentioned earlier.  Is it a license violation for a GPL app to link 
> against a non-GPL library?  Surely, the other way around is a problem, 


Actually the FSF always anticipated that case because its the same as
the GPL app on non-free OS case and the GPL there says

"However, as a special exception, the source code distributed need not
include anything that is normally distributed (in either source or
binary form) with the major components (compiler, kernel, and so on) of
the operating system on which the executable runs, unless that component
itself accompanies the executable."

> interface, which gives it shape.  So I prefer binary redirection 
> interface, or vDSO, or anything without the disparaged word "blob" in it.

Well if you are going to provide the source then its not really a binary
interface, its a jump table.

Re: A proposal - binary

[Andi Kleen]

> Has anyone measured the performance impact of rutime CLOCK_TICK_RATE 
> switching (since this will no longer allow some compile time 
> optimizations in jiffies.h)?

SUSE shipped a kernel briefly that had runtime switchable jiffies
and there were some benchmarks done and they didn't show noticeable
slowdown.

But with hr timers it should be pretty much obsolete anyways.

-Andi