RE: 32/64-bit hypercall interface

RE: 32/64-bit hypercall interface

[Ian Pratt]

> Anyway, since this is a Xen/Kernel header, perhpas we can 
> create a type that represent the Xen/Kernel machine-register size?

Since we're only talking about dom0 ops, it wouldn't be unreasonable to
turn all the long's into u64's. This would be pretty straightforward,
arguably easier than bothering with a u_reg type or similar.

Since the dom0_op interface is not guaranteed stable in the 3.0 series
[we assume you have matched tools and hypervisor] we don't have to get
this in before 3.0.0. However, it's a pretty simple change and if
someone who really cared were to knock up a tested patch immediately
we'd potentially consider it.

Ian

Re: 32/64-bit hypercall interface

[Hollis Blanchard]

On Thursday 29 September 2005 08:56, Ian Pratt wrote:
>
> Since we're only talking about dom0 ops, it wouldn't be unreasonable to
> turn all the long's into u64's. This would be pretty straightforward,
> arguably easier than bothering with a u_reg type or similar.

start_info_t: tools, kernel
xen_parameters_info_t: tools, hypervisor
gnttab_setup_table_t: kernel, hypervisor
gnttab_transfer_t: kernel, hypervisor
struct xen_memory_reservation: tools, kernel, hypervisor
struct t_rec: tools, hypervisor
struct t_buf: tools, hypervisor

(This was just a quick grep for "long" in xen/include/public.)

The tools/hypervisor interface seems to be the most important at the moment. I 
don't know if anybody has plans to do 32-bit kernel on a 64-bit hypervisor... 
but why not enable it while we're in there?

-- 
Hollis Blanchard
IBM Linux Technology Center

Re: 32/64-bit hypercall interface

[Hollis Blanchard]

[Somehow Keir was dropped from the CC list on my earlier mail; sorry.]

On Thursday 29 September 2005 13:17, Hollis Blanchard wrote:
> On Thursday 29 September 2005 08:56, Ian Pratt wrote:
> > Since we're only talking about dom0 ops, it wouldn't be unreasonable to
> > turn all the long's into u64's. This would be pretty straightforward,
> > arguably easier than bothering with a u_reg type or similar.
>
> start_info_t: tools, kernel
> xen_parameters_info_t: tools, hypervisor
> gnttab_setup_table_t: kernel, hypervisor
> gnttab_transfer_t: kernel, hypervisor
> struct xen_memory_reservation: tools, kernel, hypervisor
> struct t_rec: tools, hypervisor
> struct t_buf: tools, hypervisor
>
> (This was just a quick grep for "long" in xen/include/public.)

Of course, pointers have the same problem. Thankfully, it seems pointers are 
mostly found in the same list of offenders. (It's also a little harder to 
grep for "*", so I could be missing some problems.)

-- 
Hollis Blanchard
IBM Linux Technology Center

Re: 32/64-bit hypercall interface

[Keir Fraser]

On 29 Sep 2005, at 21:12, Hollis Blanchard wrote:

>> start_info_t: tools, kernel
>> xen_parameters_info_t: tools, hypervisor
>> gnttab_setup_table_t: kernel, hypervisor
>> gnttab_transfer_t: kernel, hypervisor
>> struct xen_memory_reservation: tools, kernel, hypervisor
>> struct t_rec: tools, hypervisor
>> struct t_buf: tools, hypervisor
>>
>> (This was just a quick grep for "long" in xen/include/public.)
>
> Of course, pointers have the same problem. Thankfully, it seems 
> pointers are
> mostly found in the same list of offenders. (It's also a little harder 
> to
> grep for "*", so I could be missing some problems.)

How would you 'fix' pointers?

  -- Keir

Re: 32/64-bit hypercall interface

[Keir Fraser]

On 29 Sep 2005, at 21:12, Hollis Blanchard wrote:

>> start_info_t: tools, kernel
>> xen_parameters_info_t: tools, hypervisor
>> gnttab_setup_table_t: kernel, hypervisor
>> gnttab_transfer_t: kernel, hypervisor
>> struct xen_memory_reservation: tools, kernel, hypervisor
>> struct t_rec: tools, hypervisor
>> struct t_buf: tools, hypervisor
>>
>> (This was just a quick grep for "long" in xen/include/public.)
>
> Of course, pointers have the same problem. Thankfully, it seems 
> pointers are
> mostly found in the same list of offenders. (It's also a little harder 
> to
> grep for "*", so I could be missing some problems.)

How about writing a script that converts the Xen public headers into a 
set that, when compiled on 32-bit ppc, match the 64-bit ppc layout? 
That might be saner than trying to manually finesse our headers around 
the ppc abi so that building in 32-bit or 64-bit environment both 
happen to give same binary layout.

  -- Keir

Re: 32/64-bit hypercall interface

[Andrei Petrov]

On Thu, Sep 29, 2005 at 11:43:10PM +0100, Keir Fraser wrote:
> 
> On 29 Sep 2005, at 21:12, Hollis Blanchard wrote:
> 
> >>start_info_t: tools, kernel
> >>xen_parameters_info_t: tools, hypervisor
> >>gnttab_setup_table_t: kernel, hypervisor
> >>gnttab_transfer_t: kernel, hypervisor
> >>struct xen_memory_reservation: tools, kernel, hypervisor
> >>struct t_rec: tools, hypervisor
> >>struct t_buf: tools, hypervisor
> >>
> >>(This was just a quick grep for "long" in xen/include/public.)
> >
> >Of course, pointers have the same problem. Thankfully, it seems 
> >pointers are
> >mostly found in the same list of offenders. (It's also a little harder 
> >to
> >grep for "*", so I could be missing some problems.)
> 
> How about writing a script that converts the Xen public headers into a 
> set that, when compiled on 32-bit ppc, match the 64-bit ppc layout? 
> That might be saner than trying to manually finesse our headers around 
> the ppc abi so that building in 32-bit or 64-bit environment both 
> happen to give same binary layout.
> 

If this type of ABI compatibility is needed, interface data structures should be declared
platform-independantly, i.e using 'sized' types (yeah, field alignment should be taken care of too).
That'll guarantee 32/64 bit independance and that's good thing to have. Once defined it can be 
managed in controlled way. There could be penalty involved for using larger data types than neccessary,
but that can be resolved having 32-bit(or 64-bit, depending whichever is attractive as basic)
hypercall subset.

-Andrei

Re: 32/64-bit hypercall interface

[Keir Fraser]

On 30 Sep 2005, at 01:54, Andrei Petrov wrote:

>> How about writing a script that converts the Xen public headers into a
>> set that, when compiled on 32-bit ppc, match the 64-bit ppc layout?
>> That might be saner than trying to manually finesse our headers around
>> the ppc abi so that building in 32-bit or 64-bit environment both
>> happen to give same binary layout.
>>
>
> If this type of ABI compatibility is needed, interface data structures 
> should be declared
> platform-independantly, i.e using 'sized' types (yeah, field alignment 
> should be taken care of too).
> That'll guarantee 32/64 bit independance and that's good thing to 
> have. Once defined it can be
> managed in controlled way. There could be penalty involved for using 
> larger data types than neccessary,
> but that can be resolved having 32-bit(or 64-bit, depending whichever 
> is attractive as basic)
> hypercall subset.

When we previously had this, by defining packed structs, there were 
plenty of screams that it wasn't ANSI compliant, and that performance 
sucks on some architectures. At the time there seemed no good reason 
for cross-ABI compatibility (we don't plan to stuff these structs into 
network packets, for example).

  -- Keir

Re: 32/64-bit hypercall interface

[Jimi Xenidis]

>>>>> "KF" == Keir Fraser <Keir.Fraser@cl.cam.ac.uk> writes:

 KF> On 30 Sep 2005, at 01:54, Andrei Petrov wrote:

 KF> When we previously had this, by defining packed structs, there were 
 KF> plenty of screams that it wasn't ANSI compliant, and that performance 
 KF> sucks on some architectures.

Tou use a research term, "thats just ka-ka" :)
Packed in no way solves the problem of selecting the appropriate types.

 KF> At the time there seemed no good reason for cross-ABI
 KF> compatibility (we don't plan to stuff these structs into network
 KF> packets, for example).

There are 4 ABIs present, Xen (internal), Domain, Managment App, and
the ABI (_not_ API) that Xen presents to all.

We only wish to formalize the Xen External ABI in a binary form, not
by lanuage.

-JX

-- 
 "I got an idea, an idea so smart my head would explode if I even
  began to know what I was talking about." -- Peter Griffin (Family Guy)

Re: 32/64-bit hypercall interface

[Ronald G Minnich]

Jimi Xenidis wrote:

>  KF> When we previously had this, by defining packed structs, there were 
>  KF> plenty of screams that it wasn't ANSI compliant, and that performance 
>  KF> sucks on some architectures.
> 
> Tou use a research term, "thats just ka-ka" :)
> Packed in no way solves the problem of selecting the appropriate types.

no, it's not ka ka at all. I had huge troubles with plan 9 c and the way 
that the linux structs were packed via the use of gcc struct packing. I 
am really glad packed went away.

The fact is, Xen has had trouble for me at times, as it is very 
linux/gcc specific. The one that hit me the hardest was exception 
frames, and second hardest was the packed keyword. I had to write 
marshall/unmarshall functions on Plan 9, just as though I were using 
RPC. It worked out, though, since (see below) I was able to end up with 
structs with better alignment.

I thank the Xen team for removing packed structs. It was a good move.

Also, there was some packing that resulted in very suboptimal structure 
member alignment. In one case, for one struct, a long struct member was 
odd-byte-aligned 1/2 the time, odd-short-aligned 1/4 the time, and 
correctly aligned 1/4 the time. There's no reason to do this.

There was also a lot of research disagreement on the plan 9 list about 
packing, when I brought this issue up; it's not like all the angels sing 
from the gcc hymnal. Comments ranged from the politest,  along the lines 
of "using packed is bad design", to less polite :-)

There are lots of good arguments on both sides. But it really boils down 
to, 'which is most portable, using packed or not'. The answer is, not.

thanks

ron

Re: 32/64-bit hypercall interface

[Hollis Blanchard]

On Friday 30 September 2005 15:05, Ronald G Minnich wrote:
> Jimi Xenidis wrote:
> >  KF> When we previously had this, by defining packed structs, there were
> >  KF> plenty of screams that it wasn't ANSI compliant, and that
> > performance KF> sucks on some architectures.
> >
> > Tou use a research term, "thats just ka-ka" :)
> > Packed in no way solves the problem of selecting the appropriate types.
>
> no, it's not ka ka at all. I had huge troubles with plan 9 c and the way
> that the linux structs were packed via the use of gcc struct packing. I
> am really glad packed went away.

I think you misunderstand: Jimi is also glad. Read again: the packed GCC 
attribute does not solve the problem.

-- 
Hollis Blanchard
IBM Linux Technology Center

Re: 32/64-bit hypercall interface

[Ronald G Minnich]

Hollis Blanchard wrote:
> On Friday 30 September 2005 15:05, Ronald G Minnich wrote:
> 
>>Jimi Xenidis wrote:
>>
>>> KF> When we previously had this, by defining packed structs, there were
>>> KF> plenty of screams that it wasn't ANSI compliant, and that
>>>performance KF> sucks on some architectures.
>>>
>>>Tou use a research term, "thats just ka-ka" :)
>>>Packed in no way solves the problem of selecting the appropriate types.
>>
>>no, it's not ka ka at all. I had huge troubles with plan 9 c and the way
>>that the linux structs were packed via the use of gcc struct packing. I
>>am really glad packed went away.
> 
> 
> I think you misunderstand: Jimi is also glad. Read again: the packed GCC 
> attribute does not solve the problem.
> 

That figures. I am really sorry it's monday.

thanks

ron
p.s. jimix, whereever you may be, I'm sorry.

Re: 32/64-bit hypercall interface

[Hollis Blanchard]

On Sep 30, 2005, at 3:03 AM, Keir Fraser wrote:
>
> On 30 Sep 2005, at 01:54, Andrei Petrov wrote:
>
>>> How about writing a script that converts the Xen public headers into 
>>> a
>>> set that, when compiled on 32-bit ppc, match the 64-bit ppc layout?
>>> That might be saner than trying to manually finesse our headers 
>>> around
>>> the ppc abi so that building in 32-bit or 64-bit environment both
>>> happen to give same binary layout.
>>
>> If this type of ABI compatibility is needed, interface data 
>> structures should be declared
>> platform-independantly, i.e using 'sized' types (yeah, field 
>> alignment should be taken care of too).

Absolutely agree.

>> That'll guarantee 32/64 bit independance and that's good thing to 
>> have. Once defined it can be
>> managed in controlled way. There could be penalty involved for using 
>> larger data types than neccessary,
>> but that can be resolved having 32-bit(or 64-bit, depending whichever 
>> is attractive as basic)
>> hypercall subset.
>
> When we previously had this, by defining packed structs, there were 
> plenty of screams that it wasn't ANSI compliant, and that performance 
> sucks on some architectures. At the time there seemed no good reason 
> for cross-ABI compatibility (we don't plan to stuff these structs into 
> network packets, for example).

The complaints I remember were specifically about using GCC's 
__attribute__((packed)) extension. Properly laying out the structures 
and/or using explicit padding is a different solution.

Example:
typedef struct {
     /* IN variables. */
     domid_t       domain;
     unsigned long max_pfns;
     void         *buffer;
     /* OUT variables. */
     unsigned long num_pfns;
} dom0_getmemlist_t;

This could be fixed by laying it out like this:
typedef struct {
     /* IN variables. */
     u64 max_pfns;
     u64 buffer;
     domid_t       domain;
     char _pad[6];
     /* OUT variables. */
     u64 num_pfns;
} dom0_getmemlist_t;

Or easier, losing the IN/OUT sections:
typedef struct {
     u64 max_pfns; /* in */
     u64 buffer; /* in */
     u64 num_pfns; /* out */
     domid_t domain; /* in */
} dom0_getmemlist_t;

And yes, pointers will need to be "fixed" into u64s.

As for performance, it does take an extra memory reference on x86 when 
assigning from a 32-bit value into a 64-bit field, but I would expect 
that cost to be insignificant compared to the cost of a hypercall. 
Maybe doing 64-bit math on x86 would be significantly slower, but 
that's not what we're talking about...

-- 
Hollis Blanchard
IBM Linux Technology Center

Re: 32/64-bit hypercall interface

[Keir Fraser]

On 30 Sep 2005, at 16:39, Hollis Blanchard wrote:

> And yes, pointers will need to be "fixed" into u64s.
>
> As for performance, it does take an extra memory reference on x86 when 
> assigning from a 32-bit value into a 64-bit field, but I would expect 
> that cost to be insignificant compared to the cost of a hypercall. 
> Maybe doing 64-bit math on x86 would be significantly slower, but 
> that's not what we're talking about...

Cross-architecture 'compatibility' (same binary layout) is not 
currently an aim for the Xen-public interfaces, and I don't expect it 
to become so. If we went down that road we'd have to stipulate things 
like endianess, which I think we can all agree is not the way to go.

I certainly don't want to wholesale restructure our interfaces just to 
fortuitously make things match up for 32- and 64-bit ppc (which is what 
you are actually arguing for, in the guise of more general 
cross-architecture compatibility).

  -- Keir

Re: 32/64-bit hypercall interface

[Hollis Blanchard]

On Sep 30, 2005, at 10:45 AM, Keir Fraser wrote:
>
> Cross-architecture 'compatibility' (same binary layout) is not 
> currently an aim for the Xen-public interfaces, and I don't expect it 
> to become so. If we went down that road we'd have to stipulate things 
> like endianess, which I think we can all agree is not the way to go.
>
> I certainly don't want to wholesale restructure our interfaces just to 
> fortuitously make things match up for 32- and 64-bit ppc (which is 
> what you are actually arguing for, in the guise of more general 
> cross-architecture compatibility).

Yes, I'm not trying to be sneaky about this: this is a problem for 
ppc64. It could also be a problem for x86-64, except that x86-32 is so 
limited compared to x86-64 that apparently nobody wants to do this.

I think the Linux kernel folks learned the "unsigned long" lesson too 
late, and now there is quite a lot of fixup code to convert 32-bit 
userspace structs to 64-bit kernel structs (have a look at 
linux/fs/compat_ioctl.c and compat.c). It seems a shame to repeat the 
same mistakes in Xen...

-- 
Hollis Blanchard
IBM Linux Technology Center

Re: 32/64-bit hypercall interface

[Keir Fraser]

On 30 Sep 2005, at 17:34, Hollis Blanchard wrote:

>> Cross-architecture 'compatibility' (same binary layout) is not 
>> currently an aim for the Xen-public interfaces, and I don't expect it 
>> to become so. If we went down that road we'd have to stipulate things 
>> like endianess, which I think we can all agree is not the way to go.
>>
>> I certainly don't want to wholesale restructure our interfaces just 
>> to fortuitously make things match up for 32- and 64-bit ppc (which is 
>> what you are actually arguing for, in the guise of more general 
>> cross-architecture compatibility).
>
> Yes, I'm not trying to be sneaky about this: this is a problem for 
> ppc64. It could also be a problem for x86-64, except that x86-32 is so 
> limited compared to x86-64 that apparently nobody wants to do this.
>
> I think the Linux kernel folks learned the "unsigned long" lesson too 
> late, and now there is quite a lot of fixup code to convert 32-bit 
> userspace structs to 64-bit kernel structs (have a look at 
> linux/fs/compat_ioctl.c and compat.c). It seems a shame to repeat the 
> same mistakes in Xen...

There's the rub: we don't expect to ever want to provide 32-bit x86 ABI 
compatibility on 64-bit x86 Xen. We will not be supporting 32-bit 
paravirtualised guests on 64-bit x86 Xen, and we've taken the approach 
of requiring separate 32- and 64-bit toolsets (which isn't too painful 
on x86 since full-fledged 64-bit filesystems are quite easy to come 
by).

Really this sounds to me like this is only going to be a problem for 
ppc. That given, knocking up a translation script to import the 
interfaces into your 32-bit toolchain does seem worthy of 
consideration. It wouldn't allow you to use the same tool binaries on 
both 32- and 64-bit ppc Xen, but I don't know how much you actually 
care about that...

  -- Keir

Re: 32/64-bit hypercall interface

[Hollis Blanchard]

On Sep 30, 2005, at 11:42 AM, Keir Fraser wrote:
>
> There's the rub: we don't expect to ever want to provide 32-bit x86 
> ABI compatibility on 64-bit x86 Xen. We will not be supporting 32-bit 
> paravirtualised guests on 64-bit x86 Xen, and we've taken the approach 
> of requiring separate 32- and 64-bit toolsets (which isn't too painful 
> on x86 since full-fledged 64-bit filesystems are quite easy to come 
> by).

That's great, if all you care about is x86. That is how Linux started 
of course, and look where we are now... I think portability and 
well-designed interfaces are nice things to think about earlier rather 
than later. :)

> Really this sounds to me like this is only going to be a problem for 
> ppc. That given, knocking up a translation script to import the 
> interfaces into your 32-bit toolchain does seem worthy of 
> consideration. It wouldn't allow you to use the same tool binaries on 
> both 32- and 64-bit ppc Xen, but I don't know how much you actually 
> care about that...

A translation script seems exceedingly difficult. A separate set of 
headers would be less so, but extremely fragile.

-- 
Hollis Blanchard
IBM Linux Technology Center

Re: 32/64-bit hypercall interface

[Jeremy Katz]

On Fri, 2005-09-30 at 17:42 +0100, Keir Fraser wrote:
> On 30 Sep 2005, at 17:34, Hollis Blanchard wrote:
> >> Cross-architecture 'compatibility' (same binary layout) is not 
> >> currently an aim for the Xen-public interfaces, and I don't expect it 
> >> to become so. If we went down that road we'd have to stipulate things 
> >> like endianess, which I think we can all agree is not the way to go.
> >>
> >> I certainly don't want to wholesale restructure our interfaces just 
> >> to fortuitously make things match up for 32- and 64-bit ppc (which is 
> >> what you are actually arguing for, in the guise of more general 
> >> cross-architecture compatibility).
> >
> > Yes, I'm not trying to be sneaky about this: this is a problem for 
> > ppc64. It could also be a problem for x86-64, except that x86-32 is so 
> > limited compared to x86-64 that apparently nobody wants to do this.
> >
> > I think the Linux kernel folks learned the "unsigned long" lesson too 
> > late, and now there is quite a lot of fixup code to convert 32-bit 
> > userspace structs to 64-bit kernel structs (have a look at 
> > linux/fs/compat_ioctl.c and compat.c). It seems a shame to repeat the 
> > same mistakes in Xen...
> 
> There's the rub: we don't expect to ever want to provide 32-bit x86 ABI 
> compatibility on 64-bit x86 Xen. We will not be supporting 32-bit 
> paravirtualised guests on 64-bit x86 Xen, 

... which I've previously said and continue to think is going to be
something that will eventually be regretted.  People are going to want
to continue to run in a 32bit OS for legacy reasons for quite a while
(even with the compatibility you get on x86_64).  Making it so they
can't do mix and match of 32 and 64 bit guests on a single host is going
to significantly reduce the utility of Xen.

Jeremy

Re: 32/64-bit hypercall interface

[Kip Macy]

[-- Attachment #1.1: Type: text/plain, Size: 1004 bytes --]

You also said you would never support PAE. PAE is second-only to SMP support
on the list of customer requirements. There may be a compelling reason not
to, but superficially running many 32-bit paravirtualized guests on a 64-bit
HV makes a great deal of sense.


-Kip



> There's the rub: we don't expect to ever want to provide 32-bit x86 ABI
> compatibility on 64-bit x86 Xen. We will not be supporting 32-bit
> paravirtualised guests on 64-bit x86 Xen, and we've taken the approach
> of requiring separate 32- and 64-bit toolsets (which isn't too painful
> on x86 since full-fledged 64-bit filesystems are quite easy to come
> by).
>
> Really this sounds to me like this is only going to be a problem for
> ppc. That given, knocking up a translation script to import the
> interfaces into your 32-bit toolchain does seem worthy of
> consideration. It wouldn't allow you to use the same tool binaries on
> both 32- and 64-bit ppc Xen, but I don't know how much you actually
> care about that...
>
>
>

[-- Attachment #1.2: Type: text/html, Size: 1209 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: 32/64-bit hypercall interface

[David]

Hi,

>
> I think the Linux kernel folks learned the "unsigned long" lesson too 
> late, and now there is quite a lot of fixup code to convert 32-bit 
> userspace structs to 64-bit kernel structs (have a look at 
> linux/fs/compat_ioctl.c and compat.c). It seems a shame to repeat the 
> same mistakes in Xen...
>
Quick question....  What is this "'unsigned long' lesson" you are 
refering to?

-- David

Re: 32/64-bit hypercall interface

[Keir Fraser]

On 30 Sep 2005, at 17:44, David wrote:

>> I think the Linux kernel folks learned the "unsigned long" lesson too 
>> late, and now there is quite a lot of fixup code to convert 32-bit 
>> userspace structs to 64-bit kernel structs (have a look at 
>> linux/fs/compat_ioctl.c and compat.c). It seems a shame to repeat the 
>> same mistakes in Xen...
>>
> Quick question....  What is this "'unsigned long' lesson" you are 
> refering to?

As a rule, longs are 4 bytes on 32-bit architectures and 8 bytes on 
64-bit architectures. If you want to support 32-bit apps on your shiny 
new 64-bit OS, it is nice if your interface doesn't contain too many 
structures with different layouts between 32-bit and 64-bit. If you 
have pointers and longs, the structures have differenrt layouts and you 
need a shim layer at the top of the OS to convert 32-bit format into 
the 64-bit format that the OS uses internally. Which is a pain.

  -- Keir

Re: 32/64-bit hypercall interface

[Hollis Blanchard]

On Sep 30, 2005, at 11:44 AM, David wrote:
>> I think the Linux kernel folks learned the "unsigned long" lesson too 
>> late, and now there is quite a lot of fixup code to convert 32-bit 
>> userspace structs to 64-bit kernel structs (have a look at 
>> linux/fs/compat_ioctl.c and compat.c). It seems a shame to repeat the 
>> same mistakes in Xen...
>>
> Quick question....  What is this "'unsigned long' lesson" you are 
> refering to?

"long" has the unfortunate tendency to change size between 32- and 
64-bit compiles, like a pointer. So if you put that in a structure, the 
structure's layout will change between 32- and 64-bit builds. When you 
make that structure part of the userland/kernel ABI, you will have 
problems on architectures that can run both 32- and 64-bit binaries, 
which include PowerPC, MIPS, x86-64, Sparc, and possibly S390 and even 
IA64.

-- 
Hollis Blanchard
IBM Linux Technology Center

Re: 32/64-bit hypercall interface

[Andrei Petrov]

On Fri, Sep 30, 2005 at 04:45:48PM +0100, Keir Fraser wrote:
> 
> Cross-architecture 'compatibility' (same binary layout) is not 
> currently an aim for the Xen-public interfaces, and I don't expect it 
> to become so. If we went down that road we'd have to stipulate things 
> like endianess, which I think we can all agree is not the way to go.
> 
> I certainly don't want to wholesale restructure our interfaces just to 
> fortuitously make things match up for 32- and 64-bit ppc (which is what 
> you are actually arguing for, in the guise of more general 
> cross-architecture compatibility).
> 

The only practical consideration would be compatibility between 32/64 modes on
the same platform for hypercalls. I would expect 32-bit guest to be able to
run on xen-64 and that easier to achive if hypercall interface is clean.

Seems(need to look more though) it could be done by redefining current data types
in such a way so that they will have the same layout on 32-bit platform.

-Andrei

Re: 32/64-bit hypercall interface

[Hollis Blanchard]

[-- Attachment #1: Type: text/plain, Size: 994 bytes --]

On Thursday 29 September 2005 08:56, Ian Pratt wrote:
> > Anyway, since this is a Xen/Kernel header, perhpas we can
> > create a type that represent the Xen/Kernel machine-register size?
>
> Since we're only talking about dom0 ops, it wouldn't be unreasonable to
> turn all the long's into u64's. This would be pretty straightforward,
> arguably easier than bothering with a u_reg type or similar.
>
> Since the dom0_op interface is not guaranteed stable in the 3.0 series
> [we assume you have matched tools and hypervisor] we don't have to get
> this in before 3.0.0. However, it's a pretty simple change and if
> someone who really cared were to knock up a tested patch immediately
> we'd potentially consider it.

I've finally tested this patch on x86, which changes only the dom0_ops 
structures/code. In your mail from yesterday though, you advocate a "ureg" 
type. I guess you would like to see "ureg" used for both dom0 and hypercalls?

-- 
Hollis Blanchard
IBM Linux Technology Center

[-- Attachment #2: xen-dom0-u64.diff --]
[-- Type: text/x-diff, Size: 10165 bytes --]

diff -r f529cd119470 tools/libxc/xc_domain.c
--- a/tools/libxc/xc_domain.c	Thu Sep 29 17:28:28 2005
+++ b/tools/libxc/xc_domain.c	Mon Oct  3 13:04:34 2005
@@ -140,7 +140,7 @@
     op.cmd = DOM0_GETDOMAININFOLIST;
     op.u.getdomaininfolist.first_domain = first_domain;
     op.u.getdomaininfolist.max_domains  = max_domains;
-    op.u.getdomaininfolist.buffer       = info;
+    op.u.getdomaininfolist.buffer       = (u64)(ulong)info;
 
     if ( xc_dom0_op(xc_handle, &op) < 0 )
         ret = -1;
@@ -194,7 +194,7 @@
     op.cmd = DOM0_SHADOW_CONTROL;
     op.u.shadow_control.domain = (domid_t)domid;
     op.u.shadow_control.op     = sop;
-    op.u.shadow_control.dirty_bitmap = dirty_bitmap;
+    op.u.shadow_control.dirty_bitmap = (u64)(ulong)dirty_bitmap;
     op.u.shadow_control.pages  = pages;
 
     rc = do_dom0_op(xc_handle, &op);
diff -r f529cd119470 tools/libxc/xc_private.c
--- a/tools/libxc/xc_private.c	Thu Sep 29 17:28:28 2005
+++ b/tools/libxc/xc_private.c	Mon Oct  3 13:04:34 2005
@@ -72,7 +72,7 @@
     op.cmd = DOM0_GETPAGEFRAMEINFO2;
     op.u.getpageframeinfo2.domain = (domid_t)dom;
     op.u.getpageframeinfo2.num    = num;
-    op.u.getpageframeinfo2.array  = arr;
+    op.u.getpageframeinfo2.array  = (u64)(ulong)arr;
     return do_dom0_op(xc_handle, &op);
 }
 
diff -r f529cd119470 tools/xenstat/libxenstat/src/xen-interface.c
--- a/tools/xenstat/libxenstat/src/xen-interface.c	Thu Sep 29 17:28:28 2005
+++ b/tools/xenstat/libxenstat/src/xen-interface.c	Mon Oct  3 13:04:34 2005
@@ -160,7 +160,7 @@
 	dom0_op_t op;
 	op.u.getdomaininfolist.first_domain = first_domain;
 	op.u.getdomaininfolist.max_domains = max_domains;
-	op.u.getdomaininfolist.buffer = info;
+	op.u.getdomaininfolist.buffer = (u64)(ulong)info;
 
 	if (mlock( info, max_domains * sizeof(dom0_getdomaininfo_t)) < 0) {
 		perror("Failed to mlock domaininfo array");
diff -r f529cd119470 xen/arch/x86/dom0_ops.c
--- a/xen/arch/x86/dom0_ops.c	Thu Sep 29 17:28:28 2005
+++ b/xen/arch/x86/dom0_ops.c	Mon Oct  3 13:04:34 2005
@@ -132,7 +132,8 @@
     case DOM0_MICROCODE:
     {
         extern int microcode_update(void *buf, unsigned long len);
-        ret = microcode_update(op->u.microcode.data, op->u.microcode.length);
+        void *buf = (void *)(ulong)op->u.microcode.data;
+        ret = microcode_update(buf, op->u.microcode.length);
     }
     break;
 
@@ -252,7 +253,7 @@
         int n,j;
         int num = op->u.getpageframeinfo2.num;
         domid_t dom = op->u.getpageframeinfo2.domain;
-        unsigned long *s_ptr = (unsigned long*) op->u.getpageframeinfo2.array;
+        ulong *s_ptr = (ulong*)(ulong)op->u.getpageframeinfo2.array;
         struct domain *d;
         unsigned long *l_arr;
         ret = -ESRCH;
@@ -399,13 +400,15 @@
 
         for ( i = 0; i < e820.nr_map; i++ )
         {
+            struct dom0_memory_map_entry *entries =
+                (struct dom0_memory_map_entry *)(ulong)op->u.physical_memory_map.memory_map;
             if ( i >= op->u.physical_memory_map.max_map_entries )
                 break;
             entry.start  = e820.map[i].addr;
             entry.end    = e820.map[i].addr + e820.map[i].size;
             entry.is_ram = (e820.map[i].type == E820_RAM);
             (void)copy_to_user(
-                &op->u.physical_memory_map.memory_map[i],
+                &entries[i],
                 &entry, sizeof(entry));
         }
 
diff -r f529cd119470 xen/arch/x86/shadow32.c
--- a/xen/arch/x86/shadow32.c	Thu Sep 29 17:28:28 2005
+++ b/xen/arch/x86/shadow32.c	Mon Oct  3 13:04:34 2005
@@ -1224,6 +1224,7 @@
     unsigned int      op = sc->op;
     int               i, rc = 0;
     struct vcpu *v;
+    void *dirty_bitmap = (void *)(ulong)sc->dirty_bitmap;
 
     ASSERT(shadow_lock_is_acquired(d));
 
@@ -1260,7 +1261,7 @@
         d->arch.shadow_dirty_block_count = 0;
  
         if ( (d->max_pages > sc->pages) || 
-             (sc->dirty_bitmap == NULL) || 
+             (dirty_bitmap == NULL) || 
              (d->arch.shadow_dirty_bitmap == NULL) )
         {
             rc = -EINVAL;
@@ -1276,7 +1277,7 @@
                           chunk : (d->max_pages - i)) + 7) / 8;
      
             if (copy_to_user(
-                    sc->dirty_bitmap + (i/(8*sizeof(unsigned long))),
+                    dirty_bitmap + (i/(8*sizeof(unsigned long))),
                     d->arch.shadow_dirty_bitmap +(i/(8*sizeof(unsigned long))),
                     bytes))
             {
@@ -1304,7 +1305,7 @@
         sc->stats.dirty_block_count = d->arch.shadow_dirty_block_count;
  
         if ( (d->max_pages > sc->pages) || 
-             (sc->dirty_bitmap == NULL) || 
+             (dirty_bitmap == NULL) || 
              (d->arch.shadow_dirty_bitmap == NULL) )
         {
             rc = -EINVAL;
@@ -1313,7 +1314,7 @@
  
         sc->pages = d->max_pages;
         if (copy_to_user(
-            sc->dirty_bitmap, d->arch.shadow_dirty_bitmap, (d->max_pages+7)/8))
+            dirty_bitmap, d->arch.shadow_dirty_bitmap, (d->max_pages+7)/8))
         {
             rc = -EINVAL;
             break;
diff -r f529cd119470 xen/common/dom0_ops.c
--- a/xen/common/dom0_ops.c	Thu Sep 29 17:28:28 2005
+++ b/xen/common/dom0_ops.c	Mon Oct  3 13:04:34 2005
@@ -359,7 +359,8 @@
     { 
         struct domain *d;
         dom0_getdomaininfo_t info;
-        dom0_getdomaininfo_t *buffer = op->u.getdomaininfolist.buffer;
+        dom0_getdomaininfo_t *buffer =
+            (dom0_getdomaininfo_t *)(ulong)op->u.getdomaininfolist.buffer;
         u32 num_domains = 0;
 
         read_lock(&domlist_lock);
diff -r f529cd119470 xen/include/public/dom0_ops.h
--- a/xen/include/public/dom0_ops.h	Thu Sep 29 17:28:28 2005
+++ b/xen/include/public/dom0_ops.h	Mon Oct  3 13:04:34 2005
@@ -27,10 +27,10 @@
 typedef struct {
     /* IN variables. */
     domid_t       domain;
-    unsigned long max_pfns;
+    u64           max_pfns;
     void         *buffer;
     /* OUT variables. */
-    unsigned long num_pfns;
+    u64           num_pfns;
 } dom0_getmemlist_t;
 
 #define DOM0_SCHEDCTL          6
@@ -83,9 +83,9 @@
 #define DOMFLAGS_SHUTDOWNMASK 255 /* DOMFLAGS_SHUTDOWN guest-supplied code.  */
 #define DOMFLAGS_SHUTDOWNSHIFT 16
     u32      flags;
-    unsigned long tot_pages;
-    unsigned long max_pages;
-    unsigned long shared_info_frame;       /* MFN of shared_info struct */
+    u64      tot_pages;
+    u64      max_pages;
+    u64      shared_info_frame;       /* MFN of shared_info struct */
     u64      cpu_time;
     u32      n_vcpu;
     s32      vcpu_to_cpu[MAX_VIRT_CPUS];  /* current mapping   */
@@ -155,7 +155,7 @@
 
 typedef struct {
     /* IN variables. */
-    unsigned long pfn;     /* Machine page frame number to query.       */
+    u64 pfn;               /* Machine page frame number to query.       */
     domid_t domain;        /* To which domain does the frame belong?    */
     /* OUT variables. */
     /* Is the page PINNED to a type? */
@@ -194,10 +194,10 @@
 #define DOM0_TBUF_SET_EVT_MASK 2
     u8 op;
     /* IN/OUT variables */
-    unsigned long cpu_mask;
+    u64           cpu_mask;
     u32           evt_mask;
     /* OUT variables */
-    unsigned long buffer_mfn;
+    u64           buffer_mfn;
     u32      size;
 } dom0_tbufcontrol_t;
 
@@ -211,8 +211,8 @@
     u32      sockets_per_node;
     u32      nr_nodes;
     u32      cpu_khz;
-    unsigned long total_pages;
-    unsigned long free_pages;
+    u64      total_pages;
+    u64      free_pages;
     u32      hw_cap[8];
 } dom0_physinfo_t;
 
@@ -251,9 +251,9 @@
     /* IN variables. */
     domid_t        domain;
     u32            op;
-    unsigned long *dirty_bitmap; /* pointer to locked buffer */
+    u64            dirty_bitmap; /* pointer to locked buffer */
     /* IN/OUT variables. */
-    unsigned long  pages;        /* size of buffer, updated with actual size */
+    u64            pages;        /* size of buffer, updated with actual size */
     /* OUT variables. */
     dom0_shadow_control_stats_t stats;
 } dom0_shadow_control_t;
@@ -262,16 +262,16 @@
 typedef struct {
     /* IN variables. */
     domid_t       domain;
-    unsigned long max_memkb;
+    u64           max_memkb;
 } dom0_setdomainmaxmem_t;
 
 #define DOM0_GETPAGEFRAMEINFO2 29   /* batched interface */
 typedef struct {
     /* IN variables. */
     domid_t        domain;
-    unsigned long  num;
+    u64            num;
     /* IN/OUT variables. */
-    unsigned long *array;
+    u64            array;
 } dom0_getpageframeinfo2_t;
 
 /*
@@ -284,8 +284,8 @@
 #define DOM0_ADD_MEMTYPE         31
 typedef struct {
     /* IN variables. */
-    unsigned long pfn;
-    unsigned long nr_pfns;
+    u64           pfn;
+    u64           nr_pfns;
     u32           type;
     /* OUT variables. */
     u32           handle;
@@ -312,8 +312,8 @@
     /* IN variables. */
     u32      reg;
     /* OUT variables. */
-    unsigned long pfn;
-    unsigned long nr_pfns;
+    u64      pfn;
+    u64      nr_pfns;
     u32      type;
 } dom0_read_memtype_t;
 
@@ -338,7 +338,7 @@
 #define DOM0_MICROCODE           35
 typedef struct {
     /* IN variables. */
-    void   *data;                     /* Pointer to microcode data */
+    u64     data;                     /* Pointer to microcode data */
     u32     length;                   /* Length of microcode data. */
 } dom0_microcode_t;
 
@@ -363,7 +363,7 @@
     /* IN variables. */
     domid_t               first_domain;
     unsigned int          max_domains;
-    dom0_getdomaininfo_t *buffer;
+    u64                   buffer; /* dom0_getdomaininfo_t* */
     /* OUT variables. */
     unsigned int          num_domains;
 } dom0_getdomaininfolist_t;
@@ -376,15 +376,16 @@
 } dom0_platform_quirk_t;
 
 #define DOM0_PHYSICAL_MEMORY_MAP 40
+struct dom0_memory_map_entry {
+    u64 start, end;
+    int is_ram;
+};
 typedef struct {
     /* IN variables. */
     int max_map_entries;
     /* OUT variables. */
     int nr_map_entries;
-    struct dom0_memory_map_entry {
-        u64 start, end;
-        int is_ram;
-    } *memory_map;
+    u64 memory_map; /* dom0_memory_map_entry* */
 } dom0_physical_memory_map_t;
 
 typedef struct {

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: 32/64-bit hypercall interface

[Nakajima, Jun]

Hollis Blanchard wrote:
> On Monday 03 October 2005 16:24, Nakajima, Jun wrote:
>> Hollis Blanchard wrote:
>>> As mentioned previously, this is the approach Linux uses
>>> (linux/fs/compat_ioctl.c), and it seems less than ideal to me. Since
>>> we have the ability to fix it now (i.e. make the 32-bit and 64-bit
>>> ABI identical), shouldn't we do that rather than this
>>> copying/munging layer?
>> 
>> The 32-bit and 64-bit hypercall ABI cannot be identical on x86
>> because of the generic ABI difference between 32-bit and 64-bit.
> 
> I am not talking about the standard ABI used by the compiler (ppc32
> and ppc64 use different ABIs as well). I am talking about the
> hypervisor/tools and hypervisor/kernel ABI.
> 
> If the hypervisor ABI does not contain types that change size, it
> will be identical for both 32- and 64-bit users.
> 
Okay, I understand what you need. How about the data layout? That's not
so trivial to have the identical ones. 
For example, the following one would have different data layout on
32-bit and 64-bit. 

typedef struct vcpu_time_info {
    /*
     * Updates to the following values are preceded and followed by an
     * increment of 'version'. The guest can therefore detect updates by
     * looking for changes to 'version'. If the least-significant bit of
     * the version number is set then an update is in progress and the
guest
     * must wait to read a consistent set of values.
     * The correct way to interact with the version number is similar to
     * Linux's seqlock: see the implementations of
read_seqbegin/read_seqretry.
     */
    u32 version;
    u64 tsc_timestamp;   /* TSC at last update of time vals.  */
    u64 system_time;     /* Time, in nanosecs, since boot.    */
    /*
     * Current system time:
     *   system_time + ((tsc - tsc_timestamp) << tsc_shift) *
tsc_to_system_mul
     * CPU frequency (Hz):
     *   ((10^9 << 32) / tsc_to_system_mul) >> tsc_shift
     */
    u32 tsc_to_system_mul;
    s8  tsc_shift;
} vcpu_time_info_t;


> If you meant something else when you said "generic ABI difference,"
> could you explain?

I meant basically the "standard ABI", such as, parameter passing, i.e.
which registers are used to pass which arguments. x86_64 xenlinux runs
in ring3, and hypercalls are not so different from system calls in terms
of the mechanism/implementation. Then when we use fast system calls, we
can use the compiler convention to minimize save/restore of the
registers.

Jun
---
Intel Open Source Technology Center

Re: 32/64-bit hypercall interface

[Hollis Blanchard]

On Tuesday 04 October 2005 16:51, Nakajima, Jun wrote:
> Okay, I understand what you need. How about the data layout? That's not
> so trivial to have the identical ones.
> For example, the following one would have different data layout on
> 32-bit and 64-bit.
>
> typedef struct vcpu_time_info {
>     u32 version;
>     u64 tsc_timestamp;   /* TSC at last update of time vals.  */
>     u64 system_time;     /* Time, in nanosecs, since boot.    */
>     u32 tsc_to_system_mul;
>     s8  tsc_shift;
> } vcpu_time_info_t;

Is that because of the u64 4-byte alignment Andi mentioned? To use u64 in 
xen/include/public, we will need to ensure that __aligned__((8)) is used.

The padding in some of these structures is unfortunate in terms of wasted 
(cacheline) space, but for now I will be happy with commonizing the size and 
layout. This particular structure couldn't be compacted anyways... However, 
Keir indicated earlier he was not interested in reordering any structures, as 
would be needed to reduce wasted padding space.

-- 
Hollis Blanchard
IBM Linux Technology Center

RE: 32/64-bit hypercall interface

[Nakajima, Jun]

ak@suse.de wrote:
> "Nakajima, Jun" <jun.nakajima@intel.com> writes:
>> 
>> The 32-bit and 64-bit hypercall ABI cannot be identical on x86
>> because of the generic ABI difference between 32-bit and 64-bit.
> 
> You use a custom ABI for hypercalls anyways, so you can define
> it to be the same with some care. That is the approach that is
> used by some newer subsystems in the Linux kernel.
> 
In 32-bit, there are a couple of different instructions used for system
calls, e.g. "int N", sysenter, and syscall. Some x86 CPUs support only
sysenter, and some syscall, and "int N" is always available and it is
the one used for 32-bit hypercalls. In 64-bit, the situation is much
better, and we are using syscall for hypercalls, which is always
available and can be more efficient than "int N". 

I don't think we should use "int N" sacrificing the good feature for
64-bit when we can handle 32-bit hypercalls using a compatibility layer.
I also think using the existing ABI convention is much better whenever
possible and appropriate because we don't need to re-invent the wheel.
After all, the only difference between system call and hyper call
basically is the system/hypercall number.

> In my experience even people with the best intentions tend to get that
> wrong at some point because there are some subtle issues so having the
> additional safety net of a compat layer is still a good idea. It would
> only emulate the ones that went wrong.
> 
> -Andi

I agree. 

Jun
---
Intel Open Source Technology Center

Re: 32/64-bit hypercall interface

[Andi Kleen]

On Tuesday 04 October 2005 18:03, Nakajima, Jun wrote:

>
> I don't think we should use "int N" sacrificing the good feature for
> 64-bit when we can handle 32-bit hypercalls using a compatibility layer.
> I also think using the existing ABI convention is much better whenever
> possible and appropriate because we don't need to re-invent the wheel.
> After all, the only difference between system call and hyper call
> basically is the system/hypercall number.

Main 32bit linux solves this problem by mapping a vsyscall page into the
address space. The ABI is then just to jump to an appropiate entry point in 
it. The entry point uses the right instruction then.
 
-Andi

RE: 32/64-bit hypercall interface

[Ian Pratt]

 > However, doesn't that same argument apply to correcting the 
> ABI in the first place? Shadow page tables will overshadow 
> the performance impact of making the ABI 32/64-bit clean.
> 
> In fact, even a plain old hypercall will also overshadow that 
> performance impact, both in terms of cycle count and cache footprint.
> 
> So if your choice then is between a compatibility translation 
> layer and altering the interface, I think it's pretty clear 
> that changing the interface will result in the least amount 
> of additional code (and associated long-term code maintenance).

This would result in doubling the size of the all the p2m and m2p
tables, which I really don't think would be sensible: the physcial
memory consumed would be somewhat annoying, but we'd also have to rejig
the virtual memory layout to accommodate the larger tables (consuming
more lowmem) and this really doesn't make sense this late in the 3.0 dev
cycle. We'd also double the cache foot print of some quite performance
critical operations. You don't need many cache misses to dominate the
cost of a system call...


Ian

Re: 32/64-bit hypercall interface

[Hollis Blanchard]

On Monday 03 October 2005 17:03, Ian Pratt wrote:
> > However, doesn't that same argument apply to correcting the
> > ABI in the first place? Shadow page tables will overshadow
> > the performance impact of making the ABI 32/64-bit clean.
> >
> > In fact, even a plain old hypercall will also overshadow that
> > performance impact, both in terms of cycle count and cache footprint.
> >
> > So if your choice then is between a compatibility translation
> > layer and altering the interface, I think it's pretty clear
> > that changing the interface will result in the least amount
> > of additional code (and associated long-term code maintenance).
>
> This would result in doubling the size of the all the p2m and m2p
> tables,

Would it (honest question)? Those tables aren't part of the hypercall 
interface itself, right? So as long as the hypercalls dealing with those 
tables are modified appropriately, the tables themselves don't need the 
change?

-- 
Hollis Blanchard
IBM Linux Technology Center

Re: 32/64-bit hypercall interface

[Keir Fraser]

On 4 Oct 2005, at 19:05, Hollis Blanchard wrote:

> Would it (honest question)? Those tables aren't part of the hypercall
> interface itself, right? So as long as the hypercalls dealing with 
> those
> tables are modified appropriately, the tables themselves don't need the
> change?

Well that's true. Same holds for the pfn lists passed to GETMEMLIST, 
GETPAGEFRAMEINFO2, and the memory_op() hypercall.

  -- Keir

RE: 32/64-bit hypercall interface

[Nakajima, Jun]

Hollis Blanchard wrote:
> On Monday 03 October 2005 14:11, Nakajima, Jun wrote:
>> In terms of ABI/API, since Xen needs to disiguish 32-bit or 64-bit
>> guests anyway at runtime, I don't think we don't need to change the
>> size of any types at this point (i.e. before 3.0).
> 
> You would instead propose a compatibility layer in Xen? So when a
> hypercall from a 32-bit guest arrives at a 64-bit hypervisor, Xen
> code converts the 32-bit structure into a 64-bit one and passes that
> pointer on to the rest of Xen? And then for return values you'd
> convert the other way. Hmm, and of course you wouldn't be able to
> pass 64-bit addresses back, such as via dom0_tbufcontrol_t.

I don't think dom0_tbufcontrol_t is a good example (as dicussed). Do you
have other examples?

> 
> As mentioned previously, this is the approach Linux uses
> (linux/fs/compat_ioctl.c), and it seems less than ideal to me. Since
> we have the ability to fix it now (i.e. make the 32-bit and 64-bit
> ABI identical), shouldn't we do that rather than this copying/munging
> layer? 

The 32-bit and 64-bit hypercall ABI cannot be identical on x86 because
of the generic ABI difference between 32-bit and 64-bit. 

Jun
---
Intel Open Source Technology Center

Re: 32/64-bit hypercall interface

[Andi Kleen]

"Nakajima, Jun" <jun.nakajima@intel.com> writes:
> 
> The 32-bit and 64-bit hypercall ABI cannot be identical on x86 because
> of the generic ABI difference between 32-bit and 64-bit. 

You use a custom ABI for hypercalls anyways, so you can define
it to be the same with some care. That is the approach that is 
used by some newer subsystems in the Linux kernel.

In my experience even people with the best intentions tend to get that
wrong at some point because there are some subtle issues so having the
additional safety net of a compat layer is still a good idea. It would
only emulate the ones that went wrong.

-Andi

Re: 32/64-bit hypercall interface

[Hollis Blanchard]

On Monday 03 October 2005 16:24, Nakajima, Jun wrote:
> Hollis Blanchard wrote:
> > As mentioned previously, this is the approach Linux uses
> > (linux/fs/compat_ioctl.c), and it seems less than ideal to me. Since
> > we have the ability to fix it now (i.e. make the 32-bit and 64-bit
> > ABI identical), shouldn't we do that rather than this copying/munging
> > layer?
>
> The 32-bit and 64-bit hypercall ABI cannot be identical on x86 because
> of the generic ABI difference between 32-bit and 64-bit.

I am not talking about the standard ABI used by the compiler (ppc32 and ppc64 
use different ABIs as well). I am talking about the hypervisor/tools and 
hypervisor/kernel ABI.

If the hypervisor ABI does not contain types that change size, it will be 
identical for both 32- and 64-bit users.

If you meant something else when you said "generic ABI difference," could you 
explain?

-- 
Hollis Blanchard
IBM Linux Technology Center

RE: 32/64-bit hypercall interface

[Ian Pratt]

> You would instead propose a compatibility layer in Xen? So 
> when a hypercall from a 32-bit guest arrives at a 64-bit 
> hypervisor, Xen code converts the 32-bit structure into a 
> 64-bit one and passes that pointer on to the rest of Xen? And 
> then for return values you'd convert the other way. Hmm, and 
> of course you wouldn't be able to pass 64-bit addresses back, 
> such as via dom0_tbufcontrol_t.

The dom0 API is a xen <-> tools API. That's *not* part of the gen guest
API.

Needing 64 bit *capable* tools for a 64 bit hypevisor is clearly a
requirement. [The fact that Power wants to implement 64 bit capable
tools as a 32 bit user space binary is totally orthogonal (and some
might say slightly perverse)]  
 
> As mentioned previously, this is the approach Linux uses 
> (linux/fs/compat_ioctl.c), and it seems less than ideal to 
> me. Since we have the ability to fix it now (i.e. make the 
> 32-bit and 64-bit ABI identical), shouldn't we do that rather 
> than this copying/munging layer?

Although I wouldn't object to using u64 everywhere in the tools (dom0)
API, I most certainly would object to using it everywhere in the 32 bit
x86 guest API as it would hurt performance and waste memory.
*Definitely*.

Just changing the tools API to all u64 seems pointless, as if you're
using a 64 bit hypervisor it seems perfectly reasonable to insist on
using 64 bit tools -- every other system binary on your 64 bit
redhat/suse install is going to be specially compiled for 64 bit, so why
not yout hypervisor tools! 

If we're going to make a change at all, switching to using ureg (or
similar) rather than 'long' is the thing to do to. This makes the job of
implementing some future compat layer very slightly easier, and helps
the Power guys do their funky 64-bit-tools-as-a-32-bit-binary thing. [*]
 
Ian

[*] implementing a hypercall compat layer is trivial compared to other
overheads you'd have (like shadow page tables). 

RE: 32/64-bit hypercall interface

[Jeremy Katz]

On Mon, 2005-10-03 at 20:56 +0100, Ian Pratt wrote:
> Just changing the tools API to all u64 seems pointless, as if you're
> using a 64 bit hypervisor it seems perfectly reasonable to insist on
> using 64 bit tools -- every other system binary on your 64 bit
> redhat/suse install is going to be specially compiled for 64 bit, so
> why not yout hypervisor tools!  

Although this is the case for x86_64, it is _NOT_ the case for ppc64 (or
sparc64 if someone is ever demented enough to do it ;)

ppc64 Linux ships with a largely 32-bit userland + a 64-bit kernel.  And
since the Xen tools are python, doing 64-bit tools in an environment
like that gets very tricky -- the various scripting languages are not
done in such a way that you can easily install both 32 and 64-bit
versions

Jeremy

Re: 32/64-bit hypercall interface

[Hollis Blanchard]

On Monday 03 October 2005 14:56, Ian Pratt wrote:
> [*] implementing a hypercall compat layer is trivial compared to other
> overheads you'd have (like shadow page tables).

I agree with this statement.

However, doesn't that same argument apply to correcting the ABI in the first 
place? Shadow page tables will overshadow the performance impact of making 
the ABI 32/64-bit clean.

In fact, even a plain old hypercall will also overshadow that performance 
impact, both in terms of cycle count and cache footprint.

So if your choice then is between a compatibility translation layer and 
altering the interface, I think it's pretty clear that changing the interface 
will result in the least amount of additional code (and associated long-term 
code maintenance).

-- 
Hollis Blanchard
IBM Linux Technology Center

Re: 32/64-bit hypercall interface

[Keir Fraser]

On 3 Oct 2005, at 22:11, Hollis Blanchard wrote:

> However, doesn't that same argument apply to correcting the ABI in the 
> first
> place? Shadow page tables will overshadow the performance impact of 
> making
> the ABI 32/64-bit clean.

The cost must also be paid by domains that don't require shadow page 
tables (32-bit on 32-bit, in particular).

The 64-bit dom0_op patch you posted misses 64-bit-ifying the page-frame 
arrays of GETMEMLIST and GETPAGEFRAMEINFO2. I had a go at 64-bit-ifying 
the dom0_op interface myself, and gave up at those two because they are 
particularly invasive to change.

  -- Keir

Re: 32/64-bit hypercall interface

[Hollis Blanchard]

On Monday 03 October 2005 17:00, Keir Fraser wrote:
>
> The cost must also be paid by domains that don't require shadow page
> tables (32-bit on 32-bit, in particular).

The next sentence in my mail said that a normal hypercall would effectively 
have the same performance impact...

> The 64-bit dom0_op patch you posted misses 64-bit-ifying the page-frame
> arrays of GETMEMLIST and GETPAGEFRAMEINFO2. I had a go at 64-bit-ifying
> the dom0_op interface myself, and gave up at those two because they are
> particularly invasive to change.

Couldn't a field be added indicating the size of the resulting type? I have to 
run right now, but I'll think about it more.

-- 
Hollis Blanchard
IBM Linux Technology Center

Re: 32/64-bit hypercall interface

[Andi Kleen]

"Ian Pratt" <m+Ian.Pratt@cl.cam.ac.uk> writes:
> 
> Just changing the tools API to all u64 seems pointless, as if you're
> using a 64 bit hypervisor it seems perfectly reasonable to insist on
> using 64 bit tools -- every other system binary on your 64 bit
> redhat/suse install is going to be specially compiled for 64 bit, so why
> not yout hypervisor tools! 

Actually a 64bit kernel works just fine with a full 32bit userland
(with only a few minor exception of some tools which were too broken
to be emulated - most notable one is iptables) People using
distributions where using true 64bit is unusually hard or broken (like on a
unnamed particular big community distribution) tend to use this setup.

I use it also regularly for testing.

On the other hand if the people need some tools as 64bit anyways
then they can install 64bit hypervisor tools too. If you do that
you should make it easy to install under a different prefix though.

> If we're going to make a change at all, switching to using ureg (or
> similar) rather than 'long' is the thing to do to. This makes the job of
> implementing some future compat layer very slightly easier, and helps
> the Power guys do their funky 64-bit-tools-as-a-32-bit-binary thing. [*]

There are some traps. e.g. i386 has a different alignment for u64 (4
bytes) rather than the natural 8 bytes used on 64bits which causes
many problems (like the iptables issue mentioned above). If you use
this way use a new type which is long long
__attribute__((aligned(8))), not just long long.

-Andi

RE: 32/64-bit hypercall interface

[Nakajima, Jun]

Jeremy Katz wrote:
> On Sat, 2005-10-01 at 11:25 -0700, Nakajima, Jun wrote:
>> Jeremy Katz wrote:
>>> On Fri, 2005-09-30 at 17:42 +0100, Keir Fraser wrote:
>>>> There's the rub: we don't expect to ever want to provide 32-bit x86
>>>> ABI compatibility on 64-bit x86 Xen. We will not be supporting
>>>> 32-bit paravirtualised guests on 64-bit x86 Xen,
>>> 
>>> ... which I've previously said and continue to think is going to be
>>> something that will eventually be regretted.  People are going to
>>> want to continue to run in a 32bit OS for legacy reasons for quite
>>> a while (even with the compatibility you get on x86_64).  Making it
>>> so they can't do mix and match of 32 and 64 bit guests on a single
>>> host is going to significantly reduce the utility of Xen.
> 
>> That's mostly because of the H/W issues. One thing we could do there
>> is to run such  paravirtualised 32-bit guests in compatibility mode.
>> But they would need to use 4-level page tables, and there are other
>> minor differences. So those 32-bit guests wouldn't be really same as
>> the ones running on 32-bit Xen. I'm not sure that's what you wanted,
>> but it's doable.
> 
> You really want to get to the same 32bit paravirt guest being used
> regardless of what HV flavor you're running.  Otherwise, you're in
> for a compatibility nitemare.  What dictates that the guest has to
> know about the 4 level page tables?  That's like saying that your
> 32bit binaries running on 64bit Linux need to know that there is a
> larger address space when instead that gets handled by the kernel.
> 

I fully agree that the same binary of 32bit paravirt guest be used
regardless of the HV flavor (i.e. 32-bit or 64-bit). 

If we get the shadow mode working for those 32-bit paravirt guests, we
could avoid using 4-level page tables in the guest kernel (although the
processor actually uses 4-level page tables). In that case, however, we
will lose the good optimizations for paravirt guests, such as writeable
page tables and "late pin, early unpin." (but then we get the
save/restore/migration feature) There are other areas to look at, such
as, ring1 vs. ring3, but I think we can handle those in Xen.

>> I think a better way is to utilize H/W-based virtualization such as
>> Intel Virtualization Technology (VT) or AMD's Pacifica. That way we
>> should be able to use the same binaries (thus ABI) for both 32-bit
>> non-PAE and PAE domU on 64-bit x86 Xen. With H/W-based
>> virtualization we can run those 32-bit guests cleanly in 32-bit on
>> 64-bit Xen. 32-bit hypercalls from such 32-bit guests should be
>> intercepted by a 32-bit ring0 component that converts "int 0x82"
>> based ones to VMCALL or VMMCALL sent to 64-bit Xen (because such
>> software interrupts won't cause VMEXIT). There are other issues like
>> impacts to the memory management in Xen, but I think those can be
>> handled as special cases of shadow mode (i.e. don't make shadow
>> pages). I don't think it would increase the utility of Xen right
>> away (because this requires new processors), but it might be a
>> sensible thing to do in the near future. 
> 
> Right, the problem is that people have bought hardware and they're
> going to want to use it.  I think that having things work on the
> hardware I've already purchased will help to give people a much
> better migration path for the future.  If I'm going to have to run my
> legacy 32bit stuff on a different machine anyway, then I can feel
> more free to look at other platforms for my "future" stuff.
> 
> Jeremy

That's a very good point, and that way we could provide more complete
32-bit compatibility than the one we get on x86_64 Linux.

In terms of ABI/API, since Xen needs to disiguish 32-bit or 64-bit
guests anyway at runtime, I don't think we don't need to change the size
of any types at this point (i.e. before 3.0).

Jun
---
Intel Open Source Technology Center

Re: 32/64-bit hypercall interface

[Hollis Blanchard]

On Monday 03 October 2005 14:11, Nakajima, Jun wrote:
> In terms of ABI/API, since Xen needs to disiguish 32-bit or 64-bit
> guests anyway at runtime, I don't think we don't need to change the size
> of any types at this point (i.e. before 3.0).

You would instead propose a compatibility layer in Xen? So when a hypercall 
from a 32-bit guest arrives at a 64-bit hypervisor, Xen code converts the 
32-bit structure into a 64-bit one and passes that pointer on to the rest of 
Xen? And then for return values you'd convert the other way. Hmm, and of 
course you wouldn't be able to pass 64-bit addresses back, such as via 
dom0_tbufcontrol_t.

As mentioned previously, this is the approach Linux uses 
(linux/fs/compat_ioctl.c), and it seems less than ideal to me. Since we have 
the ability to fix it now (i.e. make the 32-bit and 64-bit ABI identical), 
shouldn't we do that rather than this copying/munging layer?

-- 
Hollis Blanchard
IBM Linux Technology Center

RE: 32/64-bit hypercall interface

[Ian Pratt]

> start_info_t: tools, kernel
> xen_parameters_info_t: tools, hypervisor
> gnttab_setup_table_t: kernel, hypervisor
> gnttab_transfer_t: kernel, hypervisor
> struct xen_memory_reservation: tools, kernel, hypervisor 
> struct t_rec: tools, hypervisor struct t_buf: tools, hypervisor
> 
> (This was just a quick grep for "long" in xen/include/public.)
> 
> The tools/hypervisor interface seems to be the most important 
> at the moment. I don't know if anybody has plans to do 32-bit 
> kernel on a 64-bit hypervisor... 
> but why not enable it while we're in there?

There's rather more to running a 32 bit x86 xen guest on a 64 bit
hypervisor then just worrying about the header files. I wouldn't rule
out ever doing it, but there'll be a significant performance cost (e.g.
it will require shadow pagetables) and a significant amount of work.

The hypervisor to guest interface is obviously performance critical, and
just changing everything into a 64 bit quantity certainly would not make
sense from a memory usage, cache footprint and instruction count point
of view. Looking through the public guest interface, the current use of
'long' to get 32 bit quantities on x86_32 and 64 on x86_64 actually
makes good sense.

However, I don't think we should be using 'long' directly in public
headers as it makes it harder to override. For example, if we ever
support 32 bit guests on a 64 bit hypervisor we may want to use multiple
compilation to build a compatibility layer or such like. We should
probably typedef something like 'ureg' that we could override as
appropriate. This would also enable the Power folks to build a 32 bit
binary tools for a 64 bit hypervisor.

[I think I favour using a single typedef like 'ureg' rather than coming
up coming up with different names for all of the different uses of
'long' as the latter seems faorly pointless]

However, since we're not actually changing the size of any types, this
change isn't essential to rush through before 3.0, though it might be
nice as it should be very low risk.

Ian

RE: 32/64-bit hypercall interface

[Nakajima, Jun]

Jeremy Katz wrote:
> On Fri, 2005-09-30 at 17:42 +0100, Keir Fraser wrote:
>> There's the rub: we don't expect to ever want to provide 32-bit x86
>> ABI compatibility on 64-bit x86 Xen. We will not be supporting 32-bit
>> paravirtualised guests on 64-bit x86 Xen,
> 
> ... which I've previously said and continue to think is going to be
> something that will eventually be regretted.  People are going to want
> to continue to run in a 32bit OS for legacy reasons for quite a while
> (even with the compatibility you get on x86_64).  Making it so they
> can't do mix and match of 32 and 64 bit guests on a single host is
> going to significantly reduce the utility of Xen.
> 
> Jeremy
> 
> 
Jeremy, Hi

That's mostly because of the H/W issues. One thing we could do there is
to run such  paravirtualised 32-bit guests in compatibility mode. But
they would need to use 4-level page tables, and there are other minor
differences. So those 32-bit guests wouldn't be really same as the ones
running on 32-bit Xen. I'm not sure that's what you wanted, but it's
doable.

I think a better way is to utilize H/W-based virtualization such as
Intel Virtualization Technology (VT) or AMD's Pacifica. That way we
should be able to use the same binaries (thus ABI) for both 32-bit
non-PAE and PAE domU on 64-bit x86 Xen. With H/W-based virtualization we
can run those 32-bit guests cleanly in 32-bit on 64-bit Xen. 32-bit
hypercalls from such 32-bit guests should be intercepted by a 32-bit
ring0 component that converts "int 0x82" based ones to VMCALL or VMMCALL
sent to 64-bit Xen (because such software interrupts won't cause
VMEXIT). There are other issues like impacts to the memory management in
Xen, but I think those can be handled as special cases of shadow mode
(i.e. don't make shadow pages). I don't think it would increase the
utility of Xen right away (because this requires new processors), but it
might be a sensible thing to do in the near future.

For full-virutalization, today we can mix 32 and 64 bit guests on x86_64
Xen (as in the xen-unstable tree) when the H/W-based virtualization is
present on the machine.

Jun
---
Intel Open Source Technology Center

RE: 32/64-bit hypercall interface

[Jeremy Katz]

On Sat, 2005-10-01 at 11:25 -0700, Nakajima, Jun wrote:
> Jeremy Katz wrote:
> > On Fri, 2005-09-30 at 17:42 +0100, Keir Fraser wrote:
> >> There's the rub: we don't expect to ever want to provide 32-bit x86
> >> ABI compatibility on 64-bit x86 Xen. We will not be supporting 32-bit
> >> paravirtualised guests on 64-bit x86 Xen,
> > 
> > ... which I've previously said and continue to think is going to be
> > something that will eventually be regretted.  People are going to want
> > to continue to run in a 32bit OS for legacy reasons for quite a while
> > (even with the compatibility you get on x86_64).  Making it so they
> > can't do mix and match of 32 and 64 bit guests on a single host is
> > going to significantly reduce the utility of Xen.

> That's mostly because of the H/W issues. One thing we could do there is
> to run such  paravirtualised 32-bit guests in compatibility mode. But
> they would need to use 4-level page tables, and there are other minor
> differences. So those 32-bit guests wouldn't be really same as the ones
> running on 32-bit Xen. I'm not sure that's what you wanted, but it's
> doable.

You really want to get to the same 32bit paravirt guest being used
regardless of what HV flavor you're running.  Otherwise, you're in for a
compatibility nitemare.  What dictates that the guest has to know about
the 4 level page tables?  That's like saying that your 32bit binaries
running on 64bit Linux need to know that there is a larger address space
when instead that gets handled by the kernel.

> I think a better way is to utilize H/W-based virtualization such as
> Intel Virtualization Technology (VT) or AMD's Pacifica. That way we
> should be able to use the same binaries (thus ABI) for both 32-bit
> non-PAE and PAE domU on 64-bit x86 Xen. With H/W-based virtualization we
> can run those 32-bit guests cleanly in 32-bit on 64-bit Xen. 32-bit
> hypercalls from such 32-bit guests should be intercepted by a 32-bit
> ring0 component that converts "int 0x82" based ones to VMCALL or VMMCALL
> sent to 64-bit Xen (because such software interrupts won't cause
> VMEXIT). There are other issues like impacts to the memory management in
> Xen, but I think those can be handled as special cases of shadow mode
> (i.e. don't make shadow pages). I don't think it would increase the
> utility of Xen right away (because this requires new processors), but it
> might be a sensible thing to do in the near future.

Right, the problem is that people have bought hardware and they're going
to want to use it.  I think that having things work on the hardware I've
already purchased will help to give people a much better migration path
for the future.  If I'm going to have to run my legacy 32bit stuff on a
different machine anyway, then I can feel more free to look at other
platforms for my "future" stuff.

Jeremy

32/64-bit hypercall interface

[Hollis Blanchard]

I was looking at the definition of privcmd_hypercall:
typedef struct privcmd_hypercall
{
    unsigned long op;
    unsigned long arg[5];
} privcmd_hypercall_t;

For ppc64, we are using 32-bit management tools, so this is a problem: they 
will create structs where long is 32 bits, and the (64-bit) kernel and 
hypervisor will expect structs where long is 64 bits.

The standard (and awkward) way of dealing with the privcmd ioctl is to create 
an in-kernel privcmd_hypercall_t, copy the 32-bit values into it field by 
field, and then pass *that* struct on to privcmd_ioctl(). Of course, that's 
only for legacy interfaces; for all new interfaces, we can just design them 
properly so that their size and alignment doesn't change.

There are also longs in some of the dom0_op sub-structures. For example:
typedef struct {
    /* IN variables. */
    domid_t       domain;
    unsigned long max_memkb;
} dom0_setdomainmaxmem_t;

I suggest that all longs in these structures be converted to u32.

-- 
Hollis Blanchard
IBM Linux Technology Center

Re: 32/64-bit hypercall interface

[Keir Fraser]

On 28 Sep 2005, at 22:36, Hollis Blanchard wrote:

> I was looking at the definition of privcmd_hypercall:
> typedef struct privcmd_hypercall
> {
>     unsigned long op;
>     unsigned long arg[5];
> } privcmd_hypercall_t;
>
> For ppc64, we are using 32-bit management tools, so this is a problem: 
> they
> will create structs where long is 32 bits, and the (64-bit) kernel and
> hypervisor will expect structs where long is 64 bits.

We want the args to be 'machine-register sized' which is why they are 
longs. Why use 32-bit tools on 64-bit hypervisor?

  -- Keir

Re: 32/64-bit hypercall interface

[Jimi Xenidis]

>>>>> "KF" == Keir Fraser <Keir.Fraser@cl.cam.ac.uk> writes:

 KF> On 28 Sep 2005, at 22:36, Hollis Blanchard wrote:

 KF> We want the args to be 'machine-register sized' which is why they
 KF> are longs.

hmm, It appears that 32bits is sufficient, esp since almost everything
is expressed in terms of pages (takes us to 16TB). However, we should
be forward thinking :).

Anyway, since this is a Xen/Kernel header, perhpas we can create a
type that represent the Xen/Kernel machine-register size?

 KF> Why use 32-bit tools on 64-bit hypervisor?

Unlike x86_64 and ia64, there is no performance gain by making regular
apps 64bit on PPC, so we tend to have almost all our apps be the same
32 bit apps that run on 32 bit kernels.  This way, when Xen-PPC32 is
ready the same xentools binary will apply.

Besides, most systems don't have a 64bit python or libz installed or
readily available.

BTW: Is the plan to not support 32bit domain (0 or U) on a 64bit Xen?

-JX
-- 
 "I got an idea, an idea so smart my head would explode if I even
  began to know what I was talking about." -- Peter Griffin (Family Guy)

Re: 32/64-bit hypercall interface

[Keir Fraser]

On 29 Sep 2005, at 13:41, Jimi Xenidis wrote:

> BTW: Is the plan to not support 32bit domain (0 or U) on a 64bit Xen?

That's the case for paravirtualised x86 guests.

  -- Keir