xen-4.3 port

xen-4.3 port

[Igor Kozhukhov]

Hello All,

i'm working on port xen-4.3 to DilOS (illumos based platform).

i have problems with PV guest load.
dom0 started, i can see info by 'xl info'.

first: i see platform ID=38, but i couldn't found it in xen/public/platform.h

Jan 28 01:16:44 myhost privcmd: == HYPERVISOR_platform_op 38
Jan 28 01:16:44 myhost privcmd: unrecognized HYPERVISOR_platform_op 38

could you please let me know - what is it the 38 platform hypercall ?

next problem: i try to create a new PV guest by 'xm create dilos.cfg'
i see on dom0 logs requests to hypercall ID=38 - i see it is __HYPERVISOR_tmem_op

do i need implement it first ?

i have error with :
root@myhost:/xen# xm create dilos.cfg 
Using config file "./dilos.cfg".
Error: (1, 'Internal error', 'panic: xc_dom_boot.c:197: xc_dom_boot_domU_map: failed to mmap domU pages 0x0+0x826 [mmap, errno=6 (No such device or address)]')

could you please let me know - how to identify by Xen sources - what i need implement/fix on my dom0 sources ? 

maybe - how to get more debug info ?
i have output from Xen to serial console and i can get statistic, but have no more information about problem with PV guest creation.

--
Best regards,
Igor Kozhukhov

Re: xen-4.3 port

[Konrad Rzeszutek Wilk]

On Tue, Jan 28, 2014 at 11:21:34PM +0400, Igor Kozhukhov wrote:
> Hello All,
> 
> i'm working on port xen-4.3 to DilOS (illumos based platform).
> 
> i have problems with PV guest load.
> dom0 started, i can see info by 'xl info'.
> 
> first: i see platform ID=38, but i couldn't found it in xen/public/platform.h
> 
> Jan 28 01:16:44 myhost privcmd: == HYPERVISOR_platform_op 38
> Jan 28 01:16:44 myhost privcmd: unrecognized HYPERVISOR_platform_op 38
> 
> could you please let me know - what is it the 38 platform hypercall ?

tmem_op
> 
> next problem: i try to create a new PV guest by 'xm create dilos.cfg'
> i see on dom0 logs requests to hypercall ID=38 - i see it is __HYPERVISOR_tmem_op

You sure it is dom0? Not the guest?

> 
> do i need implement it first ?

No. But you should have stub functions in your hypercall page to at least
return -ENOSYS for everything you don't implement.

How do you construct your hyperpage?
> 
> i have error with :
> root@myhost:/xen# xm create dilos.cfg 
> Using config file "./dilos.cfg".
> Error: (1, 'Internal error', 'panic: xc_dom_boot.c:197: xc_dom_boot_domU_map: failed to mmap domU pages 0x0+0x826 [mmap, errno=6 (No such device or address)]')
> 
> could you please let me know - how to identify by Xen sources - what i need implement/fix on my dom0 sources ? 
> 
> maybe - how to get more debug info ?
> i have output from Xen to serial console and i can get statistic, but have no more information about problem with PV guest creation.
> 
> --
> Best regards,
> Igor Kozhukhov
> 
> 
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: xen-4.3 port

[Igor Kozhukhov]

On Jan 28, 2014, at 11:34 PM, Konrad Rzeszutek Wilk wrote:

> On Tue, Jan 28, 2014 at 11:21:34PM +0400, Igor Kozhukhov wrote:
>> Hello All,
>> 
>> i'm working on port xen-4.3 to DilOS (illumos based platform).
>> 
>> i have problems with PV guest load.
>> dom0 started, i can see info by 'xl info'.
>> 
>> first: i see platform ID=38, but i couldn't found it in xen/public/platform.h
>> 
>> Jan 28 01:16:44 myhost privcmd: == HYPERVISOR_platform_op 38
>> Jan 28 01:16:44 myhost privcmd: unrecognized HYPERVISOR_platform_op 38
>> 
>> could you please let me know - what is it the 38 platform hypercall ?
> 
> tmem_op

tmem_op defined at xen/public/xen.h, but 38 ID not defined at xen/public/platform.h

>> 
>> next problem: i try to create a new PV guest by 'xm create dilos.cfg'
>> i see on dom0 logs requests to hypercall ID=38 - i see it is __HYPERVISOR_tmem_op
> 
> You sure it is dom0? Not the guest?
yes - it is in dom0 log.

> 
>> 
>> do i need implement it first ?
> 
> No. But you should have stub functions in your hypercall page to at least
> return -ENOSYS for everything you don't implement.

based on current code i see:
return -X_EINVAL;
will it be correct to return it if ID not implemented ?


> How do you construct your hyperpage?
>> 

example here : https://github.com/illumos/illumos-gate/blob/master/usr/src/uts/i86xpv/io/privcmd_hcall.c

from line: 379


>> i have error with :
>> root@myhost:/xen# xm create dilos.cfg 
>> Using config file "./dilos.cfg".
>> Error: (1, 'Internal error', 'panic: xc_dom_boot.c:197: xc_dom_boot_domU_map: failed to mmap domU pages 0x0+0x826 [mmap, errno=6 (No such device or address)]')
>> 
>> could you please let me know - how to identify by Xen sources - what i need implement/fix on my dom0 sources ? 
>> 
>> maybe - how to get more debug info ?
>> i have output from Xen to serial console and i can get statistic, but have no more information about problem with PV guest creation.
>> 
>> --
>> Best regards,
>> Igor Kozhukhov
>> 
>> 
>> 
>> 
>> 
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xen.org
>> http://lists.xen.org/xen-devel

Re: xen-4.3 port

[Ian Campbell]

On Tue, 2014-01-28 at 23:45 +0400, Igor Kozhukhov wrote:
> On Jan 28, 2014, at 11:34 PM, Konrad Rzeszutek Wilk wrote:
> 
> > On Tue, Jan 28, 2014 at 11:21:34PM +0400, Igor Kozhukhov wrote:
> >> Hello All,
> >> 
> >> i'm working on port xen-4.3 to DilOS (illumos based platform).
> >> 
> >> i have problems with PV guest load.
> >> dom0 started, i can see info by 'xl info'.
> >> 
> >> first: i see platform ID=38, but i couldn't found it in xen/public/platform.h
> >> 
> >> Jan 28 01:16:44 myhost privcmd: == HYPERVISOR_platform_op 38
> >> Jan 28 01:16:44 myhost privcmd: unrecognized HYPERVISOR_platform_op 38
> >> 
> >> could you please let me know - what is it the 38 platform hypercall ?
> > 
> > tmem_op
> 
> tmem_op defined at xen/public/xen.h, but 38 ID not defined at xen/public/platform.h

platform.h only declares one subset of hypercall, the XENPF interfaces.
tmem_op is not one of those interfaces. You want include/public/tmem.h

> > 
> >> 
> >> do i need implement it first ?
> > 
> > No. But you should have stub functions in your hypercall page to at least
> > return -ENOSYS for everything you don't implement.
> 
> based on current code i see:
> return -X_EINVAL;
> will it be correct to return it if ID not implemented ?

This appears to be an illlumos return code. It is of course up to the OS
to decide what to return from an unimplemented ioctls, but the
hypervisor itself will return -ENOSYS to an unimplemented hypercall.

> > How do you construct your hyperpage?
> >> 
> 
> example here : https://github.com/illumos/illumos-gate/blob/master/usr/src/uts/i86xpv/io/privcmd_hcall.c
> 
> from line: 379

It seems like illumos has chosen to implement the privcmd hypercall
piecemeal on a hypercall-by-hypercall basis (in fact on a subop by subop
basis). This is up to you but you might find it easier to just do as
Linux does and mirror all hypercalls made via this path through to the
hypervisor.

One downside of your approach is that you end up hardcoding
non-stable-ABIs into your kernel -- e.g. XEN_SYSCTL and XEN_DOMCTL.
These are not considered stable across Xen releases which means that you
will need to update your kernel whenever you update Xen. If you just
mirror the hypercalls through without inspection then when upgrading Xen
you only need to update the Xen tools and the hypervisor but not the
kernel.

I suppose you could also take the middle ground and only pass through
the non-stable interfaces but continue to check the rest.

Ian.

Re: xen-4.3 port

[Igor Kozhukhov]

On Jan 29, 2014, at 1:46 PM, Ian Campbell wrote:

> On Tue, 2014-01-28 at 23:45 +0400, Igor Kozhukhov wrote:
>> On Jan 28, 2014, at 11:34 PM, Konrad Rzeszutek Wilk wrote:
>> 
>>> On Tue, Jan 28, 2014 at 11:21:34PM +0400, Igor Kozhukhov wrote:
>>>> Hello All,
>>>> 
>>>> i'm working on port xen-4.3 to DilOS (illumos based platform).
>>>> 
>>>> i have problems with PV guest load.
>>>> dom0 started, i can see info by 'xl info'.
>>>> 
>>>> first: i see platform ID=38, but i couldn't found it in xen/public/platform.h
>>>> 
>>>> Jan 28 01:16:44 myhost privcmd: == HYPERVISOR_platform_op 38
>>>> Jan 28 01:16:44 myhost privcmd: unrecognized HYPERVISOR_platform_op 38
>>>> 
>>>> could you please let me know - what is it the 38 platform hypercall ?
>>> 
>>> tmem_op
>> 
>> tmem_op defined at xen/public/xen.h, but 38 ID not defined at xen/public/platform.h
> 
> platform.h only declares one subset of hypercall, the XENPF interfaces.
> tmem_op is not one of those interfaces. You want include/public/tmem.h
Am i right here - i have to add to platform.h:
#define XENPF_tmem_op 38
?
i have no found ID 38 at xen-instable at xen/include/public/platform.h

>>> 
>>>> 
>>>> do i need implement it first ?
>>> 
>>> No. But you should have stub functions in your hypercall page to at least
>>> return -ENOSYS for everything you don't implement.
>> 
>> based on current code i see:
>> return -X_EINVAL;
>> will it be correct to return it if ID not implemented ?
> 
> This appears to be an illlumos return code. It is of course up to the OS
> to decide what to return from an unimplemented ioctls, but the
> hypervisor itself will return -ENOSYS to an unimplemented hyper call.
you mean - hypervisor = dom0 ?
or hypervisor = xen.gz + xen-syms ?

> 
>>> How do you construct your hyperpage?
>>>> 
>> 
>> example here : https://github.com/illumos/illumos-gate/blob/master/usr/src/uts/i86xpv/io/privcmd_hcall.c
>> 
>> from line: 379
> 
> It seems like illumos has chosen to implement the privcmd hypercall
> piecemeal on a hypercall-by-hypercall basis (in fact on a subop by subop
> basis). This is up to you but you might find it easier to just do as
> Linux does and mirror all hypercalls made via this path through to the
> hypervisor.
> 
> One downside of your approach is that you end up hardcoding
> non-stable-ABIs into your kernel -- e.g. XEN_SYSCTL and XEN_DOMCTL.
> These are not considered stable across Xen releases which means that you
> will need to update your kernel whenever you update Xen. If you just
> mirror the hypercalls through without inspection then when upgrading Xen
> you only need to update the Xen tools and the hypervisor but not the
> kernel.
if it is possible - could you please point me take a look some files with realization on Linux ?
where it located ?
thanks for this info.

> I suppose you could also take the middle ground and only pass through
> the non-stable interfaces but continue to check the rest.
thanks - i'll take a look.

> Ian.
> 

Re: xen-4.3 port

[Ian Campbell]

On Wed, 2014-01-29 at 18:35 +0400, Igor Kozhukhov wrote:
> On Jan 29, 2014, at 1:46 PM, Ian Campbell wrote:
> 
> > On Tue, 2014-01-28 at 23:45 +0400, Igor Kozhukhov wrote:
> >> On Jan 28, 2014, at 11:34 PM, Konrad Rzeszutek Wilk wrote:
> >> 
> >>> On Tue, Jan 28, 2014 at 11:21:34PM +0400, Igor Kozhukhov wrote:
> >>>> Hello All,
> >>>> 
> >>>> i'm working on port xen-4.3 to DilOS (illumos based platform).
> >>>> 
> >>>> i have problems with PV guest load.
> >>>> dom0 started, i can see info by 'xl info'.
> >>>> 
> >>>> first: i see platform ID=38, but i couldn't found it in xen/public/platform.h
> >>>> 
> >>>> Jan 28 01:16:44 myhost privcmd: == HYPERVISOR_platform_op 38
> >>>> Jan 28 01:16:44 myhost privcmd: unrecognized HYPERVISOR_platform_op 38
> >>>> 
> >>>> could you please let me know - what is it the 38 platform hypercall ?
> >>> 
> >>> tmem_op
> >> 
> >> tmem_op defined at xen/public/xen.h, but 38 ID not defined at xen/public/platform.h
> > 
> > platform.h only declares one subset of hypercall, the XENPF interfaces.
> > tmem_op is not one of those interfaces. You want include/public/tmem.h
> Am i right here - i have to add to platform.h:
> #define XENPF_tmem_op 38
> ?
> i have no found ID 38 at xen-instable at xen/include/public/platform.h

No. tmem is not a platform op suboperation, it is a top level hypercall
in its own right.

I think Konrad & I may have misunderstood what you are seeing though: If
you are seeing hypercall 7 (== __HYPERVISOR_platform_op) with subop 38
then I'm afraid I don't know where this has come from. I can see no
reference to it in the Xen history.

I think I would be inclined as a debugging aid to inject an error (e.g a
SIGSEGV) into the calling process at this point so that it can be
trapped at the place making the call.

> 
> >>> 
> >>>> 
> >>>> do i need implement it first ?
> >>> 
> >>> No. But you should have stub functions in your hypercall page to at least
> >>> return -ENOSYS for everything you don't implement.
> >> 
> >> based on current code i see:
> >> return -X_EINVAL;
> >> will it be correct to return it if ID not implemented ?
> > 
> > This appears to be an illlumos return code. It is of course up to the OS
> > to decide what to return from an unimplemented ioctls, but the
> > hypervisor itself will return -ENOSYS to an unimplemented hyper call.
> you mean - hypervisor = dom0 ?
> or hypervisor = xen.gz + xen-syms ?

hypervisor == xen.

> >>> How do you construct your hyperpage?
> >>>> 
> >> 
> >> example here : https://github.com/illumos/illumos-gate/blob/master/usr/src/uts/i86xpv/io/privcmd_hcall.c
> >> 
> >> from line: 379
> > 
> > It seems like illumos has chosen to implement the privcmd hypercall
> > piecemeal on a hypercall-by-hypercall basis (in fact on a subop by subop
> > basis). This is up to you but you might find it easier to just do as
> > Linux does and mirror all hypercalls made via this path through to the
> > hypervisor.
> > 
> > One downside of your approach is that you end up hardcoding
> > non-stable-ABIs into your kernel -- e.g. XEN_SYSCTL and XEN_DOMCTL.
> > These are not considered stable across Xen releases which means that you
> > will need to update your kernel whenever you update Xen. If you just
> > mirror the hypercalls through without inspection then when upgrading Xen
> > you only need to update the Xen tools and the hypervisor but not the
> > kernel.
> if it is possible - could you please point me take a look some files with realization on Linux ?
> where it located ?
> thanks for this info.

drivers/xen/privcmd.c in the upstream Linux source.

Ian.

Re: xen-4.3 port

[Ian Campbell]

On Tue, 2014-01-28 at 14:34 -0500, Konrad Rzeszutek Wilk wrote:
> > do i need implement it first ?
> 
> No. But you should have stub functions in your hypercall page to at least
> return -ENOSYS for everything you don't implement.
> 
> How do you construct your hyperpage?

Note that nobody should be constructing a hypercall page themselves,
they should rely on the hypervisor to do it (for PVH guests they will
need to request it, but for a PV guest it is just there).

If a guest calls the hypercall page entry associated with an
unimplemented hypercall then the effect will be an -ENOSYS return.

Ian.