Mapping shared pages from Windows

Mapping shared pages from Windows

[Simon Martin]

[-- Attachment #1.1: Type: text/plain, Size: 1165 bytes --]

Hi all,

I have been working on my little OS port and things seem to be going 
reasonably OK. I am writing it from the ground up using David Chisnall's 
book "The Definitive Guide to the Xen Hypervisor" and referring to the 
mini-os implementation. I find there are too many dependencies and 
additional bits in the mini-os code to just pull the bits out that I 
need. Maybe once I've got a better understanding I'll be able to go back 
and do something there.

My next channel is to talk to my little OS from a Windows HVM. As a 
start I have offered a shared page from my domain which I want to map in 
a little Windows app.

I have been looking for a Windows API to access Xen, but I haven't found 
anything. I took a quick look at the GPLPV drivers but that is quite a 
large set of files. Is there an easier starting point, and maybe even a 
little documentation?

Regards.


--
  _     _  Debian GNU User   Simon Martin
| |   (_)_ __  _   ___  __  Project Manager
| |   | | '_ \| | | \ \/ /  Milliways
| |___| | | | | |_| |>  <   mailto: smartin@milliways.cl
|_____|_|_| |_|\__,_/_/\_\
Si Hoc Legere Scis Nimium Eruditionis Habes

[-- Attachment #1.2: Type: text/html, Size: 2156 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Mapping shared pages from Windows

[James Harper]

> Hi all,
> 
> I have been working on my little OS port and things seem to be going
> reasonably OK. I am writing it from the ground up using David Chisnall's book
> "The Definitive Guide to the Xen Hypervisor" and referring to the mini-os
> implementation. I find there are too many dependencies and additional bits
> in the mini-os code to just pull the bits out that I need. Maybe once I've got a
> better understanding I'll be able to go back and do something there.
> 
> My next channel is to talk to my little OS from a Windows HVM. As a start I
> have offered a shared page from my domain which I want to map in a little
> Windows app.
> 
> I have been looking for a Windows API to access Xen, but I haven't found
> anything. I took a quick look at the GPLPV drivers but that is quite a large set
> of files. Is there an easier starting point, and maybe even a little
> documentation?
> 

Under GPLPV, everything is currently done via mapping a page from Windows into Dom0 (or driver domain). Apart from a few cases at startup I don't think I map any foreign pages into Windows.

The typical design of a xen driver (eg net/disk) is that you have a shared page for the io ring, an event channel for the front and backend drivers to deliver events to each other, and then the messages on the io ring may include grant references to other pages of memory for buffer passing. I think it would be possible to map a foreign page into Windows by swapping out the pfn of the foreign page with a windows page, or maybe to map it into the windows io space designated by the pci device used for interrupt delivery (although that is a limited resource).

The other thing is that pretty much all of this happens in kernel space. Mapping a page of memory from Windows kernel to user space is something you'd probably try and avoid if possible (read ntdev archives for many, many opinions on this :) Mapping a page from userspace into xen would be better, but then your os would need to handle mapping the page from windows. I'll ask on the ntdev list about this as I'm curious now...

GPLPV has some basic xenbus functionality available from userspace (enough for the shutdownmon app to detect that dom0 has requested a shutdown/reboot/whatever), but no functionality for event channel or grant table calls. Ideally these would be implemented in GPLPV to match the ioctl's that Linux uses and so the api libraries could be ported across, but that's a bit of work and I haven't had a need to do it so far.

So... what are you going to do with this shared memory you are mapping? Does it need an associated event channel? Will it be a shared io ring with other mapped pages or will you just be passing tiny messages? If you are just passing tiny messages, and there isn't any particular performance requirements, maybe you could use xenbus to write into the xenstore and pass messages that way. A message every few seconds would be fine. 100 messages a second via xenbus might be too much.

I'm happy to fix any bugs or shortcomings you find in the GPLPV side of the xenbus code. And I can look at adding event channel and page passing (which would allow a complete device implementation in userspace), but no guarantee I will be able to get anything done quickly. And you'd either need to handle the IOCTL calls yourself, or port over the various libraries from xen (the IOCTL's are pretty raw).

James

Re: Mapping shared pages from Windows

[Simon Martin]

[-- Attachment #1.1.1: Type: text/plain, Size: 5326 bytes --]

Wow,

Thanks James. Much more information here than I expected! This mail list 
definitely rocks. My comments in line.

Regards.

>Under GPLPV, everything is currently done via mapping a page from 
>Windows into Dom0 (or driver domain). Apart from a few cases at startup 
>I don't think I map any foreign pages into Windows.
This is my use case. I will set up a static set of pages at start and 
they won't change during execution.

>The typical design of a xen driver (eg net/disk) is that you have a 
>shared page for the io ring, an event channel for the front and backend 
>drivers to deliver events to each other, and then the messages on the 
>io ring may include grant references to other pages of memory for 
>buffer passing. I think it would be possible to map a foreign page into 
>Windows by swapping out the pfn of the foreign page with a windows 
>page, or maybe to map it into the windows io space designated by the 
>pci device used for interrupt delivery (although that is a limited 
>resource).
>
>The other thing is that pretty much all of this happens in kernel 
>space. Mapping a page of memory from Windows kernel to user space is 
>something you'd probably try and avoid if possible (read ntdev archives 
>for many, many opinions on this Mapping a page from userspace into xen 
>would be better, but then your os would need to handle mapping the page 
>from windows. I'll ask on the ntdev list about this as I'm curious 
>now...
This is exactly what I would like to do. That way I avoid Windows 
user->kernel transition to access the page. I've just found your mail on 
NTDEV. It'll be interesting to see what comes back.

At the moment I envisage sharing one page as a set of circular buffers 
and event channels. If I can run this in user space then perfect, 
otherwise I will have to write it as a kernel driver. I've written a 
couple of Windows drivers (one USB and one PCI driver) in the past, but 
userspace is much easier to debug.
>
>GPLPV has some basic xenbus functionality available from userspace 
>(enough for the shutdownmon app to detect that dom0 has requested a 
>shutdown/reboot/whatever), but no functionality for event channel or 
>grant table calls. Ideally these would be implemented in GPLPV to match 
>the ioctl's that Linux uses and so the api libraries could be ported 
>across, but that's a bit of work and I haven't had a need to do it so 
>far.
I don't know enough about xenbus yet to say anything about this. It's a 
steep learning curve. Documentation seems to be 99% oriented to 
administering Xen instances, I can find very little about Xen 
development. Is that the situation or am I missing something?
>So... what are you going to do with this shared memory you are mapping? 
>Does it need an associated event channel? Will it be a shared io ring 
>with other mapped pages or will you just be passing tiny messages? If 
>you are just passing tiny messages, and there isn't any particular 
>performance requirements, maybe you could use xenbus to write into the 
>xenstore and pass messages that way. A message every few seconds would 
>be fine. 100 messages a second via xenbus might be too much.
I am porting a real-time embedded OS that I originally developed on a 
Texas Instruments TMS320C3x DSP on proprietary hardware. This was then 
ported to MIPS64 (again on proprietary hardware) and now finally it's 
running on an ARM9. I have already ported this to a Windows user space 
application and run it as a simulator. Of course timing is all shot and 
you can't use it to control any real time processes, but it suffices to 
be able to program on the hoof without real hardware.

The current project will run this simulator on a dedicated processing 
core using the arinc653 scheduler (as recommended by Andrew Cooper) and 
making it real time again. I will provide two interfaces to Windows user 
space applications, one that emulates the current TCP/Serial command 
line interface we already have, and another that will provide low level 
hooks into the OS functionality so we can provide tight integration 
between Windows applications and the OS.

I will use a shared memory block because I can end up pushing down tons 
of data and the access is totally asynchronous and involves no other 
processes.
>
>I'm happy to fix any bugs or shortcomings you find in the GPLPV side of 
>the xenbus code. And I can look at adding event channel and page 
>passing (which would allow a complete device implementation in 
>userspace), but no guarantee I will be able to get anything done 
>quickly. And you'd either need to handle the IOCTL calls yourself, or 
>port over the various libraries from xen (the IOCTL's are pretty raw).
Thanks. Looks like GPLPV is the place to start. I already installed the 
drives into the Windows 7 HVM, and they work a dream. I'll pull the 
source code and start looking into that.

At the moment my development is in the initial stages of learning the 
Xen infrastructure and how to push things around. I expect to be another 
couple of weeks experimenting with ways of doing things before I decide 
how this should be done. Like for instance setting up a CPU pool with 
the arinc653 scheduler seems to crash the hypervisor.
>
>James
Simon.

[-- Attachment #1.1.2: Type: text/html, Size: 8754 bytes --]

[-- Attachment #1.2: Type: image/png, Size: 685 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Mapping shared pages from Windows

[Nate Studer]

On 10/27/2013 6:43 PM, Simon Martin wrote:
> I am porting a real-time embedded OS that I originally developed on a Texas
> Instruments TMS320C3x DSP on proprietary hardware. This was then ported to
> MIPS64 (again on proprietary hardware) and now finally it's running on an ARM9.
> I have already ported this to a Windows user space application and run it as a
> simulator. Of course timing is all shot and you can't use it to control any real
> time processes, but it suffices to be able to program on the hoof without real
> hardware.

Is your simulator going to be running on top of Windows?

>  
> The current project will run this simulator on a dedicated processing core using
> the arinc653 scheduler (as recommended by Andrew Cooper) and making it real time
> again. 

The arinc653 scheduler is a fixed periodic timeslice scheduler, which may not be
what you want.

Can you achieve your goal with just cpu-pinning or cpu-pools and the credit or
sedf schedulers?

>  
> At the moment my development is in the initial stages of learning the Xen
> infrastructure and how to push things around. I expect to be another couple of
> weeks experimenting with ways of doing things before I decide how this should be
> done. Like for instance setting up a CPU pool with the arinc653 scheduler seems
> to crash the hypervisor.

The arinc653 scheduler currently only supports a single physical core and does
not really support CPU pools.  It would be nice if it detected these conditions
and printed an error though, instead of simply crashing...

   Nate

>>  
>> James 
> Simon.
>  
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
> 

Re: Mapping shared pages from Windows

[Nate Studer]

> At the moment my development is in the initial stages of learning the Xen
> infrastructure and how to push things around. I expect to be another couple of
> weeks experimenting with ways of doing things before I decide how this should be
> done. Like for instance setting up a CPU pool with the arinc653 scheduler seems
> to crash the hypervisor.

I just tried to repeat your issue and I get a crash in the credit scheduler!?

(XEN) Xen BUG at sched_credit.c:917
(XEN) ----[ Xen-4.4-unstable  x86_64  debug=y  Tainted:    C ]----
(XEN) CPU:    0
(XEN) RIP:    e008:[<ffff82d08011a30d>] csched_free_vdata+0x9/0x15
(XEN) RFLAGS: 0000000000010202   CONTEXT: hypervisor

If I create a pool with any scheduler other than arinc653, I do not see this
crash, so something is definitely not right.

I am pretty sure that this line in the arinc653 scheduler is the culprit.

    if ( (vc->sched_priv = xmalloc(arinc653_vcpu_t)) == NULL )

>>  
>> James 
> Simon.
>  
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
> 

Re: Mapping shared pages from Windows

[Paul Durrant]

De-html-ing...
---
From: xen-devel-bounces@lists.xen.org [mailto:xen-devel-bounces@lists.xen.org] On Behalf Of Simon Martin
Sent: 25 October 2013 20:35
To: xen-devel@lists.xen.org
Subject: [Xen-devel] Mapping shared pages from Windows

Hi all,
 
I have been working on my little OS port and things seem to be going reasonably OK. I am writing it from the ground up using David Chisnall's book "The Definitive Guide to the Xen Hypervisor" and referring to the mini-os implementation. I find there are too many dependencies and additional bits in the mini-os code to just pull the bits out that I need. Maybe once I've got a better understanding I'll be able to go back and do something there.
 
My next channel is to talk to my little OS from a Windows HVM. As a start I have offered a shared page from my domain which I want to map in a little Windows app.
 
I have been looking for a Windows API to access Xen, but I haven't found anything. I took a quick look at the GPLPV drivers but that is quite a large set of files. Is there an easier starting point, and maybe even a little documentation?
 ---

You can look at the XenServer PV drivers on GitHub. Xenbus would be the place to start (https://github.com/pauldu/win-xenbus) and look at the 'upstream' branch. There are several header files in the include subdir there which provide kernel APIs to Xen interfaces such as store, event channels, grant table, etc.
There is one user space API: a WMI provider for xenstore implemented by the xeniface driver (in the win-xeniface repo) but that's not going to be a huge amount of use to you.
I suspect what you'll need to do is add support for grant mapping into one of the sets of Windows PV drivers, probably by placing in the platform device's memory aperture, and then mapping that page out to user space. You'd also probably need an indefinitely blocking IOCTL or something that that will tell you when the user process dies though, otherwise you won't know when to do the grant unmap.

  Paul

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel