Porting an OS to Xen

Porting an OS to Xen

[Simon Martin]

[-- Attachment #1.1.1: Type: text/plain, Size: 2343 bytes --]

Hi all,

I originally posted this on the xenproject Q&A forum, and it was 
suggested that I repost it here. If it would be better posted elsewhere 
then please let me know. Please skip the next paragraph if you're not 
interested in the why.

Over the last 2 decades I have been developing a real-time OS with one 
of my customers (I was an employee when we started, a consultant for the 
last 15 years). This runs on our own hardware (originally TI TMS320C3x, 
then MIPS64, then ARM9). The system started off 80% C 20% Assembly, and 
is now about 99% C and 1% Assembly (just the bootstrapping). We now have 
a requirement to run this OS in parallel with Windows. I have looked at 
various real-time hypervisors and Windows extensions, however I think we 
can get more mileage out of Xen due to the open source nature of the 
project.

I bought the book "The Definitive Guide to the Xen Hypervisor" as it was 
recommended by the xenproject Q&A forum and it is very interesting, 
however the examples are 32 bit and I am working 64 bit, and let's say 
that my x86 assembler is a bit rusty to say the least!

My questions (I am running Xen 4.1.4 on Debian Wheezy):

1.- I have looked at MirageOS but I think it is overkill for what I want 
to do. I just want a basic C wrapper to allow Xen to load my software, I 
would prefer not to have to learn OCaml, but if this is what it takes, 
then I'll have to do it.

2.- Is there any sample simple x86_64 code that I can work from to get 
out of the starting blocks?

3.- xl complains about my image not being a bzImage. Is there anyway I 
can avoid this error/warning? If not how do I convert my image to a 
bzImage? I'd rather not have to port in half the Linux bootstrap.

4.- Despite xl complaining about the kernel not being in bzImage format 
it still seems to run (I get the same error message from the MirageOS 
but the image loads and runs OK). However it crashes and I have no way 
of seeing what's going on. Is there any way to debug my guest startup 
inside Xen?

TIA and Regards.

--
  _     _  Debian GNU User   Simon Martin
| |   (_)_ __  _   ___  __  Project Manager
| |   | | '_ \| | | \ \/ /  Milliways
| |___| | | | | |_| |>  <   mailto: smartin@milliways.cl
|_____|_|_| |_|\__,_/_/\_\
Si Hoc Legere Scis Nimium Eruditionis Habes

[-- Attachment #1.1.2: Type: text/html, Size: 3725 bytes --]

[-- Attachment #1.2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 2136 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Porting an OS to Xen

[Andrew Cooper]

[-- Attachment #1.1: Type: text/plain, Size: 3770 bytes --]

On 13/10/2013 21:28, Simon Martin wrote:
> Hi all,
>  
> I originally posted this on the xenproject Q&A forum, and it was
> suggested that I repost it here. If it would be better posted
> elsewhere then please let me know. Please skip the next paragraph if
> you're not interested in the why.

I would say that you have found the correct place to be asking questions
like this.

>  
> Over the last 2 decades I have been developing a real-time OS with one
> of my customers (I was an employee when we started, a consultant for
> the last 15 years). This runs on our own hardware (originally TI
> TMS320C3x, then MIPS64, then ARM9). The system started off 80% C 20%
> Assembly, and is now about 99% C and 1% Assembly (just the
> bootstrapping). We now have a requirement to run this OS in parallel
> with Windows. I have looked at various real-time hypervisors and
> Windows extensions, however I think we can get more mileage out of Xen
> due to the open source nature of the project.
>  
> I bought the book "The Definitive Guide to the Xen Hypervisor" as it
> was recommended by the xenproject Q&A forum and it is very
> interesting, however the examples are 32 bit and I am working 64 bit,
> and let's say that my x86 assembler is a bit rusty to say the least!
>  
> My questions (I am running Xen 4.1.4 on Debian Wheezy):
>  
> 1.- I have looked at MirageOS but I think it is overkill for what I
> want to do. I just want a basic C wrapper to allow Xen to load my
> software, I would prefer not to have to learn OCaml, but if this is
> what it takes, then I'll have to do it.
>  
> 2.- Is there any sample simple x86_64 code that I can work from to get
> out of the starting blocks?

Yes.  I am not familiar with what is distributed with Wheezy; you are
probably better checking out from the git tree (
http://xenbits.xen.org/gitweb/?p=xen.git;a=summary ).  In extras/MiniOS,
you will find MiniOS which is a minimal example of getting code working
as a paravirtualised guest.  It should certainly be a good starting
point and reference.

>  
> 3.- xl complains about my image not being a bzImage. Is there anyway I
> can avoid this error/warning? If not how do I convert my image to a
> bzImage? I'd rather not have to port in half the Linux bootstrap.

What format are you trying to boot?  It should be happy booting any
format that a Linux kernel might be, which would typically be an Elf
multiboot image with some form of compression.

>  
> 4.- Despite xl complaining about the kernel not being in bzImage
> format it still seems to run (I get the same error message from the
> MirageOS but the image loads and runs OK). However it crashes and I
> have no way of seeing what's going on. Is there any way to debug my
> guest startup inside Xen?

There are several options depending on your preference.

on_crash="pause" in your domain configuration file will leave the VM
state to be inspected.  "coredump-destroy" and "coredump-restart" are
also alternatives.

xl dump-core <domid> <file> should create an Elf CORE file representing
the domain.

The gdbsx set of tools (tools/debugger/gdbsx) allow you to attach gdb to
a running Xen domain.

Getting a PV console working (see the early code from MiniOS) would
allow you to print inforation.

Failing a PV console, if you build yourself a debug version of Xen, you
can use the console_io hypercall from your guest and get debugging
messages into the Xen console.

Hopefully that should be enough to get started with debugging.

One final thing which come to mind given your request which you might or
might not be aware of:  Xen has "arinc653", a maintained realtime
scheduler which will likely be more appropriate to your requirements
than the default "credit" scheduler.

Hope this is somewhat helpful

~Andrew

[-- Attachment #1.2: Type: text/html, Size: 6072 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Porting an OS to Xen

[Simon Martin]

[-- Attachment #1.1.1: Type: text/plain, Size: 3115 bytes --]

Hi Andrew

Thank you very much for your response. Comments in line.

Regards.

------ Original Message ------
From: "Andrew Cooper" <andrew.cooper3@citrix.com>
To: "Simon Martin" <smartin@milliways.cl>; xen-devel@lists.xen.org
Sent: 13/10/2013 20:07:25
Subject: Re: [Xen-devel] Porting an OS to Xen
>On 13/10/2013 21:28, Simon Martin wrote:
>>2.- Is there any sample simple x86_64 code that I can work from to get 
>>out of the starting blocks?
>
>Yes.  I am not familiar with what is distributed with Wheezy; you are 
>probably better checking out from the git tree ( 
>http://xenbits.xen.org/gitweb/?p=xen.git;a=summary ).  In 
>extras/MiniOS, you will find MiniOS which is a minimal example of 
>getting code working as a paravirtualised guest.  It should certainly 
>be a good starting point and reference.
Got it. Merging it with the 32 bit example code I have. Slowly getting 
there!
>
>>
>>3.- xl complains about my image not being a bzImage. Is there anyway I 
>>can avoid this error/warning? If not how do I convert my image to a 
>>bzImage? I'd rather not have to port in half the Linux bootstrap.
>
>What format are you trying to boot?  It should be happy booting any 
>format that a Linux kernel might be, which would typically be an Elf 
>multiboot image with some form of compression.
I am generating an ELF-64 binary. I have a feeling that this is a 
"normal error" when starting a non-bzImage kernel. The mini-os example 
causes the same error, so I will ignore it.
>
>>
>>4.- Despite xl complaining about the kernel not being in bzImage 
>>format it still seems to run (I get the same error message from the 
>>MirageOS but the image loads and runs OK). However it crashes and I 
>>have no way of seeing what's going on. Is there any way to debug my 
>>guest startup inside Xen?
>
>There are several options depending on your preference.
>
>on_crash="pause" in your domain configuration file will leave the VM 
>state to be inspected.  "coredump-destroy" and "coredump-restart" are 
>also alternatives.
>
>xl dump-core <domid> <file> should create an Elf CORE file representing 
>the domain.
>
>The gdbsx set of tools (tools/debugger/gdbsx) allow you to attach gdb 
>to a running Xen domain.
>
>Getting a PV console working (see the early code from MiniOS) would 
>allow you to print inforation.
>
>Failing a PV console, if you build yourself a debug version of Xen, you 
>can use the console_io hypercall from your guest and get debugging 
>messages into the Xen console.
>
>Hopefully that should be enough to get started with debugging.
Perfect.
>
>One final thing which come to mind given your request which you might 
>or might not be aware of:  Xen has "arinc653", a maintained realtime 
>scheduler which will likely be more appropriate to your requirements 
>than the default "credit" scheduler.
For simplicity's sake at the moment I propose to run my VM as the only 
VM on a vcpu. My assumption is that this way there will be no 
scheduling. Is this assumption correct?
>
>
>Hope this is somewhat helpful
>
>~Andrew

[-- Attachment #1.1.2: Type: text/html, Size: 5701 bytes --]

[-- Attachment #1.2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 2136 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Porting an OS to Xen

[Ian Campbell]

On Mon, 2013-10-14 at 16:22 +0000, Simon Martin wrote:

> > > 3.- xl complains about my image not being a bzImage. Is there
> > > anyway I can avoid this error/warning? If not how do I convert my
> > > image to a bzImage? I'd rather not have to port in half the Linux
> > > bootstrap.
> > 
> > What format are you trying to boot?  It should be happy booting any
> > format that a Linux kernel might be, which would typically be an Elf
> > multiboot image with some form of compression.
> I am generating an ELF-64 binary. I have a feeling that this is a
> "normal error" when starting a non-bzImage kernel. The mini-os example
> causes the same error, so I will ignore it.

I think the error is just during probing where it checks to see if it is
a bzImage and falls through to trying ELF if not. The message could
probably be made less error-ry.

Ian.

Re: Porting an OS to Xen

[Andrew Cooper]

[-- Attachment #1.1: Type: text/plain, Size: 712 bytes --]

On 14/10/13 17:22, Simon Martin wrote:
>>
>> One final thing which come to mind given your request which you might
>> or might not be aware of:  Xen has "arinc653", a maintained realtime
>> scheduler which will likely be more appropriate to your requirements
>> than the default "credit" scheduler.
> For simplicity's sake at the moment I propose to run my VM as the
> only VM on a vcpu. My assumption is that this way there will be no
> scheduling. Is this assumption correct?

Your best option would be to create a cpupool for the individual cpu you
want your VM to be running on.  This way you can schedule that cpu using
arinc653 independently of dom0 and domU being scheduled using regular
credit.

~Andrew

[-- Attachment #1.2: Type: text/html, Size: 2176 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel