perl hypervisor interface

perl hypervisor interface

[David Becker]

Here's a start at porting the Xen admin interface to perl.
Mostly for those of us more prone to perl than python, and want to make
hypervisor calls directly from our perl scripts.
This example just does the getdomaininfo call for Domain-0. Just rinse
and repeat for the other calls (I hope :-).

One bit I left out from the libxc do_dom0_op() procedure, was calling mlock()
on the dom0_op struct before passing it into the ioctl().
When does the mlock() come into play?

David
--



#!/usr/bin/env perl

use strict;
require "sys/ioctl.ph";


# linux-2.4.26-xen-sparse/include/asm-xen/proc_cmd.h
sub SIZEOF_HYPERCALL () { 24; }
sub SIZEOF_MMAP () { 12; }
sub SIZEOF_MMAPBATCH () { 16; }

sub STRUCT_PRIVCMD_HYPERCALL () {"L P";}

sub IOCTL_PRIVCMD_HYPERCALL ()
	{ &_IOC( &_IOC_NONE, ord('P'), 0, SIZEOF_HYPERCALL );}
sub IOCTL_PRIVCMD_INITDOMAIN_EVTCHN () { &_IOC( &_IOC_NONE, ord('P'), 1, 0);}
sub IOCTL_PRIVCMD_MMAP () { &_IOC( &_IOC_NONE, ord('P'), 2, SIZEOF_MMAP );}
sub IOCTL_PRIVCMD_MMAPBATCH ()
	{ &_IOC( &_IOC_NONE, ord('P'), 2, SIZEOF_MMAPBATCH );}
# end of linux-2.4.26-xen-sparse/include/asm-xen/proc_cmd.h

# xen/include/hypervisor-ifs/hypervisor-if.h 
sub __HYPERVISOR_set_trap_table () {0;}
sub __HYPERVISOR_mmu_update () {1;}
sub __HYPERVISOR_set_gdt () {2;}
sub __HYPERVISOR_stack_switch () {3;}
sub __HYPERVISOR_set_callbacks () {4;}
sub __HYPERVISOR_fpu_taskswitch () {5;}
sub __HYPERVISOR_sched_op () {6;}
sub __HYPERVISOR_dom0_op () {7;}
sub __HYPERVISOR_set_debugreg () {8;}
sub __HYPERVISOR_get_debugreg () {9;}
sub __HYPERVISOR_update_descriptor () {10;}
sub __HYPERVISOR_set_fast_trap () {11;}
sub __HYPERVISOR_dom_mem_op () {12;}
sub __HYPERVISOR_multicall () {13;}
sub __HYPERVISOR_update_va_mapping () {14;}
sub __HYPERVISOR_set_timer_op () {15;}
sub __HYPERVISOR_event_channel_op () {16;}
sub __HYPERVISOR_xen_version () {17;}
sub __HYPERVISOR_console_io () {18;}
sub __HYPERVISOR_physdev_op () {19;}
sub __HYPERVISOR_update_va_mapping_otherdomain () {20;}
sub ARGS_PER_MULTICALL_ENTRY () {8;}
sub VIRQ_MISDIRECT () {0;}
sub VIRQ_TIMER () {1;}
sub VIRQ_DEBUG () {2;}
sub VIRQ_CONSOLE () {3;}
sub VIRQ_DOM_EXC () {4;}
sub NR_VIRQS () {5;}
sub MMU_NORMAL_PT_UPDATE () {0;}
sub MMU_MACHPHYS_UPDATE () {2;}
sub MMU_EXTENDED_COMMAND () {3;}
sub MMUEXT_PIN_L1_TABLE () {0;}
sub MMUEXT_PIN_L2_TABLE () {1;}
sub MMUEXT_PIN_L3_TABLE () {2;}
sub MMUEXT_PIN_L4_TABLE () {3;}
sub MMUEXT_UNPIN_TABLE () {4;}
sub MMUEXT_NEW_BASEPTR () {5;}
sub MMUEXT_TLB_FLUSH () {6;}
sub MMUEXT_INVLPG () {7;}
sub MMUEXT_SET_LDT () {8;}
sub MMUEXT_SET_SUBJECTDOM () {9;}
sub SET_PAGETABLE_SUBJECTDOM () {(1<<14);}
sub MMUEXT_REASSIGN_PAGE () {10;}
sub MMUEXT_RESET_SUBJECTDOM () {11;}
sub MMUEXT_CMD_MASK () {255;}
sub MMUEXT_CMD_SHIFT () {8;}
sub UVMF_FLUSH_TLB () {1;}
sub UVMF_INVLPG () {2;}
sub SCHEDOP_yield () {0;}
sub SCHEDOP_block () {1;}
sub SCHEDOP_shutdown () {2;}
sub SCHEDOP_cmdmask () {255;}
sub SCHEDOP_reasonshift () {8;}
sub CONSOLEIO_write () {0;}
sub CONSOLEIO_read () {1;}
sub MEMOP_increase_reservation () {0;}
sub MEMOP_decrease_reservation () {1;}
sub DOMID_SELF () {(0x7ffffffe);}
sub NR_EVENT_CHANNELS () {1024;}
sub MAX_VIRT_CPUS () {1;}
sub MAX_CMDLINE () {256;}
sub SIF_PRIVILEGED () {(1<<0);}
sub SIF_INITDOMAIN () {(1<<1);}
# end of xen/include/hypervisor-ifs/hypervisor-if.h 


# xen/include/hypervisor-ifs/dom0_ops.h 
sub DOM0_INTERFACE_VERSION () {0xaaaa0010;}
sub MAX_DOMAIN_NAME () {16;}
sub DOM0_GETMEMLIST () {2;}
sub DOM0_SCHEDCTL () {6;}
sub DOM0_ADJUSTDOM () {7;}
sub DOM0_CREATEDOMAIN () {8;}
sub DOM0_DESTROYDOMAIN () {9;}
sub DOM0_PAUSEDOMAIN () {10;}
sub DOM0_UNPAUSEDOMAIN () {11;}
sub DOM0_GETDOMAININFO () {12;}
sub DOMFLAGS_DYING () {(1<<0);}
sub DOMFLAGS_CRASHED () {(1<<1);}
sub DOMFLAGS_SHUTDOWN () {(1<<2);}
sub DOMFLAGS_PAUSED () {(1<<3);}
sub DOMFLAGS_BLOCKED () {(1<<4);}
sub DOMFLAGS_RUNNING () {(1<<5);}
sub DOMFLAGS_CPUMASK () {255;}
sub DOMFLAGS_CPUSHIFT () {8;}
sub DOMFLAGS_SHUTDOWNMASK () {255;}
sub DOMFLAGS_SHUTDOWNSHIFT () {16;}
sub DOM0_BUILDDOMAIN () {13;}
sub DOM0_IOPL () {14;}
sub DOM0_MSR () {15;}
sub DOM0_DEBUG () {16;}
sub DOM0_SETTIME () {17;}
sub DOM0_GETPAGEFRAMEINFO () {18;}
sub NOTAB () {0;}
sub L1TAB () {(1<<28);}
sub L2TAB () {(2<<28);}
sub L3TAB () {(3<<28);}
sub L4TAB () {(4<<28);}
sub XTAB () {(0xf<<28);}
sub LTAB_MASK () { &XTAB;}
sub DOM0_READCONSOLE () {19;}
sub DOM0_PINCPUDOMAIN () {20;}
sub DOM0_GETTBUFS () {21;}
sub DOM0_PHYSINFO () {22;}
sub DOM0_PCIDEV_ACCESS () {23;}
sub DOM0_SCHED_ID () {24;}
sub DOM0_SHADOW_CONTROL () {25;}
sub DOM0_SHADOW_CONTROL_OP_OFF () {0;}
sub DOM0_SHADOW_CONTROL_OP_ENABLE_TEST () {1;}
sub DOM0_SHADOW_CONTROL_OP_ENABLE_LOGDIRTY () {2;}
sub DOM0_SHADOW_CONTROL_OP_ENABLE_TRANSLATE () {3;}
sub DOM0_SHADOW_CONTROL_OP_FLUSH () {10;}
sub DOM0_SHADOW_CONTROL_OP_CLEAN () {11;}
sub DOM0_SHADOW_CONTROL_OP_PEEK () {12;}
sub DOM0_SHADOW_CONTROL_OP_CLEAN2 () {13;}
sub DOM0_SETDOMAINNAME () {26;}
sub DOM0_SETDOMAININITIALMEM () {27;}
sub DOM0_SETDOMAINMAXMEM () {28;}
sub DOM0_GETPAGEFRAMEINFO2 () {29;}

sub STRUCT_DOM0_OP_PREFIX () {"L L";}
sub STRUCT_GETDOMAININFO ()
    {STRUCT_DOM0_OP_PREFIX."L L Z".MAX_DOMAIN_NAME." L x4 L x4 L x4 L x4 L" ;}
# end of xen/include/hypervisor-ifs/dom0_ops.h 

#
sub XEN_PRIVCMD () {"/proc/xen/privcmd";}
#

sub getdomaininfo($) {
    my ($domain) = @_;
    my $msg = pack(STRUCT_GETDOMAININFO,DOM0_GETDOMAININFO,
        DOM0_INTERFACE_VERSION,  $domain);
    my $cmd = pack(STRUCT_PRIVCMD_HYPERCALL,__HYPERVISOR_dom0_op,$msg);
    open(XEN,XEN_PRIVCMD) or die "$!\n";
    ioctl(XEN, IOCTL_PRIVCMD_HYPERCALL, $cmd) or die "ioctl: $!";
    close XEN;

    my $ret = {};
    ($ret->{op}, $ret->{ver}, $ret->{domain}, $ret->{flags}, $ret->{name},
     $ret->{ctxt}, $ret->{tot_pages}, $ret->{max_pages},
     $ret->{shared_info_frame}, $ret->{cpu_time}) =
        unpack(STRUCT_GETDOMAININFO, $msg);
    $ret->{cpuid} = $ret->{flags}>>DOMFLAGS_CPUSHIFT&DOMFLAGS_CPUMASK;
    return $ret;
}


my $domain = 0;
my $info = getdomaininfo($domain);

printf "Dom  Name             Mem(kb)  CPU  State  Time(ms)\n";

my $flagstring = "";
foreach my $i (0..5) {
	$flagstring .= (($info->{flags}>>(5-$i)&1)?'x':'-');
	}

printf "%-4d %-".MAX_DOMAIN_NAME."s %8d %3d  $flagstring %5g\n",
     $info->{domain}, $info->{name}, $info->{tot_pages}*4,
     $info->{cpuid}, $info->{cpu_time};


-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training.
Attend Black Hat Briefings & Training, Las Vegas July 24-29 - 
digital self defense, top technical experts, no vendor pitches, 
unmatched networking opportunities. Visit www.blackhat.com

Re: perl hypervisor interface

[Keir Fraser]

> Here's a start at porting the Xen admin interface to perl.
> Mostly for those of us more prone to perl than python, and want to make
> hypervisor calls directly from our perl scripts.
> This example just does the getdomaininfo call for Domain-0. Just rinse
> and repeat for the other calls (I hope :-).

It very much depends on what functionality you want to
implement. Simple things like obtaining execution info, pausing a
domain, twiddling scheduling parameters, and so on, can easily be
handled by exporting the dom0_op interface to Python.

Harder things that the Python controller must also handle include
bootstrapping new domains --- for example, connecting them through to
device-driver backends, and handling console i/o. All this stuff needs
a daemon with more smarts than a simple control-interface wrapper.

If you're happy to leave xend doing some of the tedious work and just
want to write some extra value-adding tools, then I think your
approach is probably fine.

> One bit I left out from the libxc do_dom0_op() procedure, was calling mlock()
> on the dom0_op struct before passing it into the ioctl().
> When does the mlock() come into play?

Xen doesn't do paging, so any memory buffers you pass to the dom0_op
hypercall (including the dom0_op_t structure itself) must be resident
when Xen is entered. If the buffer is allocated in-kernel then this
isn't a problem, as Linux doesn't page itself. In user space we must
use mlock() to force the buffer to be resident.

 -- Keir


-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training.
Attend Black Hat Briefings & Training, Las Vegas July 24-29 - 
digital self defense, top technical experts, no vendor pitches, 
unmatched networking opportunities. Visit www.blackhat.com

Re: perl hypervisor interface

[Mark Williamson]

You need to mlock anything that Xen is going to access, otherwise XenLinux 
might swap it out (if you're unlucky), in which case things will break when 
Xen itself tries to access it.  I'm guessing this would result in the domain 
being killed by Xen.

HTH,
Mark



-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training.
Attend Black Hat Briefings & Training, Las Vegas July 24-29 - 
digital self defense, top technical experts, no vendor pitches, 
unmatched networking opportunities. Visit www.blackhat.com

Re: perl hypervisor interface

[Keir Fraser]

> You need to mlock anything that Xen is going to access, otherwise XenLinux 
> might swap it out (if you're unlucky), in which case things will break when 
> Xen itself tries to access it.  I'm guessing this would result in the domain 
> being killed by Xen.

The dom0_op should simply fail. However, I wasn't so careful to use
copy_from_user() on all privileged code paths, as we trust DOM0. If
you hit one of those, the machine would be toast.

 -- Keir


-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training.
Attend Black Hat Briefings & Training, Las Vegas July 24-29 - 
digital self defense, top technical experts, no vendor pitches, 
unmatched networking opportunities. Visit www.blackhat.com

Re: perl hypervisor interface

[Avery Pennarun]

On Fri, Jul 09, 2004 at 05:02:27PM +0100, Keir Fraser wrote:

> > One bit I left out from the libxc do_dom0_op() procedure, was calling mlock()
> > on the dom0_op struct before passing it into the ioctl().
> > When does the mlock() come into play?
> 
> Xen doesn't do paging, so any memory buffers you pass to the dom0_op
> hypercall (including the dom0_op_t structure itself) must be resident
> when Xen is entered. If the buffer is allocated in-kernel then this
> isn't a problem, as Linux doesn't page itself. In user space we must
> use mlock() to force the buffer to be resident.

Wouldn't it be better to have the ioctl() auto-mlock() the memory area,
rather than trusting userspace apps to know what they're doing?  In general,
calls from userspace shouldn't be able to *accidentally* crash the kernel...
at least not without asking the kernel for permission first :) 

Have fun,

Avery


-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training.
Attend Black Hat Briefings & Training, Las Vegas July 24-29 - 
digital self defense, top technical experts, no vendor pitches, 
unmatched networking opportunities. Visit www.blackhat.com

Re: perl hypervisor interface

[Keir Fraser]

> On Fri, Jul 09, 2004 at 05:02:27PM +0100, Keir Fraser wrote:
> 
> > > One bit I left out from the libxc do_dom0_op() procedure, was calling mlock()
> > > on the dom0_op struct before passing it into the ioctl().
> > > When does the mlock() come into play?
> > 
> > Xen doesn't do paging, so any memory buffers you pass to the dom0_op
> > hypercall (including the dom0_op_t structure itself) must be resident
> > when Xen is entered. If the buffer is allocated in-kernel then this
> > isn't a problem, as Linux doesn't page itself. In user space we must
> > use mlock() to force the buffer to be resident.
> 
> Wouldn't it be better to have the ioctl() auto-mlock() the memory area,
> rather than trusting userspace apps to know what they're doing?  In general,
> calls from userspace shouldn't be able to *accidentally* crash the kernel...
> at least not without asking the kernel for permission first :) 

Only applications with permission to open /dev/xen/privcmd can use the
hypercall interface; typically this is restricted to root.

Also, it's not just the dom0_op_t struct that needs to be mlock()ed --
some operations have secondary buffers that also need mlock()ing, and
adding auto support into the kernel would mean that we have yet
/another/ layer of code to maintain!

 -- Keir


-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training.
Attend Black Hat Briefings & Training, Las Vegas July 24-29 - 
digital self defense, top technical experts, no vendor pitches, 
unmatched networking opportunities. Visit www.blackhat.com

Re: perl hypervisor interface

[ron minnich]

On Fri, 9 Jul 2004, David Becker wrote:

> 
> Here's a start at porting the Xen admin interface to perl.

I'm hitting an issue with either perl or python. On cluster nodes,
whatever does the admin function has to be light (in terms of %CPU used,
memory footprint, and most importantly variance it will cause in
application performance) and very small. I don't see that happening in
either python or perl. Yes, they're easier. Yes, they do things you like
(GC, strings, etc).

But it's easy to write socket I/O stuff and such in C, and async is easy, 
and you can make it tight and small. Of course there could be some super 
tricky detail I am missing here so we shall see.

I think I'm going to look hard at what these things are doing and ...  
write them in C.

ron



-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training.
Attend Black Hat Briefings & Training, Las Vegas July 24-29 - 
digital self defense, top technical experts, no vendor pitches, 
unmatched networking opportunities. Visit www.blackhat.com