RE: Xen at scale

RE: Xen at scale

[MAGENHEIMER,DAN (HP-FtCollins,ex1)]

[-- Attachment #1: Type: text/plain, Size: 1645 bytes --]

>> My wish list based on my experience so far:
>>        HIGHMEM4G - This is my top wish, and I will work on it myself
>>             when I get a chance, though I can't say when that
>> will be.
>
>Agreed. It should just be a case of putting the CONFIG_HIGHMEM4G
>stuff back into the following files:
>
>./include/asm-xeno/fixmap.h
>./include/asm-xeno/highmem.h
>./include/asm-xeno/pgtable.h
>./include/asm-xeno/page.h
>./arch/xeno/config.in
>./arch/xeno/mm/init.c
>./arch/xeno/kernel/setup.c

Here's a patch that adds back the CONFIG_HIGHMEM stuff.  It works
fine with CONFIG_HIGHMEM off and with CONFIG_HIGHMEM on with
dom0_memory <= 512MB, but fails with dom0_memory == 1GB (1048576)
on my 2GB machine.

The failure condition is that DOM0 barely launches, then causes
Xen to die and the machine reboots (even with reboot disabled, so
it must be trashing something in Xen itself).  I haven't been able
to get any debugging output (even panics) to show up from dom0 at all...
perhaps this will be a good opportunity to try out Alex's pdb
stuff.  (Alex, is there any way to set a breakpoint very early
in xenolinux startup?)

Anyway, I thought I'd post it in case someone with more xenolinux
setup/init debugging experience can easily spot/fix the problem,
but I'll keep trying.  (Please email if you find/fix it, to put
me out of my misery ;-)

As it works for everything that works today, the patch could be rolled
into the mainline, possibly with a runtime check and nicer error
message added.

Dan Magenheimer
HP Labs Fort Collins

P.S. No changes appeared to be required in page.h or setup.c
and I changed defconfig instead of config.in.


[-- Attachment #2: highmem.gz --]
[-- Type: application/octet-stream, Size: 2908 bytes --]

Re: Xen at scale

[Alex Ho]

can you call pdb-stub.c::breakpoint() early on during the domain startup
to force it into the debugger?   i'm planning to check in some code later
this week which will make debugging domains and processes a lot easier.

.a

On Tue, 23 Mar 2004 14:37:20 -0800
"MAGENHEIMER,DAN (HP-FtCollins,ex1)" <dan.magenheimer@hp.com> wrote:

> >> My wish list based on my experience so far:
> >>        HIGHMEM4G - This is my top wish, and I will work on it myself
> >>             when I get a chance, though I can't say when that
> >> will be.
> >
> >Agreed. It should just be a case of putting the CONFIG_HIGHMEM4G
> >stuff back into the following files:
> >
> >./include/asm-xeno/fixmap.h
> >./include/asm-xeno/highmem.h
> >./include/asm-xeno/pgtable.h
> >./include/asm-xeno/page.h
> >./arch/xeno/config.in
> >./arch/xeno/mm/init.c
> >./arch/xeno/kernel/setup.c
> 
> Here's a patch that adds back the CONFIG_HIGHMEM stuff.  It works
> fine with CONFIG_HIGHMEM off and with CONFIG_HIGHMEM on with
> dom0_memory <= 512MB, but fails with dom0_memory == 1GB (1048576)
> on my 2GB machine.
> 
> The failure condition is that DOM0 barely launches, then causes
> Xen to die and the machine reboots (even with reboot disabled, so
> it must be trashing something in Xen itself).  I haven't been able
> to get any debugging output (even panics) to show up from dom0 at all...
> perhaps this will be a good opportunity to try out Alex's pdb
> stuff.  (Alex, is there any way to set a breakpoint very early
> in xenolinux startup?)
> 
> Anyway, I thought I'd post it in case someone with more xenolinux
> setup/init debugging experience can easily spot/fix the problem,
> but I'll keep trying.  (Please email if you find/fix it, to put
> me out of my misery ;-)
> 
> As it works for everything that works today, the patch could be rolled
> into the mainline, possibly with a runtime check and nicer error
> message added.
> 
> Dan Magenheimer
> HP Labs Fort Collins
> 
> P.S. No changes appeared to be required in page.h or setup.c
> and I changed defconfig instead of config.in.
> 
> 


-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click

Re: Xen at scale

[Ian Pratt]

Dan,

I haven't had a chance to look at your patch, but you'll get
earlier console output if you hack kernel/printk.c to call
HYPERVISOR_console_io directly in the printk function, e.g.
add the following after "va_end(args);"  :

 HYPERVISOR_console_io(CONSOLEIO_write, sizeof(printk_buf), printk_buf );

That should at least indicate where abouts the failure is
happening. It might be easier to debug it as a domain other than
0 (non privileged) as it shouldn't be able to upset Xen.

Cheers,
Ian


-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click

Re: Xen at scale

[Christian Limpach]

> Here's a patch that adds back the CONFIG_HIGHMEM stuff.  It works
> fine with CONFIG_HIGHMEM off and with CONFIG_HIGHMEM on with
> dom0_memory <= 512MB, but fails with dom0_memory == 1GB (1048576)
> on my 2GB machine.

I think you'll need support from Xen for CONFIG_HIGHMEM:  the domain builder
maps all the domain's memory starting at 0xc0000000 so there's only 1GB
(less what's reserved for the Xen) available.  This is also how Linux maps
its memory and gets very simple kva->pa translation but is limited to ~1GB
of physical kernel memory (unless there's some trick hidden somewhere, I
don't know the Linux kernel that well (yet ;-)) which is also the reason for
CONFIG_HIGHMEM in the first place (again, as far as I can tell).

My guess is that one of the DOM0 operations fails in the builder when you
request a too large domain or that page table initialization fails in the
builder.

Now it would seem easy to just not map the memory which will end up in
highmem but we need to have all the memory mapped initially so that we can
build the phys_to_machine_mapping array.  I believe you can't build it only
from the machine_to_phys_mapping array provided by Xen since you don't know
which pages belong to the domain...

I think I would map the highmem pages starting at VA 0, the alternative
seems to be to pass an array with the page numbers to the domain.  I prefer
mapping the pages over using an array because we're already using the
pagetables to pass this kind of information and using an array introduces a
2nd interface between the domain's kernel and the domain builder.  Maybe
there's a simpler solution?

     christian



-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click

Re: Xen at scale

[Kip Macy]

>
> Now it would seem easy to just not map the memory which will end up in
> highmem but we need to have all the memory mapped initially so that we can
> build the phys_to_machine_mapping array.  I believe you can't build it only
> from the machine_to_phys_mapping array provided by Xen since you don't know
> which pages belong to the domain...

The {DragonFly/Free}BSD domain builder sets the page tables up a little
differently. It populates the phys_to_machine array before starting
the domain - mapping it at a pre-determined address relative to
KERNBASE. In addition it only maps the ktext and a stack page - the
remainder of the pages are mapped at runtime by the same pmap_init used
on native i386.

> I think I would map the highmem pages starting at VA 0, the alternative
> seems to be to pass an array with the page numbers to the domain.

This is done for all pages with a pre-established P->M mapping. The
domain builder passes in the number of pages used by the domain builder
via a register. This is equivalent to the way native locore.s passes
physfree (the address of the first free physical page) to init386.

> Maybe there's a simpler solution?

I don't think passing an array is more complicated.

		-Kip


-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click

Re: Xen at scale

[Christian Limpach]

> The {DragonFly/Free}BSD domain builder sets the page tables up a little
> differently. It populates the phys_to_machine array before starting
> the domain - mapping it at a pre-determined address relative to
> KERNBASE.

How is it pre-determined?

And where do you put it physically?  I would probably put it at the very end
of the physical address space since the guest is less likely to rebuild the
mapping table than it is likely to rebuild the pagetables.  Every other
place seems to require special handling in the guests memory management...

Having a domain layout which isn't compatible with the Linux one is a
problem if you want to boot your domain as domain0 since the builder for
domain0 is inside Xen.

> > Maybe there's a simpler solution?
>
> I don't think passing an array is more complicated.

It's more complicated because you need to put it somewhere and then most
guests have to possibly move it around because they don't like where it was
put by the builder.  But most of all I think that it's a bad idea to use two
different interfaces to pass the same kind of data and that we should try to
keep the amount of pre-initialized data to a minimum.

It might still be time to switch to using an array but the sooner the better
and then we should stick with whatever we choose...

     christian



-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click

Re: Xen at scale

[Keir Fraser]

> > > Maybe there's a simpler solution?
> >
> > I don't think passing an array is more complicated.
> 
> It's more complicated because you need to put it somewhere and then most
> guests have to possibly move it around because they don't like where it was
> put by the builder.  But most of all I think that it's a bad idea to use two
> different interfaces to pass the same kind of data and that we should try to
> keep the amount of pre-initialized data to a minimum.
> 
> It might still be time to switch to using an array but the sooner the better
> and then we should stick with whatever we choose...

It needs to be something other than page tables, really. I'm inclined
to pass the new domain a preinitialised 'phys->machine' translation
table, and only create initial page tables large enough to contain the
kernel image, initial page tables, and the translation table. 16MB of
VA space would be plenty.

At that point the interface is basically "We're giving you just enough
MMU state to bootstrap yourself. Here's a list of the frames that
belong to you -- set up the pagetable structure that you'd like."

Yes, the guest may possibly end up copying the 'phys->mach' table to a
more convenient place. But at worst it will be a few megabytes.

As for breaking the interface to other types of guest OS
(ie. non-Linux) -- they can implement their own domain builder that
sets memory out just as they like.

I'm rewriting the event-callback interface at the moment, but I'll
take a look at MMU bootstrap once that's done (hopefully later
today).

 -- Keir


-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click

Re: Xen at scale

[Christian Limpach]

> It needs to be something other than page tables, really. I'm inclined
> to pass the new domain a preinitialised 'phys->machine' translation
> table, and only create initial page tables large enough to contain the
> kernel image, initial page tables, and the translation table. 16MB of
> VA space would be plenty.

VA should probably have the size of text+data+bss.  My preferred layout
would be pagetables/pgd/p2m-mapping at the end of the domain's physical
memory in that order.  And this should then be either mapped at VA 0, end
aligned with text or end aligned with the hypervisor memory.  I think I'd
choose end aligned with the hypervisor memory.

> As for breaking the interface to other types of guest OS
> (ie. non-Linux) -- they can implement their own domain builder that
> sets memory out just as they like.

How's that supposed to work for non-Linux domain0 guests?

     christian



-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click

Re: Xen at scale

[Kip Macy]

On FreeBSD i386, there is already a 4MB chunk of VA where the page
tables are mapped - remember that is why the PTD maps itself (as the comment
puts it, recursively). This is the default layout
|  USER         |PTs| KVA     |
  (3GB - 4M)     4M   1GB

I've allocated an extra page-table to the PTOM - for the following VA:

| USER        |PTOM|PTs|  KVA|
  3GB - 8M      4M   4M  1GB

This works well, because all I end up having to do is bump down
VM_MAXUSER_ADDRESS by an extra 4MB. Putting it in the kernel would
disrupt pmap bootstrap. This burns some VA for small memory allocations,
but the same can be said for the PTs and it makes easy for me to put the
PTOM on a single super-page for large memory allocations.

As for DOM0, different guests can have their own setup routines inside Xen.


			-Kip

On Wed, 24 Mar 2004, Christian Limpach wrote:

> > It needs to be something other than page tables, really. I'm inclined
> > to pass the new domain a preinitialised 'phys->machine' translation
> > table, and only create initial page tables large enough to contain the
> > kernel image, initial page tables, and the translation table. 16MB of
> > VA space would be plenty.
>
> VA should probably have the size of text+data+bss.  My preferred layout
> would be pagetables/pgd/p2m-mapping at the end of the domain's physical
> memory in that order.  And this should then be either mapped at VA 0, end
> aligned with text or end aligned with the hypervisor memory.  I think I'd
> choose end aligned with the hypervisor memory.
>
> > As for breaking the interface to other types of guest OS
> > (ie. non-Linux) -- they can implement their own domain builder that
> > sets memory out just as they like.
>
> How's that supposed to work for non-Linux domain0 guests?
>
>      christian
>
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by: IBM Linux Tutorials
> Free Linux tutorial presented by Daniel Robbins, President and CEO of
> GenToo technologies. Learn everything from fundamentals to system
> administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel
>


-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click

Re: Xen at scale

[Christian Limpach]

> I've allocated an extra page-table to the PTOM - for the following VA:
> | USER        |PTOM|PTs|  KVA|
>   3GB - 8M      4M   4M  1GB

and where does the builder put the PTD?  and what's the physical memory
layout?

> As for DOM0, different guests can have their own setup routines inside
Xen.

I think we'll have to agree to disagree on this.

    christian



-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click

Re: Xen at scale

[Kip Macy]

> and where does the builder put the PTD?  and what's the physical memory
> layout?

For BSD both are irrelevant - the PTD is passed in as an argument via a
register. Only Linux relies on physical contiguity for the PTOV trick.
BSD does a table lookup in the mapped page tables - it has no physical
contiguity requirements.

>
> > As for DOM0, different guests can have their own setup routines inside
> Xen.
>
> I think we'll have to agree to disagree on this.

That is a starting point.


			-Kip


-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click

Re: Xen at scale

[Christian Limpach]

> > and where does the builder put the PTD?  and what's the physical memory
> > layout?
>
> For BSD both are irrelevant - the PTD is passed in as an argument via a
> register. Only Linux relies on physical contiguity for the PTOV trick.
> BSD does a table lookup in the mapped page tables - it has no physical
> contiguity requirements.

The domain builder still has to put the PTD somewhere and your
drawing/explanation doesn't say where.  Same goes for physical memory
layout, you have to pick one, I'd be interested to know which you picked...

    christian



-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click

Re: Xen at scale

[Kip Macy]

On Wed, 24 Mar 2004, Christian Limpach wrote:

> > > and where does the builder put the PTD?  and what's the physical memory
> > > layout?
> >
> > For BSD both are irrelevant - the PTD is passed in as an argument via a
> > register. Only Linux relies on physical contiguity for the PTOV trick.
> > BSD does a table lookup in the mapped page tables - it has no physical
> > contiguity requirements.
>
> The domain builder still has to put the PTD somewhere and your
> drawing/explanation doesn't say where.  Same goes for physical memory
> layout, you have to pick one, I'd be interested to know which you picked...

The PTD is an arbitrary physical address mapped in right after the pages
uses by the elf-loader. BSD's VM system is sufficiently general that it
doesn't care about physical layout. Nonetheless, the physical layout
ends up looking something like:

|ktext|KPTphys|IdlePTD|ptomptpa|tmpstk|startinfopa|ptompages| <inited by kernel>

I hadn't intended to commit the domain builder until the ports had been
committed to the BSDs CVS repositories - which is going to wait until
I've had time to write the block and bi-directional console drivers.
However, if it is still unclear I can post the domain builder source
itself.



			-Kip


-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click

xen-1.2 again

[I RATTAN]

I am not having any luck with xen-1.2 install from CD.

1. I tried it on Vector-linux-4.0 by copying xen.gz, xenolinux.gz
and initrd.gz to /boot, made a grub floppy to boot it as:

title xen/xenolinux
	root (hd0,0)
	kernel /boot/xen.gz dom0_mem=131072 noht nosmp noreboot
	module /boot/xenolinux.gz root=/dev/hda1 ro console=tty0
	module /boot/initrd.gz

system boots but has problems with /dev entries for fd0, cdrom etc
(same thing that I see under Mandrake-9.1), keeps on saying that
it not a block special device (ls -l /dev/fd0 does the entry with
b in forst column).

So I decided to install Fedora-1 core (minimali install) and tried the
same thing as above and now the boot process keeps on trying to
find and mount CD-ROM (trying hda, hdb, hdc, hdd..) quits to shell
prompt. Looks like it is beacause fo xenolinux.gz (same is used to
boot from cdrom..).

Can I get a working (not for cd-boot) xenolinux.gz: minimal config
of IDE disk and cdrom (is fine by me)?

There was mention of making xen work with Debian? What is involved?
I may have time on my hands after first week of May..

-ishwar




-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click

Re: xen-1.2 again

[Ian Pratt]

> system boots but has problems with /dev entries for fd0, cdrom etc
> (same thing that I see under Mandrake-9.1), keeps on saying that
> it not a block special device (ls -l /dev/fd0 does the entry with
> b in forst column).

Xen currently doesn't support floppy drives -- I didn't know
anyone still uses them! This will be rather easier to fix in the
new IO world (1.4).

CDROMs should work, but without modifying devfs you'll probably
need to access it via its hardware device name e.g. /dev/hdc or
similar.

Best,
Ian


-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click

Re: xen-1.2 again

[I RATTAN]

On Wed, 24 Mar 2004, Ian Pratt wrote:

> Xen currently doesn't support floppy drives -- I didn't know
That is ok too.

> anyone still uses them! This will be rather easier to fix in the
> new IO world (1.4).
>
> CDROMs should work, but without modifying devfs you'll probably
> need to access it via its hardware device name e.g. /dev/hdc or
> similar.

That is what I tried

mount -t iso9660 /dev/hdc /mnt/cdrom
/dev/hdc is not a block special device..

-ishwar


-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click

RE: Xen at scale

[MAGENHEIMER,DAN (HP-FtCollins,ex1)]

> can you call pdb-stub.c::breakpoint() early on during the 
> domain startup
> to force it into the debugger?

I don't think so... pdb-stub.c is in xen and I'm not familiar
enough with the interface to add a new hypercall to do that.

I tried just adding an asm("int $3") at the point where
I want to enter gdb but I can't seem to get gdb to
capture it at that point.

> i'm planning to check in 
> some code later
> this week which will make debugging domains and processes a 
> lot easier.

Great!  Looking forward to it!

Dan


-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click

Xen at scale

[David Becker]

I'm using Xen for our cluster management project.  At the moment 219
xen-1.2 nodes  are up (my interest in multicast is to udpcast virtual disk
images to all those hosts).  At this time I'm just running two domains on
each host but that will be going up soon (as well as the total number of hosts).

For the most part all is well.  Sometimes a domain will spontaneously stop and
I've yet to figure out why.  The leading suspect is memory pressure.

My wish list based on my experience so far:
       HIGHMEM4G - This is my top wish, and I will work on it myself
            when I get a chance, though I can't say when that will be.
       halt - Xen is unstoppable!!!!   :-)
             We had an A/C outage the other night, and while trying to
             remotely shutdown sections of the cluster I discovered that
             halting DOM0 just caused xen to reboot.
       system info/logging -
           It would be very handy to have total phys memory reported
           somehow.   The cluster is heterogenous, and memory upgrades
           happen, and the net result is I rely on /proc/meminfo to know
           what is in each host.  The xenolinux meminfo just says what
           that domain was allocated.   Is total RAM reported somewhere
           I don't know?

           Viewing the vfr rules without resorting to xen_read_console
           would be handy as well.

           Getting xen console output to a log file would be useful.  
	   Running hundreds of xen_read_console is not so practical.
	   This would help in identifying those mysterious reboots.
           Most hosts in this cluster do not have serial lines or heads
	   attached.  I think I saw console output was changing in 1.3
	   so maybe this has been worked on already.

       multicast - as mentioned

       FreeBSD - I'd love a freebsd domain.  I know y'all said this was
           in the works.  Is there any current status?


Perhaps some of this is already there, and I just don't know.  More docs
are always welcome.  My own documentation track record is not so great so I
can sympathize on this one.


-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click

Re: Xen at scale

[Ian Pratt]

> 
> I'm using Xen for our cluster management project.  At the moment 219
> xen-1.2 nodes  are up (my interest in multicast is to udpcast virtual disk
> images to all those hosts).  At this time I'm just running two domains on
> each host but that will be going up soon (as well as the total number of hosts).
> 
> For the most part all is well.  Sometimes a domain will spontaneously stop and
> I've yet to figure out why.  The leading suspect is memory pressure.

That's worrying. Please try and find a reliable way to reproduce,
then try a debug build. 

We've seen some weird hangs under extreme conditions with NFS
root, but we can reproduce these on stock Linux :-(

> My wish list based on my experience so far:
>        HIGHMEM4G - This is my top wish, and I will work on it myself
>             when I get a chance, though I can't say when that
> will be.

Agreed. It should just be a case of putting the CONFIG_HIGHMEM4G
stuff back into the following files:

./include/asm-xeno/fixmap.h
./include/asm-xeno/highmem.h
./include/asm-xeno/pgtable.h
./include/asm-xeno/page.h
./arch/xeno/config.in
./arch/xeno/mm/init.c
./arch/xeno/kernel/setup.c


>        halt - Xen is unstoppable!!!!   :-)
>              We had an A/C outage the other night, and while trying to
>              remotely shutdown sections of the cluster I discovered that
>              halting DOM0 just caused xen to reboot.

I guess we need an exit code from domains...

>        system info/logging -
>            It would be very handy to have total phys memory reported
>            somehow.   The cluster is heterogenous, and memory upgrades
>            happen, and the net result is I rely on /proc/meminfo to know
>            what is in each host.  The xenolinux meminfo just says what
>            that domain was allocated.   Is total RAM reported somewhere
>            I don't know?

The xc_physinfo library returns this (via the DOM0_PHYSINFO
hypercall). It should be easy to knock up a python wrapper for
this. Would you mind having a go?


>            Viewing the vfr rules without resorting to xen_read_console
>            would be handy as well.

This should be much better under the new IO world...
 
>            Getting xen console output to a log file would be useful.  
> 	   Running hundreds of xen_read_console is not so practical.
> 	   This would help in identifying those mysterious reboots.
>            Most hosts in this cluster do not have serial lines or heads
> 	   attached.  I think I saw console output was changing in 1.3
> 	   so maybe this has been worked on already.

xen_dmesg.py any good?

>        multicast - as mentioned
> 
>        FreeBSD - I'd love a freebsd domain.  I know y'all said this was
>            in the works.  Is there any current status?

NetBSD not good enough for you? ;-)


> Perhaps some of this is already there, and I just don't know.  More docs
> are always welcome.  My own documentation track record is not so great so I
> can sympathize on this one.

We'd *love* help on user docs.

Best,
Ian


-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click

Re: Xen at scale

[David Becker]

"  Please try and find a reliable way to reproduce,

yep, working on that.  No luck so far.

" xen_dmesg.py any good?

aha.  yes, that helps.   It doesn't get copied into the install tree so
I hadn't noticed it.

" The xc_physinfo library returns this (via the DOM0_PHYSINFO
" hypercall). It should be easy to knock up a python wrapper for
" this. Would you mind having a go?

Sure.  I also have to mod xc_dom_create.py to carry along VFR rules when
domains reboot.



-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click

Re: Xen at scale

[Mark Williamson]

>       system info/logging -
>           It would be very handy to have total phys memory reported
>           somehow.   The cluster is heterogenous, and memory upgrades
>           happen, and the net result is I rely on /proc/meminfo to know
>           what is in each host.  The xenolinux meminfo just says what
>           that domain was allocated.   Is total RAM reported somewhere
>           I don't know?

In 1.2 you can find out how much memory a machine has by looking at Xen's boot 
output (tools/misc/xen_dmesg.py).  The amount of memory is near the top, e.g.:

(XEN) Initialised all memory on a 255MB machine

If you run a later version of Xen there's also the physinfo stuff that Ian 
mentioned.  I think there is already a very simple Python script that uses the 
physinfo op to get the number of CPUs, hyperthreading info, CPU speed, total 
physical memory and free physical memory.  It's in 
tools/examples/xc_physinfo.py

>           Getting xen console output to a log file would be useful.  
>	   Running hundreds of xen_read_console is not so practical.
>	   This would help in identifying those mysterious reboots.
>           Most hosts in this cluster do not have serial lines or heads
>	   attached.  I think I saw console output was changing in 1.3
>	   so maybe this has been worked on already.

If you mean domain console output, there is (I think) a daemon for logging 
domain output into a file.  It's in tools/misc/xen_log.c and AFAIK will only 
work on 1.2 and earlier.  I personally haven't used it but it might be 
something like what you want.

HTH.

Mark



-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click