* Automation scripts
@ 2004-09-27 19:23 Paul Dorman
2004-09-27 20:26 ` Paul Dorman
` (3 more replies)
0 siblings, 4 replies; 26+ messages in thread
From: Paul Dorman @ 2004-09-27 19:23 UTC (permalink / raw)
To: xen-devel
Hi all,
I wonder if anyone on the list has written any scripts to automate the
management of VMs with loopback images. Here's what I want to be able to do:
* Store existing physical machine file systems, or pristine installs in
loopback images on my Xen servers (something I'll do manually)
* Run a script that will start a VM from one of these images, automatically
associate it with a loopback device, give it a name, RAM allocation, network
addresses, and set various internal parameters, such as hostname, routes,
etc., based on a set of arguments. So something like "script <imagename>
<hostname> <netconfigs> <RAM> <.. etc.
* Have the same script take another argument that will cause it to clone a
filesystem image first before starting the VM, so that I can use a set of
images as VM templates. I intend to have a large collection of templates
which my developers can use to create VMs suited to whatever project they are
working on.
* After a VM machine has been instantiated, I would like to be able to start
and stop it with simple "start hostname" and "stop hostname" kinds of
commands.
* Have management tools so that I can for example shift a VM from one Xen
server to another (shift hostname xenservername). These would also be used by
load balancing scripts to shift machines around to manage resources.
I'd like to build a web-based management system for these scripts, so that
developers are free to create and control Xen VMs (though naturally with
limitations based on what the servers can handle -- so my bosses will know
when they need to buy me more servers :o) ).
I don't see these as particularly difficult, but if someone has done them
already .... Also, I'd appreciate any thoughts you might have on automation
of this kind, particularly in terms of functionality and practicalities.
Thanks for your time!
Paul
-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Automation scripts
2004-09-27 19:23 Automation scripts Paul Dorman
@ 2004-09-27 20:26 ` Paul Dorman
2004-09-27 21:53 ` Mark A. Williamson
2004-09-27 21:46 ` Mark A. Williamson
` (2 subsequent siblings)
3 siblings, 1 reply; 26+ messages in thread
From: Paul Dorman @ 2004-09-27 20:26 UTC (permalink / raw)
To: xen-devel
Actually, I'd also be interested in any docs or information relating to LVM
with COW as an alternative to loopback images. If I want to be able to shift
my VM images about, can I do it with LVM? The section on this in the manual
(5.2) is a little sparse on this subject to say the least! Guess you're all
too busy coding :o)
Cheers,
Paul
On Tuesday 28 September 2004 07:23 am, Paul Dorman wrote:
> Hi all,
>
> I wonder if anyone on the list has written any scripts to automate the
> management of VMs with loopback images. Here's what I want to be able to
> do:
>
> * Store existing physical machine file systems, or pristine installs in
> loopback images on my Xen servers (something I'll do manually)
>
> * Run a script that will start a VM from one of these images, automatically
> associate it with a loopback device, give it a name, RAM allocation,
> network addresses, and set various internal parameters, such as hostname,
> routes, etc., based on a set of arguments. So something like "script
> <imagename> <hostname> <netconfigs> <RAM> <.. etc.
>
> * Have the same script take another argument that will cause it to clone a
> filesystem image first before starting the VM, so that I can use a set of
> images as VM templates. I intend to have a large collection of templates
> which my developers can use to create VMs suited to whatever project they
> are working on.
>
> * After a VM machine has been instantiated, I would like to be able to
> start and stop it with simple "start hostname" and "stop hostname" kinds of
> commands.
>
> * Have management tools so that I can for example shift a VM from one Xen
> server to another (shift hostname xenservername). These would also be used
> by load balancing scripts to shift machines around to manage resources.
>
> I'd like to build a web-based management system for these scripts, so that
> developers are free to create and control Xen VMs (though naturally with
> limitations based on what the servers can handle -- so my bosses will know
> when they need to buy me more servers :o) ).
>
> I don't see these as particularly difficult, but if someone has done them
> already .... Also, I'd appreciate any thoughts you might have on automation
> of this kind, particularly in terms of functionality and practicalities.
>
> Thanks for your time!
>
> Paul
-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Automation scripts
2004-09-27 19:23 Automation scripts Paul Dorman
2004-09-27 20:26 ` Paul Dorman
@ 2004-09-27 21:46 ` Mark A. Williamson
2004-09-27 21:50 ` Paul Dorman
2004-09-30 20:58 ` Loop and ENBD device management (was Re: Automation scripts) Mark A. Williamson
2004-09-30 21:00 ` Mark A. Williamson
3 siblings, 1 reply; 26+ messages in thread
From: Mark A. Williamson @ 2004-09-27 21:46 UTC (permalink / raw)
To: xen-devel; +Cc: Paul Dorman
> I wonder if anyone on the list has written any scripts to automate the
> management of VMs with loopback images. Here's what I want to be able to
> do:
Managing loopback block devices (and other non-physical block devices) will
get friendlier than it currently is. They'll get automatically allocated,
deallocated etc by Xend.
However, the functionality you want is much like we'd envisaged for the
"cluster controller" some time in the future. The idea behind it is to
simplify the management of multiple Xen machines as a single pool of
resources.
We have some preliminary design documents on this but no implementation as
yet. There are other people working on their own cluster management schemes
(hi Brian, hi Steve ;-) but there's not a general-purpose Xen package for
doing this.
If you're interested, we can post some of our design docs on this subject.
Cheers,
Mark
-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Automation scripts
2004-09-27 21:46 ` Mark A. Williamson
@ 2004-09-27 21:50 ` Paul Dorman
2004-09-27 22:08 ` Ian Pratt
` (2 more replies)
0 siblings, 3 replies; 26+ messages in thread
From: Paul Dorman @ 2004-09-27 21:50 UTC (permalink / raw)
To: mark.williamson; +Cc: xen-devel
I would certainly appreciated this. Then I can keep my automation roughly in
line with what's going to happen anyway, so I can move from my system to the
official one without too much pain. I'm certain the official one will be much
better anyhow :o)
Another little query about the LVM stuff: I'm using kernel 2.6 so does this
rule out the LVM option? The package note for Sarge's LVM2 says that it works
with 2.4 only.
Thanks Mark.
Regards,
Paul
On Tuesday 28 September 2004 09:46 am, Mark A. Williamson wrote:
> > I wonder if anyone on the list has written any scripts to automate the
> > management of VMs with loopback images. Here's what I want to be able to
> > do:
>
> Managing loopback block devices (and other non-physical block devices) will
> get friendlier than it currently is. They'll get automatically allocated,
> deallocated etc by Xend.
>
> However, the functionality you want is much like we'd envisaged for the
> "cluster controller" some time in the future. The idea behind it is to
> simplify the management of multiple Xen machines as a single pool of
> resources.
>
> We have some preliminary design documents on this but no implementation as
> yet. There are other people working on their own cluster management
> schemes (hi Brian, hi Steve ;-) but there's not a general-purpose Xen
> package for doing this.
>
> If you're interested, we can post some of our design docs on this subject.
>
> Cheers,
> Mark
-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Automation scripts
2004-09-27 20:26 ` Paul Dorman
@ 2004-09-27 21:53 ` Mark A. Williamson
2004-09-28 14:21 ` Brian Wolfe
0 siblings, 1 reply; 26+ messages in thread
From: Mark A. Williamson @ 2004-09-27 21:53 UTC (permalink / raw)
To: xen-devel; +Cc: Paul Dorman
> Actually, I'd also be interested in any docs or information relating to LVM
> with COW as an alternative to loopback images. If I want to be able to
> shift my VM images about, can I do it with LVM?
Shift them about between machines? The ideal thing to do for that would be to
keep a copy of the base image on all machines and just copy the changes to
that around your cluster when migrating. I don't know how easy this is with
LVM... Anyone?
> The section on this in the
> manual (5.2) is a little sparse on this subject to say the least! Guess
> you're all too busy coding :o)
Yeah :-) This manual might get a bit fleshed out before the 2.0 release but I
was rather hoping someone who uses LVM would help out with the LVM
section ;-) I just use dedicated partitions for domains.
Cheers,
Mark
>
> Cheers,
> Paul
>
> On Tuesday 28 September 2004 07:23 am, Paul Dorman wrote:
> > Hi all,
> >
> > I wonder if anyone on the list has written any scripts to automate the
> > management of VMs with loopback images. Here's what I want to be able to
> > do:
> >
> > * Store existing physical machine file systems, or pristine installs in
> > loopback images on my Xen servers (something I'll do manually)
> >
> > * Run a script that will start a VM from one of these images,
> > automatically associate it with a loopback device, give it a name, RAM
> > allocation, network addresses, and set various internal parameters, such
> > as hostname, routes, etc., based on a set of arguments. So something like
> > "script <imagename> <hostname> <netconfigs> <RAM> <.. etc.
> >
> > * Have the same script take another argument that will cause it to clone
> > a filesystem image first before starting the VM, so that I can use a set
> > of images as VM templates. I intend to have a large collection of
> > templates which my developers can use to create VMs suited to whatever
> > project they are working on.
> >
> > * After a VM machine has been instantiated, I would like to be able to
> > start and stop it with simple "start hostname" and "stop hostname" kinds
> > of commands.
> >
> > * Have management tools so that I can for example shift a VM from one Xen
> > server to another (shift hostname xenservername). These would also be
> > used by load balancing scripts to shift machines around to manage
> > resources.
> >
> > I'd like to build a web-based management system for these scripts, so
> > that developers are free to create and control Xen VMs (though naturally
> > with limitations based on what the servers can handle -- so my bosses
> > will know when they need to buy me more servers :o) ).
> >
> > I don't see these as particularly difficult, but if someone has done them
> > already .... Also, I'd appreciate any thoughts you might have on
> > automation of this kind, particularly in terms of functionality and
> > practicalities.
> >
> > Thanks for your time!
> >
> > Paul
>
> -------------------------------------------------------
> This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
> Project Admins to receive an Apple iPod Mini FREE for your judgement on
> who ports your project to Linux PPC the best. Sponsored by IBM.
> Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel
-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Automation scripts
2004-09-27 21:50 ` Paul Dorman
@ 2004-09-27 22:08 ` Ian Pratt
2004-09-27 22:18 ` Michael Vrable
2004-09-28 23:59 ` Automation scripts Mark A. Williamson
2 siblings, 0 replies; 26+ messages in thread
From: Ian Pratt @ 2004-09-27 22:08 UTC (permalink / raw)
To: Paul Dorman; +Cc: mark.williamson, xen-devel, Ian.Pratt
> Another little query about the LVM stuff: I'm using kernel 2.6
> so does this rule out the LVM option? The package note for
> Sarge's LVM2 says that it works with 2.4 only.
LVM2 supports snapshots as of Linux 2.6.8
Ian
-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Automation scripts
2004-09-27 21:50 ` Paul Dorman
2004-09-27 22:08 ` Ian Pratt
@ 2004-09-27 22:18 ` Michael Vrable
2004-09-27 22:43 ` Paul Dorman
2004-09-28 8:23 ` Peri Hankey
2004-09-28 23:59 ` Automation scripts Mark A. Williamson
2 siblings, 2 replies; 26+ messages in thread
From: Michael Vrable @ 2004-09-27 22:18 UTC (permalink / raw)
To: xen-devel
On Tue, Sep 28, 2004 at 09:50:06AM +1200, Paul Dorman wrote:
> Another little query about the LVM stuff: I'm using kernel 2.6 so does this
> rule out the LVM option? The package note for Sarge's LVM2 says that it works
> with 2.4 only.
I'm using LVM2 from Sarge (a few weeks out of date now) and a 2.6.8.1
kernel on Xen. Everything works just fine. (Except for the occasional
out-of-memory conditions when using many snapshots...)
--Michael Vrable
-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Automation scripts
2004-09-27 22:18 ` Michael Vrable
@ 2004-09-27 22:43 ` Paul Dorman
2004-09-28 18:41 ` Michael Vrable
2004-09-28 8:23 ` Peri Hankey
1 sibling, 1 reply; 26+ messages in thread
From: Paul Dorman @ 2004-09-27 22:43 UTC (permalink / raw)
To: xen-devel
Hi Michael,
that's good to know. Could I ask if there's any special I have to do to work
with Sarge and Xen (and LVM2 of course!)? Any tips would be great. I'll be
mining the mailing list for a possible 'tips and tweaks' wiki section
sometime soon (as part of my contribution to Xen), so anything you can
contribute will surely make it there eventually.
Cheers,
Paul
On Tuesday 28 September 2004 10:18 am, Michael Vrable wrote:
> On Tue, Sep 28, 2004 at 09:50:06AM +1200, Paul Dorman wrote:
> > Another little query about the LVM stuff: I'm using kernel 2.6 so does
> > this rule out the LVM option? The package note for Sarge's LVM2 says that
> > it works with 2.4 only.
>
> I'm using LVM2 from Sarge (a few weeks out of date now) and a 2.6.8.1
> kernel on Xen. Everything works just fine. (Except for the occasional
> out-of-memory conditions when using many snapshots...)
>
> --Michael Vrable
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
> Project Admins to receive an Apple iPod Mini FREE for your judgement on
> who ports your project to Linux PPC the best. Sponsored by IBM.
> Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel
-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Automation scripts
2004-09-27 22:18 ` Michael Vrable
2004-09-27 22:43 ` Paul Dorman
@ 2004-09-28 8:23 ` Peri Hankey
2004-09-28 12:26 ` Ian Pratt
1 sibling, 1 reply; 26+ messages in thread
From: Peri Hankey @ 2004-09-28 8:23 UTC (permalink / raw)
To: Michael Vrable; +Cc: xen-devel
My experience (LVM2.2.00.24 - Mandrake 10.0 - 2.6.8.1-xen0) has been
that when lvm2 runs out of memory (particularly when creating a new
snapshot of an original that already has other snaphots against it) the
whole lvm system with any xenU domains based on it becomes unusable
until the xen0 system is rebooted. Has lvm2 behaved better for you in
that kind of circumstance?
Other than that, from a short acquaintance it looks as if it should be good.
-- Peri
Michael Vrable wrote:
>On Tue, Sep 28, 2004 at 09:50:06AM +1200, Paul Dorman wrote:
>
>
>>Another little query about the LVM stuff: I'm using kernel 2.6 so does this
>>rule out the LVM option? The package note for Sarge's LVM2 says that it works
>>with 2.4 only.
>>
>>
>
>I'm using LVM2 from Sarge (a few weeks out of date now) and a 2.6.8.1
>kernel on Xen. Everything works just fine. (Except for the occasional
>out-of-memory conditions when using many snapshots...)
>
>--Michael Vrable
>
>
>-------------------------------------------------------
>This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
>Project Admins to receive an Apple iPod Mini FREE for your judgement on
>who ports your project to Linux PPC the best. Sponsored by IBM.
>Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.sourceforge.net
>https://lists.sourceforge.net/lists/listinfo/xen-devel
>
>
>
>
-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Automation scripts
2004-09-28 8:23 ` Peri Hankey
@ 2004-09-28 12:26 ` Ian Pratt
2004-09-28 14:55 ` Peri Hankey
0 siblings, 1 reply; 26+ messages in thread
From: Ian Pratt @ 2004-09-28 12:26 UTC (permalink / raw)
To: Peri Hankey; +Cc: Michael Vrable, xen-devel, Ian.Pratt
> My experience (LVM2.2.00.24 - Mandrake 10.0 - 2.6.8.1-xen0) has been
> that when lvm2 runs out of memory (particularly when creating a new
> snapshot of an original that already has other snaphots against it) the
> whole lvm system with any xenU domains based on it becomes unusable
> until the xen0 system is rebooted. Has lvm2 behaved better for you in
> that kind of circumstance?
I haven't really used multiple snapshots enough to experience
this. Do you really mean that you're running out of memory, or
that the volume for holding the snapshot deltas is filling up?
I presume LVM2 just stores a cache of the remapped extents table
in memory, so I'm surprised that there's a significant memory
overhead. Maybe its not really that smart.
Either way, it might be interesting to see the output of
/proc/slabinfo when its low on memory.
Ian
-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Automation scripts
2004-09-27 21:53 ` Mark A. Williamson
@ 2004-09-28 14:21 ` Brian Wolfe
2004-09-29 10:36 ` Christian Limpach
0 siblings, 1 reply; 26+ messages in thread
From: Brian Wolfe @ 2004-09-28 14:21 UTC (permalink / raw)
To: mark.williamson; +Cc: xen-devel, Paul Dorman
The best solution so far that I have found is to have 1 (or more) NFS
servers on a backside network that provide disk IO to the individual
XenLinux unprivileged domains. This allows me to restart a xenlinux
instance on a different machine should the primary fail. This also
allows me to have a more centralized filesystem that uses nfs/iSCSI on
LVM on raid-5 for reliability.
I'm far from having a working automated system right now. I still have
the iSCSI death issues to deal with when combining 2 iSCSI target
devices into a raid-1 array in Domain-0 to be exported to the XenLinux
unprived domain. 8-( NFS roots still periodically "lock up" momentarilly
(~2 to 10 seconds) when using nfs root. SO the solution isn't ready yet.
I do highly reccomend that people use LVM LVs for individual exportable
block devices and to forget about using COWs to "save space". Adding a
200GB disk is dirt cheap (unless you are talking about scsi) so it's not
worth the savings IMHO to multi-cow a filesystem into 2+ unprived
domains on a single machine. The 1 to 2 GB of saved space just isn't
worth it at $0.10 per GB of disk space. :) Add in the system overhead
and seek times required for COWs after your first FS upgrade and your
return is worse than using separate partitions/LVs in my experience.
It's better performance and maintainability wise IMHO to use a common
RO partition for your basic OS root fs and /usr and use the traditional
/etc, /var, /usr/local redirection to a tiny config partition and unique
/home partition for per-vhost files. This is only worth it IF you can't
afford the ~2GB for independant root and /usr filesystems.
For anyone using Debian Sarge for your Domain-0 (and unprived domains) I
have a repo that's updated when I see a "stable" snapshot go by. It can
be found at http://www.terrabox.com/debian. It's a simple debian
repository of binary and source. I'm still working on getting the
packaging refined and approved by my new-DM sponsor so that the
snapshots will make it into debian testing distro in the next month or
two.
I'm working on a first draft of a Debian on xen and LVM howto. But like
Ian and group, dev work comes first till it doesn't flake out on me. ;)
If anyone is interested in a working HA solution using Xen for their
business, I can asist with design and deployment for a small donation to
my living expences fund. ;)
Brian Wolfe
TerraBox.com Inc.
linux and HA Contracting.
On Mon, 2004-09-27 at 16:53, Mark A. Williamson wrote:
> > Actually, I'd also be interested in any docs or information relating to LVM
> > with COW as an alternative to loopback images. If I want to be able to
> > shift my VM images about, can I do it with LVM?
>
> Shift them about between machines? The ideal thing to do for that would be to
> keep a copy of the base image on all machines and just copy the changes to
> that around your cluster when migrating. I don't know how easy this is with
> LVM... Anyone?
>
> > The section on this in the
> > manual (5.2) is a little sparse on this subject to say the least! Guess
> > you're all too busy coding :o)
>
> Yeah :-) This manual might get a bit fleshed out before the 2.0 release but I
> was rather hoping someone who uses LVM would help out with the LVM
> section ;-) I just use dedicated partitions for domains.
>
> Cheers,
> Mark
>
> >
> > Cheers,
> > Paul
> >
> > On Tuesday 28 September 2004 07:23 am, Paul Dorman wrote:
> > > Hi all,
> > >
> > > I wonder if anyone on the list has written any scripts to automate the
> > > management of VMs with loopback images. Here's what I want to be able to
> > > do:
> > >
> > > * Store existing physical machine file systems, or pristine installs in
> > > loopback images on my Xen servers (something I'll do manually)
> > >
> > > * Run a script that will start a VM from one of these images,
> > > automatically associate it with a loopback device, give it a name, RAM
> > > allocation, network addresses, and set various internal parameters, such
> > > as hostname, routes, etc., based on a set of arguments. So something like
> > > "script <imagename> <hostname> <netconfigs> <RAM> <.. etc.
> > >
> > > * Have the same script take another argument that will cause it to clone
> > > a filesystem image first before starting the VM, so that I can use a set
> > > of images as VM templates. I intend to have a large collection of
> > > templates which my developers can use to create VMs suited to whatever
> > > project they are working on.
> > >
> > > * After a VM machine has been instantiated, I would like to be able to
> > > start and stop it with simple "start hostname" and "stop hostname" kinds
> > > of commands.
> > >
> > > * Have management tools so that I can for example shift a VM from one Xen
> > > server to another (shift hostname xenservername). These would also be
> > > used by load balancing scripts to shift machines around to manage
> > > resources.
> > >
> > > I'd like to build a web-based management system for these scripts, so
> > > that developers are free to create and control Xen VMs (though naturally
> > > with limitations based on what the servers can handle -- so my bosses
> > > will know when they need to buy me more servers :o) ).
> > >
> > > I don't see these as particularly difficult, but if someone has done them
> > > already .... Also, I'd appreciate any thoughts you might have on
> > > automation of this kind, particularly in terms of functionality and
> > > practicalities.
> > >
> > > Thanks for your time!
> > >
> > > Paul
> >
> > -------------------------------------------------------
> > This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
> > Project Admins to receive an Apple iPod Mini FREE for your judgement on
> > who ports your project to Linux PPC the best. Sponsored by IBM.
> > Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/xen-devel
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
> Project Admins to receive an Apple iPod Mini FREE for your judgement on
> who ports your project to Linux PPC the best. Sponsored by IBM.
> Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel
-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Automation scripts
2004-09-28 12:26 ` Ian Pratt
@ 2004-09-28 14:55 ` Peri Hankey
2004-09-28 15:43 ` Ian Pratt
0 siblings, 1 reply; 26+ messages in thread
From: Peri Hankey @ 2004-09-28 14:55 UTC (permalink / raw)
To: Ian Pratt; +Cc: Michael Vrable, xen-devel
[-- Attachment #1: Type: text/plain, Size: 7250 bytes --]
It looks as if lvm2 uses quite a lot of memory analysing multiple
snapshots of a single origin. I rebooted the dom0 machine and
immediately did 'vchange -ay vmspace' which makes the volume group
vmspace active. This claims to run out of memory - always so far on one
of a list of snapshots of a common origin, but not always on the same
one. When it runs out memory adding a new snapshot, it tends to mention
one of the other snapshots, and as I have said, the whole caboodle then
becomes unusable until after a reboot.
Here is the sequence of events, with slabinfo before and after the
second 'vchange -ay vmspace':
... login after reboot (no xenU systems started) ...
[root@a4 root]# vgchange -ay vmspace
device-mapper ioctl cmd 9 failed: Cannot allocate memory
Couldn't load device 'vmspace-a44'.
26 logical volume(s) in volume group "vmspace" now active
...
[root@a4 root]# vgchange -an vmspace
0 logical volume(s) in volume group "vmspace" now active
...
[root@a4 root]# cat /proc/slabinfo >slabinfo-1
[root@a4 root]# vgchange -ay vmspace
device-mapper ioctl cmd 9 failed: Cannot allocate memory
Couldn't load device 'vmspace-a51'.
25 logical volume(s) in volume group "vmspace" now active
[root@a4 root]# cat /proc/slabinfo >slabinfo-2
[root@a4 root]# lvs
LV VG Attr LSize Origin Snap% Move Copy%
a32 vmspace -wi-a- 2.00G
a32-swap vmspace -wi-a- 64.00M
a33 vmspace -wi-a- 2.00G
a37 vmspace -wi-a- 2.00G
a38-swap vmspace -wi-a- 64.00M
a39 vmspace swi-a- 100.00M mdk10.0-a 0.02
a39-swap vmspace -wi-a- 64.00M
a40 vmspace swi--- 100.00M mdk10.0
a40-swap vmspace -wi-a- 64.00M
a41 vmspace swi--- 100.00M mdk10.0
a41-swap vmspace -wi-a- 64.00M
a42 vmspace swi-s- 100.00M mdk10.0 36.77
a42-swap vmspace -wi-a- 64.00M
a43 vmspace swi-s- 100.00M mdk10.0 35.14
a43-swap vmspace -wi-a- 64.00M
a44 vmspace Swi-S- 100.00M mdk10.0 100.00
a44-swap vmspace -wi-a- 64.00M
a45 vmspace swi-s- 100.00M mdk10.0 0.45
a45-swap vmspace -wi-a- 64.00M
a46 vmspace swi-a- 100.00M mdk10.0-a 42.78
a46-swap vmspace -wi-a- 64.00M
a47 vmspace swi-a- 100.00M mdk10.0-a 0.45
a47-swap vmspace -wi-a- 64.00M
a48 vmspace swi-a- 100.00M mdk10.0-a 0.45
a48-swap vmspace -wi-a- 64.00M
a49 vmspace swi-a- 100.00M mdk10.0-a 0.02
a49-swap vmspace -wi-a- 64.00M
a50 vmspace swi-a- 100.00M mdk10.0-a 0.45
a50-swap vmspace -wi-a- 64.00M
a51 vmspace Swi-I- 256.00M fedora-core2 100.00
a51-swap vmspace -wi-a- 64.00M
a52 vmspace swi-a- 256.00M fedora-core2 0.01
a52-swap vmspace -wi-a- 64.00M
a55 vmspace swi-a- 256.00M fedora-core2 2.53
a55-swap vmspace -wi-a- 64.00M
a56 vmspace swi-a- 256.00M fedora-core2 11.80
a56-swap vmspace -wi-a- 64.00M
a57 vmspace swi--- 256.00M fedora-core2
a57-swap vmspace -wi-a- 64.00M
fedora-core2 vmspace owi--- 2.00G
gentoo-2004.2 vmspace -wi-a- 6.00G
mdk10.0 vmspace owi--- 6.00G
mdk10.0-a vmspace owi-a- 6.00G
mdk10.0-b vmspace -wi-a- 6.00G
... reboot and login again ...
[root@a4 root]# lvs
LV VG Attr LSize Origin Snap% Move Copy%
a32 vmspace -wi--- 2.00G
a32-swap vmspace -wi--- 64.00M
a33 vmspace -wi--- 2.00G
a37 vmspace -wi--- 2.00G
a38-swap vmspace -wi--- 64.00M
a39 vmspace swi--- 100.00M mdk10.0-a
a39-swap vmspace -wi--- 64.00M
a40 vmspace swi--- 100.00M mdk10.0
a40-swap vmspace -wi--- 64.00M
a41 vmspace swi--- 100.00M mdk10.0
a41-swap vmspace -wi--- 64.00M
a42 vmspace swi--- 100.00M mdk10.0
a42-swap vmspace -wi--- 64.00M
a43 vmspace swi--- 100.00M mdk10.0
a43-swap vmspace -wi--- 64.00M
a44 vmspace swi--- 100.00M mdk10.0
a44-swap vmspace -wi--- 64.00M
a45 vmspace swi--- 100.00M mdk10.0
a45-swap vmspace -wi--- 64.00M
a46 vmspace swi--- 100.00M mdk10.0-a
a46-swap vmspace -wi--- 64.00M
a47 vmspace swi--- 100.00M mdk10.0-a
a47-swap vmspace -wi--- 64.00M
a48 vmspace swi--- 100.00M mdk10.0-a
a48-swap vmspace -wi--- 64.00M
a49 vmspace swi--- 100.00M mdk10.0-a
a49-swap vmspace -wi--- 64.00M
a50 vmspace swi--- 100.00M mdk10.0-a
a50-swap vmspace -wi--- 64.00M
a51 vmspace swi--- 256.00M fedora-core2
a51-swap vmspace -wi--- 64.00M
a52 vmspace swi--- 256.00M fedora-core2
a52-swap vmspace -wi--- 64.00M
a55 vmspace swi--- 256.00M fedora-core2
a55-swap vmspace -wi--- 64.00M
a56 vmspace swi--- 256.00M fedora-core2
a56-swap vmspace -wi--- 64.00M
a57 vmspace swi--- 256.00M fedora-core2
a57-swap vmspace -wi--- 64.00M
fedora-core2 vmspace owi--- 2.00G
gentoo-2004.2 vmspace -wi--- 6.00G
mdk10.0 vmspace owi--- 6.00G
mdk10.0-a vmspace owi--- 6.00G
mdk10.0-b vmspace -wi--- 6.00G
[root@a4 root]# lvs |wc
45 201 3195
[root@a4 root]# lvchange -a y vmspace/fedora-core2
device-mapper ioctl cmd 9 failed: Cannot allocate memory
Couldn't load device 'vmspace-a52'.
Here it runs out memory dealing making active an origin volume with just
5 snapshots. So there's clearly something wrong, and as I have said,
lvm2 snapshots weren't really designed for these purposes. It may be a
case of DIY, as Christian suggested.
-- Peri
Ian Pratt wrote:
>>My experience (LVM2.2.00.24 - Mandrake 10.0 - 2.6.8.1-xen0) has been
>>that when lvm2 runs out of memory (particularly when creating a new
>>snapshot of an original that already has other snaphots against it) the
>>whole lvm system with any xenU domains based on it becomes unusable
>>until the xen0 system is rebooted. Has lvm2 behaved better for you in
>>that kind of circumstance?
>>
>>
>
>I haven't really used multiple snapshots enough to experience
>this. Do you really mean that you're running out of memory, or
>that the volume for holding the snapshot deltas is filling up?
>
>I presume LVM2 just stores a cache of the remapped extents table
>in memory, so I'm surprised that there's a significant memory
>overhead. Maybe its not really that smart.
>
>Either way, it might be interesting to see the output of
>/proc/slabinfo when its low on memory.
>
>Ian
>
>
>-------------------------------------------------------
>This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
>Project Admins to receive an Apple iPod Mini FREE for your judgement on
>who ports your project to Linux PPC the best. Sponsored by IBM.
>Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.sourceforge.net
>https://lists.sourceforge.net/lists/listinfo/xen-devel
>
>
>
>
[-- Attachment #2: slabinfo-1 --]
[-- Type: text/plain, Size: 13367 bytes --]
slabinfo - version: 2.0
# name <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <batchcount> <limit> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail>
kcopyd-jobs 512 525 184 21 1 : tunables 120 60 0 : slabdata 25 25 0
dm-io-5 16 16 3072 2 2 : tunables 24 12 0 : slabdata 8 8 0
dm-io-4 32 35 1536 5 2 : tunables 24 12 0 : slabdata 7 7 0
dm-io-3 64 65 768 5 1 : tunables 54 27 0 : slabdata 13 13 0
dm-io-2 128 140 192 20 1 : tunables 120 60 0 : slabdata 7 7 0
dm-io-1 256 305 64 61 1 : tunables 120 60 0 : slabdata 5 5 0
dm-io-0 512 678 16 226 1 : tunables 120 60 0 : slabdata 3 3 0
dm-io-bio 512 549 64 61 1 : tunables 120 60 0 : slabdata 9 9 0
rpc_buffers 8 8 2048 2 1 : tunables 24 12 0 : slabdata 4 4 0
rpc_tasks 8 25 160 25 1 : tunables 120 60 0 : slabdata 1 1 0
rpc_inode_cache 0 0 416 9 1 : tunables 54 27 0 : slabdata 0 0 0
bridge_fdb_cache 0 0 64 61 1 : tunables 120 60 0 : slabdata 0 0 0
unix_sock 101 110 384 10 1 : tunables 54 27 0 : slabdata 11 11 0
tcp_tw_bucket 0 0 96 41 1 : tunables 120 60 0 : slabdata 0 0 0
tcp_bind_bucket 19 226 16 226 1 : tunables 120 60 0 : slabdata 1 1 0
tcp_open_request 0 0 64 61 1 : tunables 120 60 0 : slabdata 0 0 0
inet_peer_cache 1 61 64 61 1 : tunables 120 60 0 : slabdata 1 1 0
ip_fib_hash 10 226 16 226 1 : tunables 120 60 0 : slabdata 1 1 0
ip_dst_cache 10 15 256 15 1 : tunables 120 60 0 : slabdata 1 1 0
arp_cache 3 31 128 31 1 : tunables 120 60 0 : slabdata 1 1 0
raw4_sock 0 0 480 8 1 : tunables 54 27 0 : slabdata 0 0 0
udp_sock 7 8 480 8 1 : tunables 54 27 0 : slabdata 1 1 0
tcp_sock 29 32 1024 4 1 : tunables 54 27 0 : slabdata 8 8 0
flow_cache 0 0 96 41 1 : tunables 120 60 0 : slabdata 0 0 0
dm-snapshot-in 128 162 48 81 1 : tunables 120 60 0 : slabdata 2 2 0
dm-snapshot-ex 9240 9266 16 226 1 : tunables 120 60 0 : slabdata 41 41 0
dm_tio 2304 2486 16 226 1 : tunables 120 60 0 : slabdata 11 11 0
dm_io 2304 2486 16 226 1 : tunables 120 60 0 : slabdata 11 11 0
blkif_cache 0 0 112 35 1 : tunables 120 60 0 : slabdata 0 0 0
nfs_write_data 36 36 448 9 1 : tunables 54 27 0 : slabdata 4 4 0
nfs_read_data 32 36 416 9 1 : tunables 54 27 0 : slabdata 4 4 0
nfs_inode_cache 0 0 544 7 1 : tunables 54 27 0 : slabdata 0 0 0
nfs_page 0 0 96 41 1 : tunables 120 60 0 : slabdata 0 0 0
isofs_inode_cache 0 0 320 12 1 : tunables 54 27 0 : slabdata 0 0 0
ext2_inode_cache 0 0 416 9 1 : tunables 54 27 0 : slabdata 0 0 0
journal_handle 8 135 28 135 1 : tunables 120 60 0 : slabdata 1 1 0
journal_head 67 162 48 81 1 : tunables 120 60 0 : slabdata 2 2 0
revoke_table 8 290 12 290 1 : tunables 120 60 0 : slabdata 1 1 0
revoke_record 0 0 16 226 1 : tunables 120 60 0 : slabdata 0 0 0
ext3_inode_cache 4376 5517 448 9 1 : tunables 54 27 0 : slabdata 613 613 0
ext3_xattr 0 0 44 88 1 : tunables 120 60 0 : slabdata 0 0 0
reiser_inode_cache 0 0 384 10 1 : tunables 54 27 0 : slabdata 0 0 0
eventpoll_pwq 0 0 36 107 1 : tunables 120 60 0 : slabdata 0 0 0
eventpoll_epi 0 0 96 41 1 : tunables 120 60 0 : slabdata 0 0 0
kioctx 0 0 160 25 1 : tunables 120 60 0 : slabdata 0 0 0
kiocb 0 0 96 41 1 : tunables 120 60 0 : slabdata 0 0 0
dnotify_cache 0 0 20 185 1 : tunables 120 60 0 : slabdata 0 0 0
file_lock_cache 5 43 92 43 1 : tunables 120 60 0 : slabdata 1 1 0
fasync_cache 0 0 16 226 1 : tunables 120 60 0 : slabdata 0 0 0
shmem_inode_cache 5 10 384 10 1 : tunables 54 27 0 : slabdata 1 1 0
posix_timers_cache 0 0 96 41 1 : tunables 120 60 0 : slabdata 0 0 0
uid_cache 8 119 32 119 1 : tunables 120 60 0 : slabdata 1 1 0
xen-skb 17 17 4096 1 1 : tunables 24 12 0 : slabdata 17 17 0
sgpool-128 32 32 2048 2 1 : tunables 24 12 0 : slabdata 16 16 0
sgpool-64 32 32 1024 4 1 : tunables 54 27 0 : slabdata 8 8 0
sgpool-32 32 32 512 8 1 : tunables 54 27 0 : slabdata 4 4 0
sgpool-16 32 45 256 15 1 : tunables 120 60 0 : slabdata 3 3 0
sgpool-8 32 62 128 31 1 : tunables 120 60 0 : slabdata 2 2 0
cfq_pool 64 119 32 119 1 : tunables 120 60 0 : slabdata 1 1 0
crq_pool 0 0 36 107 1 : tunables 120 60 0 : slabdata 0 0 0
deadline_drq 0 0 48 81 1 : tunables 120 60 0 : slabdata 0 0 0
as_arq 523 585 60 65 1 : tunables 120 60 0 : slabdata 9 9 0
blkdev_ioc 57 185 20 185 1 : tunables 120 60 0 : slabdata 1 1 0
blkdev_queue 412 414 448 9 1 : tunables 54 27 0 : slabdata 46 46 0
blkdev_requests 522 546 152 26 1 : tunables 120 60 0 : slabdata 21 21 0
biovec-(256) 60 60 3072 2 2 : tunables 24 12 0 : slabdata 30 30 0
biovec-128 121 125 1536 5 2 : tunables 24 12 0 : slabdata 25 25 0
biovec-64 242 245 768 5 1 : tunables 54 27 0 : slabdata 49 49 0
biovec-16 242 260 192 20 1 : tunables 120 60 0 : slabdata 13 13 0
biovec-4 242 244 64 61 1 : tunables 120 60 0 : slabdata 4 4 0
biovec-1 267 452 16 226 1 : tunables 120 60 0 : slabdata 2 2 0
bio 272 305 64 61 1 : tunables 120 60 0 : slabdata 5 5 0
sock_inode_cache 139 143 352 11 1 : tunables 54 27 0 : slabdata 13 13 0
skbuff_head_cache 25 40 192 20 1 : tunables 120 60 0 : slabdata 2 2 0
sock 2 12 320 12 1 : tunables 54 27 0 : slabdata 1 1 0
proc_inode_cache 151 312 320 12 1 : tunables 54 27 0 : slabdata 26 26 0
sigqueue 16 27 148 27 1 : tunables 120 60 0 : slabdata 1 1 0
radix_tree_node 787 1876 276 14 1 : tunables 54 27 0 : slabdata 134 134 0
bdev_cache 14 18 416 9 1 : tunables 54 27 0 : slabdata 2 2 0
mnt_cache 17 41 96 41 1 : tunables 120 60 0 : slabdata 1 1 0
inode_cache 4405 4410 288 14 1 : tunables 54 27 0 : slabdata 315 315 0
dentry_cache 12133 17836 140 28 1 : tunables 120 60 0 : slabdata 637 637 0
filp 825 825 160 25 1 : tunables 120 60 0 : slabdata 33 33 0
names_cache 1 1 4096 1 1 : tunables 24 12 0 : slabdata 1 1 0
idr_layer_cache 66 87 136 29 1 : tunables 120 60 0 : slabdata 3 3 0
buffer_head 350 1620 48 81 1 : tunables 120 60 0 : slabdata 20 20 0
mm_struct 61 63 512 7 1 : tunables 54 27 0 : slabdata 9 9 0
vm_area_struct 2491 2491 84 47 1 : tunables 120 60 0 : slabdata 53 53 0
fs_cache 68 119 32 119 1 : tunables 120 60 0 : slabdata 1 1 0
files_cache 61 72 416 9 1 : tunables 54 27 0 : slabdata 8 8 0
signal_cache 97 123 96 41 1 : tunables 120 60 0 : slabdata 3 3 0
sighand_cache 72 72 1312 3 1 : tunables 24 12 0 : slabdata 24 24 0
task_struct 95 95 1424 5 2 : tunables 24 12 0 : slabdata 19 19 0
anon_vma 740 814 8 407 1 : tunables 120 60 0 : slabdata 2 2 0
pgd 47 47 4096 1 1 : tunables 24 12 0 : slabdata 47 47 0
pte 183 183 4096 1 1 : tunables 24 12 0 : slabdata 183 183 0
size-131072(DMA) 0 0 131072 1 32 : tunables 8 4 0 : slabdata 0 0 0
size-131072 1 1 131072 1 32 : tunables 8 4 0 : slabdata 1 1 0
size-65536(DMA) 0 0 65536 1 16 : tunables 8 4 0 : slabdata 0 0 0
size-65536 1 1 65536 1 16 : tunables 8 4 0 : slabdata 1 1 0
size-32768(DMA) 0 0 32768 1 8 : tunables 8 4 0 : slabdata 0 0 0
size-32768 0 0 32768 1 8 : tunables 8 4 0 : slabdata 0 0 0
size-16384(DMA) 0 0 16384 1 4 : tunables 8 4 0 : slabdata 0 0 0
size-16384 0 0 16384 1 4 : tunables 8 4 0 : slabdata 0 0 0
size-8192(DMA) 0 0 8192 1 2 : tunables 8 4 0 : slabdata 0 0 0
size-8192 84 84 8192 1 2 : tunables 8 4 0 : slabdata 84 84 0
size-4096(DMA) 0 0 4096 1 1 : tunables 24 12 0 : slabdata 0 0 0
size-4096 42 42 4096 1 1 : tunables 24 12 0 : slabdata 42 42 0
size-2048(DMA) 0 0 2048 2 1 : tunables 24 12 0 : slabdata 0 0 0
size-2048 20 20 2048 2 1 : tunables 24 12 0 : slabdata 10 10 0
size-1024(DMA) 0 0 1024 4 1 : tunables 54 27 0 : slabdata 0 0 0
size-1024 110 140 1024 4 1 : tunables 54 27 0 : slabdata 35 35 0
size-512(DMA) 0 0 512 8 1 : tunables 54 27 0 : slabdata 0 0 0
size-512 281 536 512 8 1 : tunables 54 27 0 : slabdata 67 67 0
size-256(DMA) 0 0 256 15 1 : tunables 120 60 0 : slabdata 0 0 0
size-256 678 690 256 15 1 : tunables 120 60 0 : slabdata 46 46 0
size-128(DMA) 0 0 128 31 1 : tunables 120 60 0 : slabdata 0 0 0
size-128 960 1054 128 31 1 : tunables 120 60 0 : slabdata 34 34 0
size-64(DMA) 0 0 64 61 1 : tunables 120 60 0 : slabdata 0 0 0
size-64 491 671 64 61 1 : tunables 120 60 0 : slabdata 11 11 0
size-32(DMA) 0 0 32 119 1 : tunables 120 60 0 : slabdata 0 0 0
size-32 6243 8211 32 119 1 : tunables 120 60 0 : slabdata 69 69 0
kmem_cache 124 124 128 31 1 : tunables 120 60 0 : slabdata 4 4 0
[-- Attachment #3: slabinfo-2 --]
[-- Type: text/plain, Size: 13367 bytes --]
slabinfo - version: 2.0
# name <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <batchcount> <limit> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail>
kcopyd-jobs 512 525 184 21 1 : tunables 120 60 0 : slabdata 25 25 0
dm-io-5 16 16 3072 2 2 : tunables 24 12 0 : slabdata 8 8 0
dm-io-4 32 35 1536 5 2 : tunables 24 12 0 : slabdata 7 7 0
dm-io-3 64 65 768 5 1 : tunables 54 27 0 : slabdata 13 13 0
dm-io-2 128 140 192 20 1 : tunables 120 60 0 : slabdata 7 7 0
dm-io-1 256 305 64 61 1 : tunables 120 60 0 : slabdata 5 5 0
dm-io-0 512 678 16 226 1 : tunables 120 60 0 : slabdata 3 3 0
dm-io-bio 512 549 64 61 1 : tunables 120 60 0 : slabdata 9 9 0
rpc_buffers 8 8 2048 2 1 : tunables 24 12 0 : slabdata 4 4 0
rpc_tasks 8 25 160 25 1 : tunables 120 60 0 : slabdata 1 1 0
rpc_inode_cache 0 0 416 9 1 : tunables 54 27 0 : slabdata 0 0 0
bridge_fdb_cache 0 0 64 61 1 : tunables 120 60 0 : slabdata 0 0 0
unix_sock 102 110 384 10 1 : tunables 54 27 0 : slabdata 11 11 0
tcp_tw_bucket 0 0 96 41 1 : tunables 120 60 0 : slabdata 0 0 0
tcp_bind_bucket 19 226 16 226 1 : tunables 120 60 0 : slabdata 1 1 0
tcp_open_request 0 0 64 61 1 : tunables 120 60 0 : slabdata 0 0 0
inet_peer_cache 1 61 64 61 1 : tunables 120 60 0 : slabdata 1 1 0
ip_fib_hash 10 226 16 226 1 : tunables 120 60 0 : slabdata 1 1 0
ip_dst_cache 9 15 256 15 1 : tunables 120 60 0 : slabdata 1 1 0
arp_cache 3 31 128 31 1 : tunables 120 60 0 : slabdata 1 1 0
raw4_sock 0 0 480 8 1 : tunables 54 27 0 : slabdata 0 0 0
udp_sock 7 8 480 8 1 : tunables 54 27 0 : slabdata 1 1 0
tcp_sock 29 32 1024 4 1 : tunables 54 27 0 : slabdata 8 8 0
flow_cache 0 0 96 41 1 : tunables 120 60 0 : slabdata 0 0 0
dm-snapshot-in 128 162 48 81 1 : tunables 120 60 0 : slabdata 2 2 0
dm-snapshot-ex 19556 19662 16 226 1 : tunables 120 60 0 : slabdata 87 87 0
dm_tio 14336 14464 16 226 1 : tunables 120 60 0 : slabdata 64 64 0
dm_io 14336 14464 16 226 1 : tunables 120 60 0 : slabdata 64 64 0
blkif_cache 0 0 112 35 1 : tunables 120 60 0 : slabdata 0 0 0
nfs_write_data 36 36 448 9 1 : tunables 54 27 0 : slabdata 4 4 0
nfs_read_data 32 36 416 9 1 : tunables 54 27 0 : slabdata 4 4 0
nfs_inode_cache 0 0 544 7 1 : tunables 54 27 0 : slabdata 0 0 0
nfs_page 0 0 96 41 1 : tunables 120 60 0 : slabdata 0 0 0
isofs_inode_cache 0 0 320 12 1 : tunables 54 27 0 : slabdata 0 0 0
ext2_inode_cache 0 0 416 9 1 : tunables 54 27 0 : slabdata 0 0 0
journal_handle 3 135 28 135 1 : tunables 120 60 0 : slabdata 1 1 0
journal_head 135 162 48 81 1 : tunables 120 60 0 : slabdata 2 2 0
revoke_table 8 290 12 290 1 : tunables 120 60 0 : slabdata 1 1 0
revoke_record 0 0 16 226 1 : tunables 120 60 0 : slabdata 0 0 0
ext3_inode_cache 4193 5445 448 9 1 : tunables 54 27 0 : slabdata 605 605 0
ext3_xattr 0 0 44 88 1 : tunables 120 60 0 : slabdata 0 0 0
reiser_inode_cache 0 0 384 10 1 : tunables 54 27 0 : slabdata 0 0 0
eventpoll_pwq 0 0 36 107 1 : tunables 120 60 0 : slabdata 0 0 0
eventpoll_epi 0 0 96 41 1 : tunables 120 60 0 : slabdata 0 0 0
kioctx 0 0 160 25 1 : tunables 120 60 0 : slabdata 0 0 0
kiocb 0 0 96 41 1 : tunables 120 60 0 : slabdata 0 0 0
dnotify_cache 0 0 20 185 1 : tunables 120 60 0 : slabdata 0 0 0
file_lock_cache 5 43 92 43 1 : tunables 120 60 0 : slabdata 1 1 0
fasync_cache 0 0 16 226 1 : tunables 120 60 0 : slabdata 0 0 0
shmem_inode_cache 5 10 384 10 1 : tunables 54 27 0 : slabdata 1 1 0
posix_timers_cache 0 0 96 41 1 : tunables 120 60 0 : slabdata 0 0 0
uid_cache 8 119 32 119 1 : tunables 120 60 0 : slabdata 1 1 0
xen-skb 17 17 4096 1 1 : tunables 24 12 0 : slabdata 17 17 0
sgpool-128 32 32 2048 2 1 : tunables 24 12 0 : slabdata 16 16 0
sgpool-64 32 32 1024 4 1 : tunables 54 27 0 : slabdata 8 8 0
sgpool-32 32 32 512 8 1 : tunables 54 27 0 : slabdata 4 4 0
sgpool-16 32 45 256 15 1 : tunables 120 60 0 : slabdata 3 3 0
sgpool-8 32 62 128 31 1 : tunables 120 60 0 : slabdata 2 2 0
cfq_pool 64 119 32 119 1 : tunables 120 60 0 : slabdata 1 1 0
crq_pool 0 0 36 107 1 : tunables 120 60 0 : slabdata 0 0 0
deadline_drq 0 0 48 81 1 : tunables 120 60 0 : slabdata 0 0 0
as_arq 533 585 60 65 1 : tunables 120 60 0 : slabdata 9 9 0
blkdev_ioc 41 185 20 185 1 : tunables 120 60 0 : slabdata 1 1 0
blkdev_queue 459 459 448 9 1 : tunables 54 27 0 : slabdata 51 51 0
blkdev_requests 546 546 152 26 1 : tunables 120 60 0 : slabdata 21 21 0
biovec-(256) 60 60 3072 2 2 : tunables 24 12 0 : slabdata 30 30 0
biovec-128 121 125 1536 5 2 : tunables 24 12 0 : slabdata 25 25 0
biovec-64 242 245 768 5 1 : tunables 54 27 0 : slabdata 49 49 0
biovec-16 242 260 192 20 1 : tunables 120 60 0 : slabdata 13 13 0
biovec-4 242 244 64 61 1 : tunables 120 60 0 : slabdata 4 4 0
biovec-1 256 452 16 226 1 : tunables 120 60 0 : slabdata 2 2 0
bio 305 305 64 61 1 : tunables 120 60 0 : slabdata 5 5 0
sock_inode_cache 143 143 352 11 1 : tunables 54 27 0 : slabdata 13 13 0
skbuff_head_cache 19 40 192 20 1 : tunables 120 60 0 : slabdata 2 2 0
sock 12 12 320 12 1 : tunables 54 27 0 : slabdata 1 1 0
proc_inode_cache 151 312 320 12 1 : tunables 54 27 0 : slabdata 26 26 0
sigqueue 4 27 148 27 1 : tunables 120 60 0 : slabdata 1 1 0
radix_tree_node 784 1876 276 14 1 : tunables 54 27 0 : slabdata 134 134 0
bdev_cache 30 36 416 9 1 : tunables 54 27 0 : slabdata 4 4 0
mnt_cache 17 41 96 41 1 : tunables 120 60 0 : slabdata 1 1 0
inode_cache 4687 4690 288 14 1 : tunables 54 27 0 : slabdata 335 335 0
dentry_cache 11606 17528 140 28 1 : tunables 120 60 0 : slabdata 626 626 0
filp 785 850 160 25 1 : tunables 120 60 0 : slabdata 34 34 0
names_cache 1 1 4096 1 1 : tunables 24 12 0 : slabdata 1 1 0
idr_layer_cache 66 87 136 29 1 : tunables 120 60 0 : slabdata 3 3 0
buffer_head 387 1620 48 81 1 : tunables 120 60 0 : slabdata 20 20 0
mm_struct 70 70 512 7 1 : tunables 54 27 0 : slabdata 10 10 0
vm_area_struct 2486 2491 84 47 1 : tunables 120 60 0 : slabdata 53 53 0
fs_cache 68 119 32 119 1 : tunables 120 60 0 : slabdata 1 1 0
files_cache 46 72 416 9 1 : tunables 54 27 0 : slabdata 8 8 0
signal_cache 97 123 96 41 1 : tunables 120 60 0 : slabdata 3 3 0
sighand_cache 71 78 1312 3 1 : tunables 24 12 0 : slabdata 25 26 0
task_struct 84 95 1424 5 2 : tunables 24 12 0 : slabdata 19 19 0
anon_vma 732 814 8 407 1 : tunables 120 60 0 : slabdata 2 2 0
pgd 58 58 4096 1 1 : tunables 24 12 0 : slabdata 58 58 0
pte 193 196 4096 1 1 : tunables 24 12 0 : slabdata 193 196 0
size-131072(DMA) 0 0 131072 1 32 : tunables 8 4 0 : slabdata 0 0 0
size-131072 2 2 131072 1 32 : tunables 8 4 0 : slabdata 2 2 0
size-65536(DMA) 0 0 65536 1 16 : tunables 8 4 0 : slabdata 0 0 0
size-65536 0 0 65536 1 16 : tunables 8 4 0 : slabdata 0 0 0
size-32768(DMA) 0 0 32768 1 8 : tunables 8 4 0 : slabdata 0 0 0
size-32768 0 0 32768 1 8 : tunables 8 4 0 : slabdata 0 0 0
size-16384(DMA) 0 0 16384 1 4 : tunables 8 4 0 : slabdata 0 0 0
size-16384 0 0 16384 1 4 : tunables 8 4 0 : slabdata 0 0 0
size-8192(DMA) 0 0 8192 1 2 : tunables 8 4 0 : slabdata 0 0 0
size-8192 87 88 8192 1 2 : tunables 8 4 0 : slabdata 87 88 0
size-4096(DMA) 0 0 4096 1 1 : tunables 24 12 0 : slabdata 0 0 0
size-4096 42 42 4096 1 1 : tunables 24 12 0 : slabdata 42 42 0
size-2048(DMA) 0 0 2048 2 1 : tunables 24 12 0 : slabdata 0 0 0
size-2048 20 20 2048 2 1 : tunables 24 12 0 : slabdata 10 10 0
size-1024(DMA) 0 0 1024 4 1 : tunables 54 27 0 : slabdata 0 0 0
size-1024 156 156 1024 4 1 : tunables 54 27 0 : slabdata 39 39 0
size-512(DMA) 0 0 512 8 1 : tunables 54 27 0 : slabdata 0 0 0
size-512 282 536 512 8 1 : tunables 54 27 0 : slabdata 67 67 0
size-256(DMA) 0 0 256 15 1 : tunables 120 60 0 : slabdata 0 0 0
size-256 780 780 256 15 1 : tunables 120 60 0 : slabdata 52 52 0
size-128(DMA) 0 0 128 31 1 : tunables 120 60 0 : slabdata 0 0 0
size-128 1061 1085 128 31 1 : tunables 120 60 0 : slabdata 35 35 0
size-64(DMA) 0 0 64 61 1 : tunables 120 60 0 : slabdata 0 0 0
size-64 491 671 64 61 1 : tunables 120 60 0 : slabdata 11 11 0
size-32(DMA) 0 0 32 119 1 : tunables 120 60 0 : slabdata 0 0 0
size-32 23680 23681 32 119 1 : tunables 120 60 0 : slabdata 199 199 0
kmem_cache 124 124 128 31 1 : tunables 120 60 0 : slabdata 4 4 0
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Automation scripts
2004-09-28 14:55 ` Peri Hankey
@ 2004-09-28 15:43 ` Ian Pratt
2004-09-28 18:09 ` LVM Snapshot Troubles Michael Vrable
0 siblings, 1 reply; 26+ messages in thread
From: Ian Pratt @ 2004-09-28 15:43 UTC (permalink / raw)
To: Peri Hankey; +Cc: Ian Pratt, Michael Vrable, xen-devel
> It looks as if lvm2 uses quite a lot of memory analysing multiple
> snapshots of a single origin. I rebooted the dom0 machine and
> immediately did 'vchange -ay vmspace' which makes the volume group
> vmspace active. This claims to run out of memory - always so far on one
> of a list of snapshots of a common origin, but not always on the same
> one. When it runs out memory adding a new snapshot, it tends to mention
> one of the other snapshots, and as I have said, the whole caboodle then
> becomes unusable until after a reboot.
> dm-snapshot-in 128 162 48
> dm-snapshot-ex 19556 19662 16
> dm_tio 14336 14464 16
> dm_io 14336 14464 16
20,000 dm-snapshot-ex entries sounds like a lot, but they're only
16 bytes each. From looking at the code it looks like it needs to
store the entire exception table in memory rather than using it
as a cache of what's on disk, which is a shame. Still, 50k
x16 bytes is less than 1MB.
There's nothing in slabinfo that looks crazy. I wander where all
your memory is gone? BTW: how big is your dom0?
It's possible that dm-io or kcopyd is chewing up pages (which
won't show up in the slab allocator). I'm surprised they're not
just transient, though.
Perhaps someone's going to need to take a look at device
mapper. Building a version with debug printk's enabled would be a
good start. I'd like to know what allocation is failing.
Ian
-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: LVM Snapshot Troubles
2004-09-28 15:43 ` Ian Pratt
@ 2004-09-28 18:09 ` Michael Vrable
2004-09-28 20:06 ` Ian Pratt
0 siblings, 1 reply; 26+ messages in thread
From: Michael Vrable @ 2004-09-28 18:09 UTC (permalink / raw)
To: xen-devel
[-- Attachment #1: Type: text/plain, Size: 2433 bytes --]
On Tue, Sep 28, 2004 at 04:43:25PM +0100, Ian Pratt wrote:
> There's nothing in slabinfo that looks crazy. I wander where all
> your memory is gone? BTW: how big is your dom0?
>
> It's possible that dm-io or kcopyd is chewing up pages (which
> won't show up in the slab allocator). I'm surprised they're not
> just transient, though.
When I've run into memory trouble with snapshots, I've always seen a
stack backtrace that points me at kcopyd_client_create. Following the
code: when creating a snapshot, a new kcopyd client is created with 256
(SNAPSHOT_PAGES in dm-snap.c) pages (= 1 MB) dedicated to that snapshot.
I think I managed to dig up the logs from one of the failures I've seen;
I've attached them to this message.
The problem seems to be made worse by the fact that all 256 pages are
allocated in a fairly short span of time, and (at least this is my
guess) the allocation fails even if it would be possible for the kernel
to free up the necessary memory with a bit more work.
(I've been able to create many more snapshots before running into
trouble if I try to make sure the kernel has a bit of extra free memory
before each lvcreate call--using dd to create a several megabyte file,
then deleting it to free up that space in the page cache.)
As has been noted, LVM doesn't have a very graceful failure mode when
this memory allocation problem is hit--I lose access to all the
snapshots when that happens.
I have also found that I can use dmsetup to create the COW devices
myself, which did at least (if I'm remembering correctly--this was a
little bit ago) have the benefit that if one snapshot failed, the others
were still available. Basically, I used the same setup that LVM
normally would, except that I didn't create a snapshot-origin device
layered over the original device (this is what intercepts writes to the
source device and propagates a copy of the original data to each
snapshot, if needed). Doing this manually isn't ideal, however.
Improvements that I think could be made:
- Change the dm-snapshot driver in the kernel to (optionally?)
allocate less memory for each snapshot, and fail more gracefully if
unable to allocate the memory.
- Adjust the LVM userspace tool to fail more gracefully if the device
mapper driver gives an out-of-memory error.
- Add an option to LVM for snapshots with a read-only origin (as I was
doing manually with dmsetup).
--Michael Vrable
[-- Attachment #2: lvm_error_log --]
[-- Type: text/plain, Size: 1600 bytes --]
Sep 3 18:33:51 localhost kernel: dmsetup: page allocation failure. order:0, mode:0xd0
Sep 3 18:33:51 localhost kernel: [__alloc_pages+727/839] __alloc_pages+0x2d7/0x347
Sep 3 18:33:51 localhost kernel: [kmem_cache_alloc+108/112] kmem_cache_alloc+0x6c/0x70
Sep 3 18:33:51 localhost kernel: [alloc_pl+51/82] alloc_pl+0x33/0x52
Sep 3 18:33:51 localhost kernel: [client_alloc_pages+28/87] client_alloc_pages+0x1c/0x57
Sep 3 18:33:51 localhost kernel: [vmalloc+32/36] vmalloc+0x20/0x24
Sep 3 18:33:51 localhost kernel: [kcopyd_client_create+104/185] kcopyd_client_create+0x68/0xb9
Sep 3 18:33:51 localhost kernel: [dm_create_persistent+199/305] dm_create_persistent+0xc7/0x131
Sep 3 18:33:51 localhost kernel: [snapshot_ctr+676/874] snapshot_ctr+0x2a4/0x36a
Sep 3 18:33:51 localhost kernel: [dm_table_add_target+262/422] dm_table_add_target+0x106/0x1a6
Sep 3 18:33:51 localhost kernel: [populate_table+125/210] populate_table+0x7d/0xd2
Sep 3 18:33:51 localhost kernel: [table_load+103/298] table_load+0x67/0x12a
Sep 3 18:33:51 localhost kernel: [ctl_ioctl+242/336] ctl_ioctl+0xf2/0x150
Sep 3 18:33:51 localhost kernel: [table_load+0/298] table_load+0x0/0x12a
Sep 3 18:33:51 localhost kernel: [ctl_ioctl+0/336] ctl_ioctl+0x0/0x150
Sep 3 18:33:51 localhost kernel: [sys_ioctl+247/588] sys_ioctl+0xf7/0x24c
Sep 3 18:33:51 localhost kernel: [syscall_call+7/11] syscall_call+0x7/0xb
Sep 3 18:33:51 localhost kernel:
Sep 3 18:33:51 localhost kernel: device-mapper: : Could not create kcopyd client
Sep 3 18:33:51 localhost kernel: device-mapper: error adding target to table
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Automation scripts
2004-09-27 22:43 ` Paul Dorman
@ 2004-09-28 18:41 ` Michael Vrable
0 siblings, 0 replies; 26+ messages in thread
From: Michael Vrable @ 2004-09-28 18:41 UTC (permalink / raw)
To: xen-devel
On Tue, Sep 28, 2004 at 10:43:47AM +1200, Paul Dorman wrote:
> that's good to know. Could I ask if there's any special I have to do to work
> with Sarge and Xen (and LVM2 of course!)? Any tips would be great. I'll be
> mining the mailing list for a possible 'tips and tweaks' wiki section
> sometime soon (as part of my contribution to Xen), so anything you can
> contribute will surely make it there eventually.
No special tricks, really. My basic setup:
- Set up domain-0 using raw disk partitions, so I don't have to worry
about starting LVM from an initrd.
- Leave a large disk partition unused initially; it will be devoted to
LVM.
- Install the lvm2 package from Debian in domain-0.
- Run pvcreate on the empty large partition, followed by vgcreate.
- Use the LVM tools in domain-0 to allocate space for the guest
domains, then export those devices to the guests. The guests don't
know the space comes from LVM.
If necessary, I can add more space to domain-0 with LVM as well, but
thus far I've been getting by just fine with the initial partitions for
domain-0, and using LVM solely for space for xenU domains.
--Michael Vrable
-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: LVM Snapshot Troubles
2004-09-28 18:09 ` LVM Snapshot Troubles Michael Vrable
@ 2004-09-28 20:06 ` Ian Pratt
2004-09-28 20:52 ` Michael Vrable
0 siblings, 1 reply; 26+ messages in thread
From: Ian Pratt @ 2004-09-28 20:06 UTC (permalink / raw)
To: Michael Vrable; +Cc: xen-devel, Ian.Pratt
> On Tue, Sep 28, 2004 at 04:43:25PM +0100, Ian Pratt wrote:
> > There's nothing in slabinfo that looks crazy. I wander where all
> > your memory is gone? BTW: how big is your dom0?
> >
> > It's possible that dm-io or kcopyd is chewing up pages (which
> > won't show up in the slab allocator). I'm surprised they're not
> > just transient, though.
>
> When I've run into memory trouble with snapshots, I've always seen a
> stack backtrace that points me at kcopyd_client_create. Following the
> code: when creating a snapshot, a new kcopyd client is created with 256
> (SNAPSHOT_PAGES in dm-snap.c) pages (= 1 MB) dedicated to that snapshot.
> I think I managed to dig up the logs from one of the failures I've seen;
> I've attached them to this message.
It might be worth adding "| __GFP_REPEAT" to the alloc_page in
drivers/md/kcopyd.c
If this fixes things, we could probably get the patch accepted
into mainline Linux.
(Of course, GFP_REPEAT may be a nop under Linux's VM...)
Ian
-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: LVM Snapshot Troubles
2004-09-28 20:06 ` Ian Pratt
@ 2004-09-28 20:52 ` Michael Vrable
2004-09-28 21:23 ` Ian Pratt
0 siblings, 1 reply; 26+ messages in thread
From: Michael Vrable @ 2004-09-28 20:52 UTC (permalink / raw)
To: xen-devel
On Tue, Sep 28, 2004 at 09:06:59PM +0100, Ian Pratt wrote:
> It might be worth adding "| __GFP_REPEAT" to the alloc_page in
> drivers/md/kcopyd.c
I think that __GFP_REPEAT is a no-op for single-page allocations, as in
this case (though I haven't tried it). __GFP_NOFAIL might work, but
that sounds like a cure worse than the disease.
--Michael Vrable
-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: LVM Snapshot Troubles
2004-09-28 20:52 ` Michael Vrable
@ 2004-09-28 21:23 ` Ian Pratt
2004-09-29 8:39 ` Keir Fraser
2004-09-30 21:02 ` Michael Vrable
0 siblings, 2 replies; 26+ messages in thread
From: Ian Pratt @ 2004-09-28 21:23 UTC (permalink / raw)
To: Michael Vrable; +Cc: xen-devel, Ian.Pratt
> On Tue, Sep 28, 2004 at 09:06:59PM +0100, Ian Pratt wrote:
> > It might be worth adding "| __GFP_REPEAT" to the alloc_page in
> > drivers/md/kcopyd.c
>
> I think that __GFP_REPEAT is a no-op for single-page allocations, as in
> this case (though I haven't tried it). __GFP_NOFAIL might work, but
> that sounds like a cure worse than the disease.
Yep, you're right. From mm/page_alloc.c:
if (!(gfp_mask & __GFP_NORETRY)) {
if ((order <= 3) || (gfp_mask & __GFP_REPEAT))
do_retry = 1;
if (gfp_mask & __GFP_NOFAIL)
do_retry = 1;
}
if (do_retry) {
blk_congestion_wait(WRITE, HZ/50);
I think it's worth trying GFP_NOFAIL just to see what happens.
The correct fix is probably to wrap the page_alloc in a loop that
retries a few times, maybe something like:
unsigned long start = jiffies;
while( (pl->page = alloc_page(GFP_KERNEL)) == NULL &&
jiffies - start < 5*HZ )
blk_congestion_wait(WRITE, HZ/5);
Ian
-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Automation scripts
2004-09-27 21:50 ` Paul Dorman
2004-09-27 22:08 ` Ian Pratt
2004-09-27 22:18 ` Michael Vrable
@ 2004-09-28 23:59 ` Mark A. Williamson
2 siblings, 0 replies; 26+ messages in thread
From: Mark A. Williamson @ 2004-09-28 23:59 UTC (permalink / raw)
To: Paul Dorman; +Cc: xen-devel
[-- Attachment #1: Type: text/plain, Size: 2618 bytes --]
> I would certainly appreciated this. Then I can keep my automation roughly
> in line with what's going to happen anyway, so I can move from my system to
> the official one without too much pain. I'm certain the official one will
> be much better anyhow :o)
OK, here's some thoughts that clarify the existing control structure and what
a cluster controller ought to do. I attach an old design doc from before the
current set of control tools, which I've updated to correspond better to
reality.
The crux of the cluster controller is that it really consists of two parts:
* Stuff to manage the cluster resources in a uniform way (e.g. migration,
start / stop domains anywhere in the cluster, monitor domains... + the
console concentrator)
* Stuff to manage VMs (image templates, etc) more effectively.
These are pretty independent. We imagine all state being stored in an SQL
server. Hopefully, the SQL server can provide us replication of this data,
etc.
On top of this, command line (and at some stage) web-based interfaces would be
required to run the show.
A lot of the code shouldn't be too technically difficult, just time-consuming.
The SQL database will need to be carefully designed. We would probably tend
to implement daemon code in Python, like all the rest of the tools.
We would love to see contributions in this area.
HTH,
Mark
> Another little query about the LVM stuff: I'm using kernel 2.6 so does this
> rule out the LVM option? The package note for Sarge's LVM2 says that it
> works with 2.4 only.
>
> Thanks Mark.
>
> Regards,
> Paul
>
> On Tuesday 28 September 2004 09:46 am, Mark A. Williamson wrote:
> > > I wonder if anyone on the list has written any scripts to automate the
> > > management of VMs with loopback images. Here's what I want to be able
> > > to do:
> >
> > Managing loopback block devices (and other non-physical block devices)
> > will get friendlier than it currently is. They'll get automatically
> > allocated, deallocated etc by Xend.
> >
> > However, the functionality you want is much like we'd envisaged for the
> > "cluster controller" some time in the future. The idea behind it is to
> > simplify the management of multiple Xen machines as a single pool of
> > resources.
> >
> > We have some preliminary design documents on this but no implementation
> > as yet. There are other people working on their own cluster management
> > schemes (hi Brian, hi Steve ;-) but there's not a general-purpose Xen
> > package for doing this.
> >
> > If you're interested, we can post some of our design docs on this
> > subject.
> >
> > Cheers,
> > Mark
[-- Attachment #2: xen-ctl-plane.txt --]
[-- Type: text/plain, Size: 4018 bytes --]
Xen Cluster Control Tools
=================================
by Mark Williamson
(C) 2004 Intel Research Cambridge
This document gives an overview of the functionality currently
available and discusses some of the issues involved in the design of a
Xen cluster controller.
Decomposition of functionality
------------------------------
Logically, the control software required for managing Xen servers can
be divided into three components:
Basic control functions:
All Xen nodes, whether they are standalone or part of a cluster will
need a set of basic control functions to be performed locally.
- (de)multiplex console messages
- (de)multiplex other control messages
- any other low-level management functions required
- supervising operations associated with domain migration
- domain reboots
- startup & shutdown domains on request
- suspend / resume domains given a location on disk for the state file
- coredump / reap domains that exit with an error code
Standalone node management:
Standalone Xen servers require some higher-level management
functionality. This is currently provided by a web interface and by
the xm command-line tool.
This software does:
- automate startup & shutdown domains when the main system starts /
shuts down
- manage domain memory and disk images within one machine to provide
an abstraction of a persistent "virtual machine" that the user can
assign some name to
- recovers from restart, so Xend crashes shouldn't necessarily be fatal to
domains running on the host
It would be nice to have:
- provide some means to move, copy or remotely access the local disk
state, etc of a "virtual machine" to another machine, either whilst
suspended or whilst running, when requested by the user, as well as
transferring things like memory images using the functionality
provided by the basic control layer.
This is to facilitate migration of domains that have state on the local
disk and could be requested directly by the user, or by the cluster
controller.
Cluster management:
This software would be used to manage a cluster of heterogeneous Xen
services in a uniform way.
This software should ideally provide:
- global awareness of the cluster (where virtual machines are running, what
resources are available in the cluster...)
* resource awareness could also include - free IPs, free MAC addresses
- intelligent application of migration to facilitate load balancing,
taking down physical hosts for maintenance, etc.
- some means for detecting and recovering from failed cluster nodes. This
could sensibly be achieved by polling all running Xends for their current
state and noting failures to respond. Once a node has failed a few times,
the cluster controller would take corrective action or report an error.
- recovery from cluster controller failure, possibly using redundant
controller nodes
- provide a separate layer of indirection for domain consoles so that migration
can occur transparently without breaking console TCP connections
- a concept of a virtual machine class and a virtual machine instance
(i.e. "Fedora dynamic web server" vs. "Fred's server"). Instances can be
created based on classes.
The last point makes virtual machine management nicer but could be
quite complex in itself. It is also largely orthogonal to the other
points, which are concerned with managing already-defined virtual
machines.
Thoughts on the implementation
------------------------------
The cluster management software will likely utilise a full-featured
database server to maintain global state for the cluster. A control
daemon would run on the cluster's "master" machine and would monitor
the cluster and issue commands to nodes on behalf of the user.
The cluster management software in particular is a very complex
problem and so its feature set should be carefully chosen for the
initial implementation in order to ensure that a product is eventually
completed! Not all of the above is required for it to be useful.
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: LVM Snapshot Troubles
2004-09-28 21:23 ` Ian Pratt
@ 2004-09-29 8:39 ` Keir Fraser
2004-09-30 21:02 ` Michael Vrable
1 sibling, 0 replies; 26+ messages in thread
From: Keir Fraser @ 2004-09-29 8:39 UTC (permalink / raw)
To: Ian Pratt; +Cc: Michael Vrable, xen-devel
> I think it's worth trying GFP_NOFAIL just to see what happens.
> The correct fix is probably to wrap the page_alloc in a loop that
> retries a few times, maybe something like:
>
> unsigned long start = jiffies;
>
> while( (pl->page = alloc_page(GFP_KERNEL)) == NULL &&
> jiffies - start < 5*HZ )
> blk_congestion_wait(WRITE, HZ/5);
It's likely not helped by the fact the network driver has big
allocation spikes periodically when it refills the packet receive
ring. I'll check in something to smooth out those allocations...
-- Keir
-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Automation scripts
2004-09-28 14:21 ` Brian Wolfe
@ 2004-09-29 10:36 ` Christian Limpach
2004-09-29 19:47 ` Paul Dorman
0 siblings, 1 reply; 26+ messages in thread
From: Christian Limpach @ 2004-09-29 10:36 UTC (permalink / raw)
To: Brian Wolfe; +Cc: mark.williamson, xen-devel, Paul Dorman
On Tue, Sep 28, 2004 at 09:21:12AM -0500, Brian Wolfe wrote:
> I'm far from having a working automated system right now. I still have
> the iSCSI death issues to deal with when combining 2 iSCSI target
> devices into a raid-1 array in Domain-0 to be exported to the XenLinux
> unprived domain. 8-( NFS roots still periodically "lock up" momentarilly
> (~2 to 10 seconds) when using nfs root. SO the solution isn't ready yet.
> I do highly reccomend that people use LVM LVs for individual exportable
> block devices and to forget about using COWs to "save space". Adding a
> 200GB disk is dirt cheap (unless you are talking about scsi) so it's not
> worth the savings IMHO to multi-cow a filesystem into 2+ unprived
> domains on a single machine. The 1 to 2 GB of saved space just isn't
> worth it at $0.10 per GB of disk space. :) Add in the system overhead
> and seek times required for COWs after your first FS upgrade and your
> return is worse than using separate partitions/LVs in my experience.
I think the best use one could get from LVM snapshots is background
cloning of filesystems, i.e. you'd use a modified snapshot driver
which would slowly create a 1-to-1 copy of the whole partition but
allow it to be used immediately, copying being done when the
disk/machine is otherwise idle.
christian
-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Automation scripts
2004-09-29 10:36 ` Christian Limpach
@ 2004-09-29 19:47 ` Paul Dorman
2004-09-29 20:23 ` Ian Pratt
0 siblings, 1 reply; 26+ messages in thread
From: Paul Dorman @ 2004-09-29 19:47 UTC (permalink / raw)
To: xen-devel
I like this suggestion. Instead of having a bunch of templates only, I could
also have standby partitions, ready to start immediately. Storage *is* cheap,
so the cost is low for this.
Paul
On Wednesday 29 September 2004 10:36 pm, Christian Limpach wrote:
> On Tue, Sep 28, 2004 at 09:21:12AM -0500, Brian Wolfe wrote:
> > I'm far from having a working automated system right now. I still have
> > the iSCSI death issues to deal with when combining 2 iSCSI target
> > devices into a raid-1 array in Domain-0 to be exported to the XenLinux
> > unprived domain. 8-( NFS roots still periodically "lock up" momentarilly
> > (~2 to 10 seconds) when using nfs root. SO the solution isn't ready yet.
> > I do highly reccomend that people use LVM LVs for individual exportable
> > block devices and to forget about using COWs to "save space". Adding a
> > 200GB disk is dirt cheap (unless you are talking about scsi) so it's not
> > worth the savings IMHO to multi-cow a filesystem into 2+ unprived
> > domains on a single machine. The 1 to 2 GB of saved space just isn't
> > worth it at $0.10 per GB of disk space. :) Add in the system overhead
> > and seek times required for COWs after your first FS upgrade and your
> > return is worse than using separate partitions/LVs in my experience.
>
> I think the best use one could get from LVM snapshots is background
> cloning of filesystems, i.e. you'd use a modified snapshot driver
> which would slowly create a 1-to-1 copy of the whole partition but
> allow it to be used immediately, copying being done when the
> disk/machine is otherwise idle.
>
> christian
-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Automation scripts
2004-09-29 19:47 ` Paul Dorman
@ 2004-09-29 20:23 ` Ian Pratt
0 siblings, 0 replies; 26+ messages in thread
From: Ian Pratt @ 2004-09-29 20:23 UTC (permalink / raw)
To: Paul Dorman; +Cc: xen-devel, Ian.Pratt
> I like this suggestion. Instead of having a bunch of templates only, I could
> also have standby partitions, ready to start immediately. Storage *is* cheap,
> so the cost is low for this.
I expect it wouldn't be too hard to modify the dm-snap driver to
do this. I think we'd end up with something rather more robust,
as the current snapshot mechanism is very vulnerable to disk
errors and other corruption.
In any event, it snap-dm should probably be modified such that it
shares a kcopyd memory pool among all the active snapshots rather
than allocating 1MB for each.
It's a pity that the snapshot exception table is entirely memory
resident, stored as a hash table. Turning it into an on-disk
tree cached in memory would be better.
All things for the todo list...
Ian
> Paul
>
> On Wednesday 29 September 2004 10:36 pm, Christian Limpach wrote:
> > On Tue, Sep 28, 2004 at 09:21:12AM -0500, Brian Wolfe wrote:
> > > I'm far from having a working automated system right now. I still have
> > > the iSCSI death issues to deal with when combining 2 iSCSI target
> > > devices into a raid-1 array in Domain-0 to be exported to the XenLinux
> > > unprived domain. 8-( NFS roots still periodically "lock up" momentarilly
> > > (~2 to 10 seconds) when using nfs root. SO the solution isn't ready yet.
> > > I do highly reccomend that people use LVM LVs for individual exportable
> > > block devices and to forget about using COWs to "save space". Adding a
> > > 200GB disk is dirt cheap (unless you are talking about scsi) so it's not
> > > worth the savings IMHO to multi-cow a filesystem into 2+ unprived
> > > domains on a single machine. The 1 to 2 GB of saved space just isn't
> > > worth it at $0.10 per GB of disk space. :) Add in the system overhead
> > > and seek times required for COWs after your first FS upgrade and your
> > > return is worse than using separate partitions/LVs in my experience.
> >
> > I think the best use one could get from LVM snapshots is background
> > cloning of filesystems, i.e. you'd use a modified snapshot driver
> > which would slowly create a 1-to-1 copy of the whole partition but
> > allow it to be used immediately, copying being done when the
> > disk/machine is otherwise idle.
> >
> > christian
>
>
> -------------------------------------------------------
> This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
> Use IT products in your business? Tell us what you think of them. Give us
> Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
> http://productguide.itmanagersjournal.com/guidepromo.tmpl
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel
-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
^ permalink raw reply [flat|nested] 26+ messages in thread
* Loop and ENBD device management (was Re: Automation scripts)
2004-09-27 19:23 Automation scripts Paul Dorman
2004-09-27 20:26 ` Paul Dorman
2004-09-27 21:46 ` Mark A. Williamson
@ 2004-09-30 20:58 ` Mark A. Williamson
2004-09-30 21:00 ` Mark A. Williamson
3 siblings, 0 replies; 26+ messages in thread
From: Mark A. Williamson @ 2004-09-30 20:58 UTC (permalink / raw)
To: xen-devel; +Cc: Paul Dorman
Initial support for Xend to automatically bind loop and ENBD devices is now in
the unstable tree.
To use a file for a domain's disk the syntax is:
'file:/path/to/file,target_dev,mode'
For ENBD (Disclaimer! The scripts for ENBD are untested and probably won't
work yet. If someone could test this on a non-production system that'd be
good):
'enbd:host:port,target_dev,mode'
Binding and unbinding these devices to local device nodes is done in the
scripts /etc/xen/{block-file,block-enbd}.
You can edit those scripts to change their behaviour. To add support for a
device of type "foo" you can just add a script /etc/xen/block-foo and a
config item in the Xend config.
This should already be useful, at least for file-based disks. It'd be nice to
have the ENBD script working and have some kind of iSCSI script at some
stage.
Cheers,
Mark
On Monday 27 September 2004 19:23, Paul Dorman wrote:
> Hi all,
>
> I wonder if anyone on the list has written any scripts to automate the
> management of VMs with loopback images. Here's what I want to be able to
> do:
>
> * Store existing physical machine file systems, or pristine installs in
> loopback images on my Xen servers (something I'll do manually)
>
> * Run a script that will start a VM from one of these images, automatically
> associate it with a loopback device, give it a name, RAM allocation,
> network addresses, and set various internal parameters, such as hostname,
> routes, etc., based on a set of arguments. So something like "script
> <imagename> <hostname> <netconfigs> <RAM> <.. etc.
>
> * Have the same script take another argument that will cause it to clone a
> filesystem image first before starting the VM, so that I can use a set of
> images as VM templates. I intend to have a large collection of templates
> which my developers can use to create VMs suited to whatever project they
> are working on.
>
> * After a VM machine has been instantiated, I would like to be able to
> start and stop it with simple "start hostname" and "stop hostname" kinds of
> commands.
>
> * Have management tools so that I can for example shift a VM from one Xen
> server to another (shift hostname xenservername). These would also be used
> by load balancing scripts to shift machines around to manage resources.
>
> I'd like to build a web-based management system for these scripts, so that
> developers are free to create and control Xen VMs (though naturally with
> limitations based on what the servers can handle -- so my bosses will know
> when they need to buy me more servers :o) ).
>
> I don't see these as particularly difficult, but if someone has done them
> already .... Also, I'd appreciate any thoughts you might have on automation
> of this kind, particularly in terms of functionality and practicalities.
>
> Thanks for your time!
>
> Paul
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
> Project Admins to receive an Apple iPod Mini FREE for your judgement on
> who ports your project to Linux PPC the best. Sponsored by IBM.
> Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel
-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
^ permalink raw reply [flat|nested] 26+ messages in thread
* Loop and ENBD device management (was Re: Automation scripts)
2004-09-27 19:23 Automation scripts Paul Dorman
` (2 preceding siblings ...)
2004-09-30 20:58 ` Loop and ENBD device management (was Re: Automation scripts) Mark A. Williamson
@ 2004-09-30 21:00 ` Mark A. Williamson
3 siblings, 0 replies; 26+ messages in thread
From: Mark A. Williamson @ 2004-09-30 21:00 UTC (permalink / raw)
To: xen-devel; +Cc: Paul Dorman
Initial support for Xend to automatically bind loop and ENBD devices is now in
the unstable tree.
To use a file for a domain's disk the syntax is:
'file:/path/to/file,target_dev,mode'
For ENBD (Disclaimer! The scripts for ENBD are untested and probably won't
work yet. Could someone please test this on a non-production system?)
'enbd:host:port,target_dev,mode'
Binding and unbinding these devices to local device nodes is done in the
scripts /etc/xen/{block-file,block-enbd}. You can edit those scripts to
change their behaviour.
To add support for a device of type "foo" you can just add a
script /etc/xen/block-foo and an appropriate config entry.
Cheers,
Mark
On Monday 27 September 2004 19:23, Paul Dorman wrote:
> Hi all,
>
> I wonder if anyone on the list has written any scripts to automate the
> management of VMs with loopback images. Here's what I want to be able to
> do:
>
> * Store existing physical machine file systems, or pristine installs in
> loopback images on my Xen servers (something I'll do manually)
>
> * Run a script that will start a VM from one of these images, automatically
> associate it with a loopback device, give it a name, RAM allocation,
> network addresses, and set various internal parameters, such as hostname,
> routes, etc., based on a set of arguments. So something like "script
> <imagename> <hostname> <netconfigs> <RAM> <.. etc.
>
> * Have the same script take another argument that will cause it to clone a
> filesystem image first before starting the VM, so that I can use a set of
> images as VM templates. I intend to have a large collection of templates
> which my developers can use to create VMs suited to whatever project they
> are working on.
>
> * After a VM machine has been instantiated, I would like to be able to
> start and stop it with simple "start hostname" and "stop hostname" kinds of
> commands.
>
> * Have management tools so that I can for example shift a VM from one Xen
> server to another (shift hostname xenservername). These would also be used
> by load balancing scripts to shift machines around to manage resources.
>
> I'd like to build a web-based management system for these scripts, so that
> developers are free to create and control Xen VMs (though naturally with
> limitations based on what the servers can handle -- so my bosses will know
> when they need to buy me more servers :o) ).
>
> I don't see these as particularly difficult, but if someone has done them
> already .... Also, I'd appreciate any thoughts you might have on automation
> of this kind, particularly in terms of functionality and practicalities.
>
> Thanks for your time!
>
> Paul
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
> Project Admins to receive an Apple iPod Mini FREE for your judgement on
> who ports your project to Linux PPC the best. Sponsored by IBM.
> Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel
-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: LVM Snapshot Troubles
2004-09-28 21:23 ` Ian Pratt
2004-09-29 8:39 ` Keir Fraser
@ 2004-09-30 21:02 ` Michael Vrable
1 sibling, 0 replies; 26+ messages in thread
From: Michael Vrable @ 2004-09-30 21:02 UTC (permalink / raw)
To: xen-devel
I've been looking into the LVM snapshot/memory allocation troubles and
will try to come up with a fix.
FYI: In doing more searching for information about the problem, I did
come across this:
http://www.redhat.com/archives/dm-devel/2004-January/msg00068.html
"The problem seems to be that dm-ioctl-v4.c sets the PF_MEMALLOC
flag for the current process.
Lookaing at the memory allocator (__alloc_page) this means that the
VM will think the memory allocation is already running (and this is
a recursion) so it will not try to free pages / rebalance page or
whatever." ...
I'm looking into sharing memory between the snapshots instead of giving
each snapshot its own private allocation of pages for I/O. (As I'd like
to scale to a large number of snapshots, and don't want to need >1 MB of
kernel memory per snapshot.)
--Michael Vrable
-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
^ permalink raw reply [flat|nested] 26+ messages in thread
end of thread, other threads:[~2004-09-30 21:02 UTC | newest]
Thread overview: 26+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2004-09-27 19:23 Automation scripts Paul Dorman
2004-09-27 20:26 ` Paul Dorman
2004-09-27 21:53 ` Mark A. Williamson
2004-09-28 14:21 ` Brian Wolfe
2004-09-29 10:36 ` Christian Limpach
2004-09-29 19:47 ` Paul Dorman
2004-09-29 20:23 ` Ian Pratt
2004-09-27 21:46 ` Mark A. Williamson
2004-09-27 21:50 ` Paul Dorman
2004-09-27 22:08 ` Ian Pratt
2004-09-27 22:18 ` Michael Vrable
2004-09-27 22:43 ` Paul Dorman
2004-09-28 18:41 ` Michael Vrable
2004-09-28 8:23 ` Peri Hankey
2004-09-28 12:26 ` Ian Pratt
2004-09-28 14:55 ` Peri Hankey
2004-09-28 15:43 ` Ian Pratt
2004-09-28 18:09 ` LVM Snapshot Troubles Michael Vrable
2004-09-28 20:06 ` Ian Pratt
2004-09-28 20:52 ` Michael Vrable
2004-09-28 21:23 ` Ian Pratt
2004-09-29 8:39 ` Keir Fraser
2004-09-30 21:02 ` Michael Vrable
2004-09-28 23:59 ` Automation scripts Mark A. Williamson
2004-09-30 20:58 ` Loop and ENBD device management (was Re: Automation scripts) Mark A. Williamson
2004-09-30 21:00 ` Mark A. Williamson
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.