Re: XEN - networking and performance

Re: XEN - networking and performance

[D. Duckworth]

Salutations,

From the xen-users list:
> I would like some advice from people how are/were using  Xen 3.4.2 -
> it should be a rather stable release. Dom0 is CentOS 5.5 64bit with
> Xen 3.4.2 installed with the default settings (minus cpu pinning to
> Dom0 and memory setup for 2GB). There are 2 built-in nics (Broadcom)
> and an add-on network card (Intel) with another 4 nics. Currently
> only one NIC is used for all network access, and as far as
> networking, the default setings are used - xend-config.sxp:
> (network-script network-bridge) (vif-script vif-bridge)
> 
> The questions are:
> 
> How can I improve the network performance (now all the VM are sharing
> one bridge):
> 
> a. creating multiple bridges and assigning a VM (DomU) per bridge
> b. trying to hide the NICs from Dom0 using something like "pciback
> hide" - (pointers/example of how one would do this in Centos 5.5
> would be highly appreciated...)

Xen Networking has been a thorn in my eye and a similar question has
been with me for a long time now. So prepare, for this response contains
rage.

Xen networking has room for many different approaches, yet the best
thing about its scripts is that they are not mandatory to use. You can
fully adjust the scripts to your needs or even replace them with your
own. You can find modifications on forums and blogs although most of
them just seem to be copies of few suggestions made by few people.
Logical, because quite frankly it's a pain to grope what the Xen
scripts and udev rules really do, let alone grok most of what they
do out of the box.

Right now I just care about creating my ideal networking solution, i.e.
routing, bridging and firewall stuff for vms with different roles. I am
running Xen 4.1.2-rc3-pre non-professionally on a quad core single cpu
1U server. with 4 hard drives in RAID10 configuration. The server has
one usable Ethernet port with multiple globally routable IPs. I can't
use the other ethernet port; the server has no IPMI and the ISP declines
use of two ports by one system because the data center is a no smoking
zone for both humans and routers and switches.

So the highest priority is to reach dom0 from the Internet and
therefore my grub has fallback options, one of which is a boot to Linux
with no Xen. In turn this means that dom0's networking boot scripts
may not depend on Xen at all, and Xen may not change networking in any
way unless specified. My dom0 is a minimal system that only controls vms
and networking. I want dom0 to be small and simple so the obvious
choice is Arch Linux. Dom0 should be separated from the domUs in that
the domUs cannot reach dom0 and one domU (domN) should do all
networking for the other domUs.

I tried to use xl with xen4 for a while but due to bugs and missing
features I had to go back to xm and xend. This is where the fun
begins. In the past I used xend with network-bridge and for some strange
reason (voodoo probably) I blindly accepted that script in the past and
blamed myself for not appreciating it. But let's be blunt and honest:
the scripts, in particular the script that *modifies dom0 networking
during xend startup* is the biggest piece of sh!# idea I have ever seen
in Xen. It creates bridges, takes eth0 down, tortures dom0 with occult
ip addr/route, brctl, sysctl and iptables awk/sed manipulations and then
it has you looking at your screen yearning for the moment that ping
timeouts become ping replies, telling you that your box is reachable
again. This script is a malevolent demon from the sewers of Norman the
Golgothan and the worst part is that network-bridge is also still the
recommended default!

On the more positive side there was a fantastic update in Xen 4 where
network-bridge changed a bit so that "it will not bring a bridge up if
one already exists". Whoever wrote it should get a corporate medal for
that and then a long vacation to a deserted island with an MSX II and
no floppies. How can this even be approved by Xen's senior project
manager, or is that a vacant position?

It surely seems so. Xen's /etc/xen/scripts (another design fail, why
not /usr?) and udev scripts are confusing ad-hoc bloatware routines and
are not transparent at all. With the current xen4 I saw the premature
advice to more or less 'prepare for migration from xm to xl'. Yet, xl
supports less and is conflicting: there is no vifname, no 'xl
new/delete', no more python, no relocation and suddenly there is a
conflict between 'xm start domain' versus 'xl start /etc/xen/domain'.

So new features emerge, adding to the confusion of the end user, while
old problems are not being fixed properly. I wonder why, especially
because it does not seem that xm and xend are the broken parts that
need to be replaced by an unstable interface.

What needs attention first and foremost are two things, first of which
is real and wise effort into one simple, minimal script that just
handles the minimum in a transparent way e.g. control the hypervisor,
manage vms, manage the backend. Of course networking can be done on
domain start too, but this has happen in an entirely different way from
what it does and how it does it. This is so important because it gives
more control to the user that runs Xen. It's also a good moment to
build in proper and mature support for IPv6.

Secondly, the website and documentation should be cleaned up and
revised where appropriate. The current situation is a mess that has
a much too steep and incompatible learning curve right now - for
example, a bridge should just not be named eth0 and a physical device
should not be renamed at all. It's fundamentally wrong, stupid, mad as
hell and a PR failure for Xen to do it this way out of the box. No
matter how often and detailed it has been documented on the website. 

I propose something like the following for xen networking:

* Xen will not manipulate non-xen devices or a firewall under any
  circumstance, it might only add or substract routes and/or rules from
  the routing tables,
* Allow for networking configuration per domU. For example let
  networking per device be nat, routed, bridged or custom, where
  all name the interface and bring it up; nat only adds the ip to the
  routing table; routed could be an array of routes and rules that need
  to be added or subtracted from various routing tables and it might
  support proxyarp; bridged turns off arp, sets the mac on the vif and
  then adds the interface to a bridge that should already be created by
  the user; and custom is a custom set of unmanaged commands after
  creating and destroying a domain.

I am aware that this can already be done with Xen. However, that
process is quite arbitrary and it does things no one asked for. So one
has to read the scripts. For example with the iptables part of
vif-bridge. It is not handled transparently, it is quite arbitrary and
it automatically executes for all vms that are being started. This leads
you to wonder what more it does without you knowing it...

So, with that off my chest and the second line of my network-bridge
being the words "exit 0" Xen lets my dom0 configuration alone
like it is supposed to do. While KVM is becoming a 'next cool
thing' for many people I would still prefer a separate hypervisor so now
the fat just has to be removed from Xen.

RE: [Xen-users] XEN - networking and performance

[James Harper]

> the scripts, in particular the script that *modifies dom0 networking
during
> xend startup* is the biggest piece of sh!# idea I have ever seen in
Xen. It
> creates bridges, takes eth0 down, tortures dom0 with occult ip
addr/route,
> brctl, sysctl and iptables awk/sed manipulations and then it has you
looking at
> your screen yearning for the moment that ping timeouts become ping
> replies, telling you that your box is reachable again. This script is
a malevolent
> demon from the sewers of Norman the Golgothan and the worst part is
that
> network-bridge is also still the recommended default!

That script was designed to make the network look like it was before you
installed xen. Anyone with anything beyond a basic single-port
single-vlan setup comments that line out and creates their own bridges.

> It surely seems so. Xen's /etc/xen/scripts (another design fail, why
not /usr?)
> and udev scripts are confusing ad-hoc bloatware routines and are not
> transparent at all. With the current xen4 I saw the premature advice
to more
> or less 'prepare for migration from xm to xl'. Yet, xl supports less
and is
> conflicting: there is no vifname, no 'xl new/delete', no more python,
no
> relocation and suddenly there is a conflict between 'xm start domain'
versus
> 'xl start /etc/xen/domain'.
> 

This is an open source project. Please feel free to submit your own
scripts. I'd definitely like to see something that didn't create
firewall rules as I don't even want iptables loaded on my xen systems.

James

Re: [Xen-users] XEN - networking and performance

[D. Duckworth]

Salutations,

From the xen-users list:
> I would like some advice from people how are/were using  Xen 3.4.2 -
> it should be a rather stable release. Dom0 is CentOS 5.5 64bit with
> Xen 3.4.2 installed with the default settings (minus cpu pinning to
> Dom0 and memory setup for 2GB). There are 2 built-in nics (Broadcom)
> and an add-on network card (Intel) with another 4 nics. Currently
> only one NIC is used for all network access, and as far as
> networking, the default setings are used - xend-config.sxp:
> (network-script network-bridge) (vif-script vif-bridge)
> 
> The questions are:
> 
> How can I improve the network performance (now all the VM are sharing
> one bridge):
> 
> a. creating multiple bridges and assigning a VM (DomU) per bridge
> b. trying to hide the NICs from Dom0 using something like "pciback
> hide" - (pointers/example of how one would do this in Centos 5.5
> would be highly appreciated...)

Xen Networking has been a thorn in my eye and a similar question has
been with me for a long time now. So prepare, for this response contains
rage.

Xen networking has room for many different approaches, yet the best
thing about its scripts is that they are not mandatory to use. You can
fully adjust the scripts to your needs or even replace them with your
own. You can find modifications on forums and blogs although most of
them just seem to be copies of few suggestions made by few people.
Logical, because quite frankly it's a pain to grope what the Xen
scripts and udev rules really do, let alone grok most of what they
do out of the box.

Right now I just care about creating my ideal networking solution, i.e.
routing, bridging and firewall stuff for vms with different roles. I am
running Xen 4.1.2-rc3-pre non-professionally on a quad core single cpu
1U server. with 4 hard drives in RAID10 configuration. The server has
one usable Ethernet port with multiple globally routable IPs. I can't
use the other ethernet port; the server has no IPMI and the ISP declines
use of two ports by one system because the data center is a no smoking
zone for both humans and routers and switches.

So the highest priority is to reach dom0 from the Internet and
therefore my grub has fallback options, one of which is a boot to Linux
with no Xen. In turn this means that dom0's networking boot scripts
may not depend on Xen at all, and Xen may not change networking in any
way unless specified. My dom0 is a minimal system that only controls vms
and networking. I want dom0 to be small and simple so the obvious
choice is Arch Linux. Dom0 should be separated from the domUs in that
the domUs cannot reach dom0 and one domU (domN) should do all
networking for the other domUs.

I tried to use xl with xen4 for a while but due to bugs and missing
features I had to go back to xm and xend. This is where the fun
begins. In the past I used xend with network-bridge and for some strange
reason (voodoo probably) I blindly accepted that script in the past and
blamed myself for not appreciating it. But let's be blunt and honest:
the scripts, in particular the script that *modifies dom0 networking
during xend startup* is the biggest piece of sh!# idea I have ever seen
in Xen. It creates bridges, takes eth0 down, tortures dom0 with occult
ip addr/route, brctl, sysctl and iptables awk/sed manipulations and then
it has you looking at your screen yearning for the moment that ping
timeouts become ping replies, telling you that your box is reachable
again. This script is a malevolent demon from the sewers of Norman the
Golgothan and the worst part is that network-bridge is also still the
recommended default!

On the more positive side there was a fantastic update in Xen 4 where
network-bridge changed a bit so that "it will not bring a bridge up if
one already exists". Whoever wrote it should get a corporate medal for
that and then a long vacation to a deserted island with an MSX II and
no floppies. How can this even be approved by Xen's senior project
manager, or is that a vacant position?

It surely seems so. Xen's /etc/xen/scripts (another design fail, why
not /usr?) and udev scripts are confusing ad-hoc bloatware routines and
are not transparent at all. With the current xen4 I saw the premature
advice to more or less 'prepare for migration from xm to xl'. Yet, xl
supports less and is conflicting: there is no vifname, no 'xl
new/delete', no more python, no relocation and suddenly there is a
conflict between 'xm start domain' versus 'xl start /etc/xen/domain'.

So new features emerge, adding to the confusion of the end user, while
old problems are not being fixed properly. I wonder why, especially
because it does not seem that xm and xend are the broken parts that
need to be replaced by an unstable interface.

What needs attention first and foremost are two things, first of which
is real and wise effort into one simple, minimal script that just
handles the minimum in a transparent way e.g. control the hypervisor,
manage vms, manage the backend. Of course networking can be done on
domain start too, but this has happen in an entirely different way from
what it does and how it does it. This is so important because it gives
more control to the user that runs Xen. It's also a good moment to
build in proper and mature support for IPv6.

Secondly, the website and documentation should be cleaned up and
revised where appropriate. The current situation is a mess that has
a much too steep and incompatible learning curve right now - for
example, a bridge should just not be named eth0 and a physical device
should not be renamed at all. It's fundamentally wrong, stupid, mad as
hell and a PR failure for Xen to do it this way out of the box. No
matter how often and detailed it has been documented on the website. 

I propose something like the following for xen networking:

* Xen will not manipulate non-xen devices or a firewall under any
  circumstance, it might only add or substract routes and/or rules from
  the routing tables,
* Allow for networking configuration per domU. For example let
  networking per device be nat, routed, bridged or custom, where
  all name the interface and bring it up; nat only adds the ip to the
  routing table; routed could be an array of routes and rules that need
  to be added or subtracted from various routing tables and it might
  support proxyarp; bridged turns off arp, sets the mac on the vif and
  then adds the interface to a bridge that should already be created by
  the user; and custom is a custom set of unmanaged commands after
  creating and destroying a domain.

I am aware that this can already be done with Xen. However, that
process is quite arbitrary and it does things no one asked for. So one
has to read the scripts. For example with the iptables part of
vif-bridge. It is not handled transparently, it is quite arbitrary and
it automatically executes for all vms that are being started. This leads
you to wonder what more it does without you knowing it...

So, with that off my chest and the second line of my network-bridge
being the words "exit 0" Xen lets my dom0 configuration alone
like it is supposed to do. While KVM is becoming a 'next cool
thing' for many people I would still prefer a separate hypervisor so now
the fat just has to be removed from Xen.

Re: Re: [Xen-users] XEN - networking and performance

[Konrad Rzeszutek Wilk]

On Sat, Oct 08, 2011 at 07:07:01PM +0000, D. Duckworth wrote:
> Salutations,
> 
> From the xen-users list:
> > I would like some advice from people how are/were using  Xen 3.4.2 -
> > it should be a rather stable release. Dom0 is CentOS 5.5 64bit with
> > Xen 3.4.2 installed with the default settings (minus cpu pinning to
> > Dom0 and memory setup for 2GB). There are 2 built-in nics (Broadcom)
> > and an add-on network card (Intel) with another 4 nics. Currently
> > only one NIC is used for all network access, and as far as
> > networking, the default setings are used - xend-config.sxp:
> > (network-script network-bridge) (vif-script vif-bridge)
> > 
> > The questions are:
> > 
> > How can I improve the network performance (now all the VM are sharing
> > one bridge):
> > 
> > a. creating multiple bridges and assigning a VM (DomU) per bridge
> > b. trying to hide the NICs from Dom0 using something like "pciback
> > hide" - (pointers/example of how one would do this in Centos 5.5
> > would be highly appreciated...)
> 
> Xen Networking has been a thorn in my eye and a similar question has
> been with me for a long time now. So prepare, for this response contains
> rage.
> 
> Xen networking has room for many different approaches, yet the best
> thing about its scripts is that they are not mandatory to use. You can
> fully adjust the scripts to your needs or even replace them with your
> own. You can find modifications on forums and blogs although most of
> them just seem to be copies of few suggestions made by few people.
> Logical, because quite frankly it's a pain to grope what the Xen
> scripts and udev rules really do, let alone grok most of what they
> do out of the box.
> 
> Right now I just care about creating my ideal networking solution, i.e.
> routing, bridging and firewall stuff for vms with different roles. I am
> running Xen 4.1.2-rc3-pre non-professionally on a quad core single cpu
> 1U server. with 4 hard drives in RAID10 configuration. The server has
> one usable Ethernet port with multiple globally routable IPs. I can't
> use the other ethernet port; the server has no IPMI and the ISP declines
> use of two ports by one system because the data center is a no smoking
> zone for both humans and routers and switches.
> 
> So the highest priority is to reach dom0 from the Internet and
> therefore my grub has fallback options, one of which is a boot to Linux
> with no Xen. In turn this means that dom0's networking boot scripts
> may not depend on Xen at all, and Xen may not change networking in any
> way unless specified. My dom0 is a minimal system that only controls vms
> and networking. I want dom0 to be small and simple so the obvious
> choice is Arch Linux. Dom0 should be separated from the domUs in that
> the domUs cannot reach dom0 and one domU (domN) should do all
> networking for the other domUs.
> 
> I tried to use xl with xen4 for a while but due to bugs and missing
> features I had to go back to xm and xend. This is where the fun
> begins. In the past I used xend with network-bridge and for some strange
> reason (voodoo probably) I blindly accepted that script in the past and
> blamed myself for not appreciating it. But let's be blunt and honest:
> the scripts, in particular the script that *modifies dom0 networking
> during xend startup* is the biggest piece of sh!# idea I have ever seen
> in Xen. It creates bridges, takes eth0 down, tortures dom0 with occult
> ip addr/route, brctl, sysctl and iptables awk/sed manipulations and then
> it has you looking at your screen yearning for the moment that ping
> timeouts become ping replies, telling you that your box is reachable
> again. This script is a malevolent demon from the sewers of Norman the
> Golgothan and the worst part is that network-bridge is also still the

<laughs>
> recommended default!

Can you point me to where it mentions that please?
> 
> On the more positive side there was a fantastic update in Xen 4 where
> network-bridge changed a bit so that "it will not bring a bridge up if
> one already exists". Whoever wrote it should get a corporate medal for
> that and then a long vacation to a deserted island with an MSX II and
> no floppies. How can this even be approved by Xen's senior project
> manager, or is that a vacant position?

We realized that the networking setup is quite complex and would be best
left in the hands of the admins. The problem is that..
> 
> It surely seems so. Xen's /etc/xen/scripts (another design fail, why
> not /usr?) and udev scripts are confusing ad-hoc bloatware routines and
> are not transparent at all. With the current xen4 I saw the premature
> advice to more or less 'prepare for migration from xm to xl'. Yet, xl
> supports less and is conflicting: there is no vifname, no 'xl
> new/delete', no more python, no relocation and suddenly there is a
> conflict between 'xm start domain' versus 'xl start /etc/xen/domain'.
> 
> So new features emerge, adding to the confusion of the end user, while
> old problems are not being fixed properly. I wonder why, especially
> because it does not seem that xm and xend are the broken parts that
> need to be replaced by an unstable interface.
> 
> What needs attention first and foremost are two things, first of which
> is real and wise effort into one simple, minimal script that just
> handles the minimum in a transparent way e.g. control the hypervisor,
> manage vms, manage the backend. Of course networking can be done on
> domain start too, but this has happen in an entirely different way from
> what it does and how it does it. This is so important because it gives
> more control to the user that runs Xen. It's also a good moment to
> build in proper and mature support for IPv6.
> 
> Secondly, the website and documentation should be cleaned up and
> revised where appropriate. The current situation is a mess that has
> a much too steep and incompatible learning curve right now - for
> example, a bridge should just not be named eth0 and a physical device
> should not be renamed at all. It's fundamentally wrong, stupid, mad as
> hell and a PR failure for Xen to do it this way out of the box. No
> matter how often and detailed it has been documented on the website. 

.. the documentation and setup is sometimes quite hard. BTW, we are
going to do on Oct 26th a Documentation Day to clean up some of this mess.
Would you be intereted in helping along - perhaps in the networking Wiki?
> 
> I propose something like the following for xen networking:
> 
> * Xen will not manipulate non-xen devices or a firewall under any
>   circumstance, it might only add or substract routes and/or rules from
>   the routing tables,

Uh, what is 'non-xen' devices? Like bridges?

> * Allow for networking configuration per domU. For example let
>   networking per device be nat, routed, bridged or custom, where
>   all name the interface and bring it up; nat only adds the ip to the
>   routing table; routed could be an array of routes and rules that need
>   to be added or subtracted from various routing tables and it might
>   support proxyarp; bridged turns off arp, sets the mac on the vif and
>   then adds the interface to a bridge that should already be created by
>   the user; and custom is a custom set of unmanaged commands after
>   creating and destroying a domain.

You lost me. <sigh> I am using a bridge configuration and just
do:

auto lo
iface lo inet loopback

auto switch
iface switch inet static
	address 192.168.101.16
	netmask 255.255.255.0
	gateway 192.168.101.1
	bridge_ports eth2

And just use that 'bridge=switch' in all my configuration. And that
seems to work just fine - wouldn't that be best way of providing
the first network setup to users? I would think the majority of folks
do something akin to this?

> 
> I am aware that this can already be done with Xen. However, that
> process is quite arbitrary and it does things no one asked for. So one
> has to read the scripts. For example with the iptables part of
> vif-bridge. It is not handled transparently, it is quite arbitrary and
> it automatically executes for all vms that are being started. This leads
> you to wonder what more it does without you knowing it...
> 
> So, with that off my chest and the second line of my network-bridge
> being the words "exit 0" Xen lets my dom0 configuration alone
> like it is supposed to do. While KVM is becoming a 'next cool
> thing' for many people I would still prefer a separate hypervisor so now
> the fat just has to be removed from Xen.
> 

I am all for removing fat. Do you have links to some of particularly
bad Wiki pages that should be heavily audited?

Re: XEN - networking and performance

[D. Duckworth]

I've removed xen-users, the subject seems appropriate for xen-devel now.

> > I tried to use xl with xen4 for a while but due to bugs and missing
> > features I had to go back to xm and xend. This is where the fun
> > begins. In the past I used xend with network-bridge and for some
> > strange reason (voodoo probably) I blindly accepted that script in
> > the past and blamed myself for not appreciating it. But let's be
> > blunt and honest: the scripts, in particular the script that
> > *modifies dom0 networking during xend startup* is the biggest piece
> > of sh!# idea I have ever seen in Xen. It creates bridges, takes
> > eth0 down, tortures dom0 with occult ip addr/route, brctl, sysctl
> > and iptables awk/sed manipulations and then it has you looking at
> > your screen yearning for the moment that ping timeouts become ping
> > replies, telling you that your box is reachable again. This script
> > is a malevolent demon from the sewers of Norman the Golgothan and
> > the worst part is that network-bridge is also still the
> 
> <laughs>
> > recommended default!
> 
> Can you point me to where it mentions that please?

Sorry for the word flood & glad you had to laugh at my cynicism, I had
wanted to use my Norman comparison at least once somewhere and this was
the perfect opportunity. I'll explain the subtlety of the above problem.

So in late 2008 installed xen, then went through xend-config.sxp a
couple of times and came across bridges as the documented feature and
then two non-descriptive oneliners of alternatives. No need for NAT in
my situation but I skipped routing due to a lack of understanding on my
side. Heh what can I say, "know thine limitations?" I did search for
info about the routing option and it did not make sense to me at the
time. I think there was no such thing on the wiki back then either. So
i shelved the idea for later consideration and a few months later there
was a wiki update that did vaguely mention routing. But I already had a
working bridge setup, a tentative one due to the mysterious voodoo
happening somewhere on the machine. I probably have made this point
enough now but if it happens to me it probably happens to others too. I
had no clue why arp was being disabled, why the mac got
fe:ff:ff:ff:ff:ff. I do now thanks to other puzzled people who emailed
their questions and got them answered.

> We realized that the networking setup is quite complex and would be
> best left in the hands of the admins. The problem is that..
> > 
> > It surely seems so. Xen's /etc/xen/scripts (another design fail, why
> > not /usr?) and udev scripts are confusing ad-hoc bloatware routines
> > and are not transparent at all. With the current xen4 I saw the
> > premature advice to more or less 'prepare for migration from xm to
> > xl'. Yet, xl supports less and is conflicting: there is no vifname,
> > no 'xl new/delete', no more python, no relocation and suddenly there
> > is a conflict between 'xm start domain' versus 'xl start
> > /etc/xen/domain'.
> > 
> > So new features emerge, adding to the confusion of the end user,
> > while old problems are not being fixed properly. I wonder why,
> > especially because it does not seem that xm and xend are the broken
> > parts that need to be replaced by an unstable interface.
> > 
> > What needs attention first and foremost are two things, first of
> > which is real and wise effort into one simple, minimal script that
> > just handles the minimum in a transparent way e.g. control the
> > hypervisor, manage vms, manage the backend. Of course networking can
> > be done on domain start too, but this has happen in an entirely
> > different way from what it does and how it does it. This is so
> > important because it gives more control to the user that runs Xen.
> > It's also a good moment to build in proper and mature support for
> > IPv6.
> > 
> > Secondly, the website and documentation should be cleaned up and
> > revised where appropriate. The current situation is a mess that has
> > a much too steep and incompatible learning curve right now - for
> > example, a bridge should just not be named eth0 and a physical
> > device should not be renamed at all. It's fundamentally wrong,
> > stupid, mad as hell and a PR failure for Xen to do it this way out
> > of the box. No matter how often and detailed it has been documented
> > on the website. 
> 
> .. the documentation and setup is sometimes quite hard. BTW, we are
> going to do on Oct 26th a Documentation Day to clean up some of this
> mess. Would you be intereted in helping along - perhaps in the
> networking Wiki?

Interested yes, perhaps. Could you please send me the details? As for
the notion that documentation and setup can be hard, that's exactly why
I, like you, suggest to move away from OS configuration as much as
possible -- obviously the hypervisor and kernel integration are the
primary domain. Daily commits by committed people seem to support this
theory.

Moving away from OS integration narrows the scope of the Xen project
which means less specific documentation needs to be maintained while
making it easier for everyone to invoke their own methods. OS
integration, even udev scripts, should move to the distribution's
developers and they can leave the rest up to the end user. Supporting
everything from within Xen is like trying to be everybody's friend. An
impossible task, just ask Ghandi, he tried it and got shot.

So take vif-nat. It does stuff with dhcp with all kinds of assumptions.
I don't even use ISC's dhcpd, I use dnsmasq. And this script does stuff
with iptables. I use shorewall and shreeked when I noticed a sudden
change in my tables. A fast but not fun two or three hours later I
found what did do it... after having searched through /usr and the domU.
So I found this whole script an abomination. Here's one other example
from the function dhcp_arg_add_entry (!)

# handle Red Hat, SUSE, and Debian styles, with or without quotes

Just...don't do this. These might be three big distributions but I
wonder whether they outnumber all users of all other distributions. And
the BSD camp doesn't seem to be really fond of Xen at the moment either.
In fact to me it seems like Xen's arrows all focus on Linux while a
bigger arrow marked KVM points towards the same direction. But I'm not
deeply involved with any project's development and never have been, so I
could be totally wrong. I don't even know Xen's project roadmap, future
outlook et cetera. I'll even admit I'm just a boob!

Still, it seems only logical to move to a more generic way of
doing things while OS/distribution devs and admins take care of their
respective domains. (Is this even remotely making sense to anyone?)

> > 
> > I propose something like the following for xen networking:
> > 
> > * Xen will not manipulate non-xen devices or a firewall under any
> >   circumstance, it might only add or substract routes and/or rules
> > from the routing tables,
> 
> Uh, what is 'non-xen' devices? Like bridges?

Yes. If I would use a Xen created bridge it has STP disabled, more
occult features, It's evil. So Xen must not create or touch those and
also not touch the routing/iptables with automated scripts. It is
clearly a much better idea to create transparent per-domain networking
options. Can be done in different ways like with pre/post-up/down
networking scripts that the dev/admin has to write. It still allows for
Xen to supply examples and suggest sane defaults. This time
transparently, making it all easier to maintain for everyone because
more modular. There can be a syntax check that spawns a warning if a
vif is configured without configured settings/scripts etc...

> 
> > * Allow for networking configuration per domU. For example let
> >   networking per device be nat, routed, bridged or custom, where
> >   all name the interface and bring it up; nat only adds the ip to
> > the routing table; routed could be an array of routes and rules
> > that need to be added or subtracted from various routing tables and
> > it might support proxyarp; bridged turns off arp, sets the mac on
> > the vif and then adds the interface to a bridge that should already
> > be created by the user; and custom is a custom set of unmanaged
> > commands after creating and destroying a domain.
> 
> You lost me. <sigh> I am using a bridge configuration and just
> do:
> 
> auto lo
> iface lo inet loopback
> 
> auto switch
> iface switch inet static
> 	address 192.168.101.16
> 	netmask 255.255.255.0
> 	gateway 192.168.101.1
> 	bridge_ports eth2
> 
> And just use that 'bridge=switch' in all my configuration. And that
> seems to work just fine - wouldn't that be best way of providing
> the first network setup to users? I would think the majority of folks
> do something akin to this?

Well I said a lot of things but not that it's broken. :) Not that I
couldn't, though. The above stuff works in Debian, but Arch Linux has no
ifup, neither does bsd. So those maintainers have to port Xen's code
with patches making it time expensive (and painfully dull). Instead the
whole routine that removes interfaces, adds new bridges, sets iptables,
etc. should simply be deleted. Not Xen's business; makes grown men cry.
How the majority of users handles it is not really relevant imho
because they are also bound to what Xen supplies. A lot of good people
see no obvious choice and a lot of good people have no clue about STP.
Consequently you'll find a lot of the same help requests on various
forums and lists when searching for clues. 

I tried to make sense of the minimum that the xen scripts do (with
xenstore, vif creation, device and udev interaction etc.) but the jungle
of 118 functions (grep '()' /etc/xen/scripts/* | wc -l, take or leave a
few) was not helping. I'd appreciate it if someone would write me a
detailed step by step hierarchy that explains what happens right in a
Xen4.2 dom0 from when a domain is created until it's running, what might
happen in between, and what happens from a domain's shutdown or crash to
domain deletion.

> > So, with that off my chest and the second line of my network-bridge
> > being the words "exit 0" Xen lets my dom0 configuration alone
> > like it is supposed to do. While KVM is becoming a 'next cool
> > thing' for many people I would still prefer a separate hypervisor
> > so now the fat just has to be removed from Xen.
> > 
> 
> I am all for removing fat. Do you have links to some of particularly
> bad Wiki pages that should be heavily audited?

I would be more entertained to see how far I can take this plan and
if/how discussion will shape it into a workable form. I think that the
entire networking section and more can be wholly rewritten together
with new scripts. Mind that I am only talking about core Xen and not
about XCP/libvirt/foo/bar, in fact I don't even have experience with
any of that and I simply assume that they handle stuff in their own way.

One question that has been bugging me is the xm/xl thing. What are the
exact plans, will xl indeed be phased in while xm will be phased out and
what will/won't be supported by xl? Or did I misinterpret it. Imho
a suggestion to switch over should only be come out when xl becomes
'distribution-grade stable'.

Right now it's just a tool worth testing.
(-That's what "she" said..)

_donduq.

Re: XEN - networking and performance

[Ian Campbell]

On Wed, 2011-10-12 at 05:50 +0100, D. Duckworth wrote:
> > auto lo
> > iface lo inet loopback
> >
> > auto switch
> > iface switch inet static
> >       address 192.168.101.16
> >       netmask 255.255.255.0
> >       gateway 192.168.101.1
> >       bridge_ports eth2
> >
> > And just use that 'bridge=switch' in all my configuration. And that
> > seems to work just fine - wouldn't that be best way of providing
> > the first network setup to users? I would think the majority of
> folks
> > do something akin to this?
> 
> Well I said a lot of things but not that it's broken. :) Not that I
> couldn't, though. The above stuff works in Debian, but Arch Linux has
> no ifup, neither does bsd. 

If you know how to setup a bridge on those systems please could you
update http://wiki.xen.org/xenwiki/HostConfiguration/Networking to add
links to the relevant Arch / BSD documentation. A couple of simple
examples for the most common case would also be useful.

> So those maintainers have to port Xen's code with patches making it
> time expensive (and painfully dull). Instead the whole routine that
> removes interfaces, adds new bridges, sets iptables, etc. should
> simply be deleted.

We agree and this is why with the xl toolstack we do not support the use
of these scripts or ever call out to them automatically (they haven't
actually been deleted, since xend still uses them). With xl we recommend
that folks use their distribution provided mechanisms to setup the host
network configuration. 

We made some attempt mitigate the network-* insanity for xend users, by
making it such that the script will only do the mad things if it is
(heuristically) detected that the admin hasn't already set something up
themselves, it's not clear how effective this strategy was in practice,
even better is to just comment out the relevant line in your
xend-config.sxp IMHO. Hopefully it is explained below why we aren't
doing any wholesale reworking of how xend does things.

None of that really addresses the complexity of the existing vif-*
scripts. I suspect that deprecating the network-* ones for xl has
effectively deprecated all but the vif-bridge one since e.g. vif-nat
depends quite heavily on the setup which network-nat has done.

There is probably scope for providing a much simpler vif-bridge script
for use with xl, the existing one does have some odd stuff in it.

For more complex scenarios like nat etc there is certainly value in
having examples of how people have done stuff and as James suggested
we'd certainly like to see people posting (or writing up on the wiki)
their own configurations and scripts etc.

> One question that has been bugging me is the xm/xl thing. What are the
> exact plans, will xl indeed be phased in while xm will be phased out
> and what will/won't be supported by xl? Or did I misinterpret it. Imho
> a suggestion to switch over should only be come out when xl becomes
> 'distribution-grade stable'.

xend has been effectively unmaintained for several releases now and
there is nobody who is willing to step up and support it. Unless someone
steps up as a maintainer it will become more deprecated with time and in
a few releases I expect it will be removed from the tree.

On the other hand we are actively developing xl and supporting it.

In 4.1 we recommended that people try xl and report the bugs and missing
features which would prevent them from transitioning from xend. We are
doing our best to address these bugs and short-comings as they are
reported to us. It is our hope that we can fully recommend xl in the 4.2
or 4.3 time scale. Obviously if people delay trying xl until we've
switched to it then there is something of a chicken and egg problem wrt
making sure it supports their needs.

It seems that you have several issues which are impacting you but I'm
having trouble digesting them all out of your mails. Posting
individually about any the specific issues which you have that are
preventing you from using xl (or indeed Xen generally) will ensure a
much greater chance that someone will notice and do something about
them. I'm sorry to say that posting long rants is unlikely to have the
same effect...

Thanks,
Ian.

Re: XEN - networking and performance

[Konrad Rzeszutek Wilk]

> > .. the documentation and setup is sometimes quite hard. BTW, we are
> > going to do on Oct 26th a Documentation Day to clean up some of this
> > mess. Would you be intereted in helping along - perhaps in the
> > networking Wiki?
> 
> Interested yes, perhaps. Could you please send me the details? As for

 http://lists.xensource.com/archives/html/xen-users/2011-09/msg00494.html

and #xendocday on Oct 26th.