Network-bridge script with bonding and vlan

Network-bridge script with bonding and vlan

[Greg Brackley]

I'm trying to get the latest x86_64 development release running with eth0 & 
eth1 bonded (using 802.3ad) with VLAN support.  I am trying to get a VLAN 
support running with the intention of putting each domU on its own VLAN. 
Given that the dom0 machine won't have an IP address on any of the domU 
VLAN's, there should be reasonable network isolation between the domains.

I can get the bonding/vlan configuration working on a machine without Xen. 
However I am having troubles getting the VLAN interfaces bridged correctly 
to the xen0 and xenU domains vif interfaces. I'm unsure as to which 
interfaces should have what MAC address, and/or how to do that.

I've managed to get the VLAN support working on an older version of Xen 
devel x86, but found I had to run tcpdump in dom0 on each VLAN interface 
(presumably to put it into promiscuous mode).

Any help would be appreciated.

Greg.

Re: Network-bridge script with bonding and vlan

[Ewan Mellor]

On Fri, Oct 21, 2005 at 12:34:40AM +1300, Greg Brackley wrote:

> I'm trying to get the latest x86_64 development release running with eth0 & 
> eth1 bonded (using 802.3ad) with VLAN support.  I am trying to get a VLAN 
> support running with the intention of putting each domU on its own VLAN. 
> Given that the dom0 machine won't have an IP address on any of the domU 
> VLAN's, there should be reasonable network isolation between the domains.
> 
> I can get the bonding/vlan configuration working on a machine without Xen. 
> However I am having troubles getting the VLAN interfaces bridged correctly 
> to the xen0 and xenU domains vif interfaces. I'm unsure as to which 
> interfaces should have what MAC address, and/or how to do that.

Firstly, I would wait for the new network-bridge script to be pushed, or at
the very least use the one that Kurt Garloff posted to the list yesterday.

The topology we use is, in domain 0:

eth0 in dom0, virtual device, good IP address and physical device's MAC
|
(loopback)
|
vif0.0, virtual device bound to bridge, no IP, fake MAC
|
xenbr0, bridge interface, no IP, fake MAC
|
peth0, physical device, no IP, fake MAC

and for the guest domains

eth0, virtual device in guest domain, good IP address, and random MAC
|
(interdomain connection)
|
vif<domain id>.<device id>, in domain 0, bound to bridge, no IP, fake MAC
|
xenbr0, peth0 as above.

The physical device starts of as eth0, and then is renamed to peth0 by the
network-bridge script.

All the routing points at the interface with a good IP address.

The fake MAC address we use is FE:FF:FF:FF:FF:FF, which is this value for
reasons of compatibility with STP, but I don't understand this, I just do as
I'm told ;-)

HTH,

Ewan.

Re: Network-bridge script with bonding and vlan

[Greg Brackley]

----- Original Message ----- 
From: "Ewan Mellor" <ewan@xensource.com>
> Firstly, I would wait for the new network-bridge script to be pushed, or 
> at
> the very least use the one that Kurt Garloff posted to the list yesterday.

I've been following that thread. From my limited understanding of the xen 
bridged networking I see it decomposed into three areas:

1. The hardware driver support
2. The bridge
3. The xen virtual device support (backend-front driver)

In a trivial case, the hardware support might just have eth0. In my case I 
want to have eth0, eth1, bond0, and several eth0.xx VLAN interfaces.  I 
understand most people are configuring these with their distribution 
specific network support.

In the trivial case, the bridge is xenbr0 (was xen-br0).  In my case I think 
I want one bridge per domain (including one for xen0). I'm trying to use FC4 
network configuration to do this, because I don't see how the xen script 
helps do this. I see that I need to create a bridge per domain (and the 
standard scripts assume one per machine).

In the trivial case the xen support is vif0.0->veth0, and vifN.0->eth0 
(where N is a non-zero domain number). The scripts seem to treat xen0 as a 
special case, and I'm not sure why.

In a production system (c.f. a development system) I feel it would be 
desirable to be able to mix and match the three different areas. It seems 
that the scripts assume a fixed xen0 configuration (of 1 interface and one 
bridge, and one vif), and don't allow the running of vif script for domain 
0.

In the configuration that I'm trying to get going, I am not renaming veth0 
in domain0. Instead I just treat veth0 as the main interface for that 
domain, and give it an IP address and route to it. Is that bad idea?  By not 
doing renaming of interfaces, I am trying to avoid having to copy over 
addresses.

> The fake MAC address we use is FE:FF:FF:FF:FF:FF, which is this value for
> reasons of compatibility with STP, but I don't understand this, I just do 
> as
> I'm told ;-)

I'm not too sure about this either, but I would like to know what is really 
means. I can't find a definition for it , e.g.on 
http://standards.ieee.org/regauth/oui/index.shtml. Which drivers understand 
this mac address? What does it really mean? I'm guessing it means something 
along the lines of use the mac address from the source when forwarding.

Greg :-)

Re: Network-bridge script with bonding and vlan

[Ewan Mellor]

On Fri, Oct 21, 2005 at 02:16:49PM +1300, Greg Brackley wrote:

> ----- Original Message ----- 
> From: "Ewan Mellor" <ewan@xensource.com>
> >Firstly, I would wait for the new network-bridge script to be pushed, or 
> >at
> >the very least use the one that Kurt Garloff posted to the list yesterday.
> 
> I've been following that thread. From my limited understanding of the xen 
> bridged networking I see it decomposed into three areas:
> 
> 1. The hardware driver support
> 2. The bridge
> 3. The xen virtual device support (backend-front driver)
> 
> In a trivial case, the hardware support might just have eth0. In my case I 
> want to have eth0, eth1, bond0, and several eth0.xx VLAN interfaces.  I 
> understand most people are configuring these with their distribution 
> specific network support.
> 
> In the trivial case, the bridge is xenbr0 (was xen-br0).  In my case I 
> think I want one bridge per domain (including one for xen0). I'm trying to 
> use FC4 network configuration to do this, because I don't see how the xen 
> script helps do this. I see that I need to create a bridge per domain (and 
> the standard scripts assume one per machine).

The new script allows for one bridge per outgoing interface, I think, so in
your case one bridge per VLAN ought to be possible.  Why do you need one
bridge per domain?  What are you bridging in this case?  The topology we use
uses a bridge to plex all of the guests onto a specific outgoing interface --
so what are you bridging to require one per domain?

> In the trivial case the xen support is vif0.0->veth0, and vifN.0->eth0 
> (where N is a non-zero domain number).

Do your arrows refer to incoming packets?  If so, I think that is right,
though veth0 is renamed to eth0 now, so that dom0 sees a "normal" eth0
interface.

> The scripts seem to treat xen0 as a special case, and I'm not sure why.

Domain 0 is the privileged domain i.e. the one that manages the creation of
other domains.  It comes up on boot, and is where things like the script in
question are running.  That is why it is a special case.  Also, it is arranged
so that dom0's interface appears to have the MAC address of the underlying
NIC -- you can't do this with the guest domains, of course.

> In a production system (c.f. a development system) I feel it would be 
> desirable to be able to mix and match the three different areas. It seems 
> that the scripts assume a fixed xen0 configuration (of 1 interface and one 
> bridge, and one vif),

Now, N interfaces, N bridges, N vifs for domain 0 (where N is the same in each
case).

> and don't allow the running of vif script for domain 0.

No, this wouldn't really make sense, and would be difficult to arrange.

> In the configuration that I'm trying to get going, I am not renaming veth0 
> in domain0. Instead I just treat veth0 as the main interface for that 
> domain, and give it an IP address and route to it. Is that bad idea?  By 
> not doing renaming of interfaces, I am trying to avoid having to copy over 
> addresses.

I've no idea whether that makes sense, but why are you trying to do it?


The bottom line, I think, is that Xen does not try to do everything with
network configuration.  We are happy to help, and happy to provide example
scripts for simple configurations, but I don't think that we will ever try to
detect every distro, and configure every convoluted VLAN configuration.
That's why the network handling is scripted -- so that you can get your hands
dirty if needs be.  Now I'm not saying that I wash my hands of you!  But if
you really do need a topology that's way beyond the capabilities of the
network-bridge script, then you ought to think about writing a different
script rather than try and shoehorn the wrong things in the wrong places.
Maybe we'll accept your script into the repository if it's clean and general
enough, but whether you maintain it or we do, I don't think that it
(necessarily) belongs in network-bridge.

Ewan.

Re: Network-bridge script with bonding and vlan

[Greg Brackley]

----- Original Message ----- 
From: "Ewan Mellor" <ewan@xensource.com>


> The new script allows for one bridge per outgoing interface, I think, so 
> in
> your case one bridge per VLAN ought to be possible.  Why do you need one
> bridge per domain?  What are you bridging in this case?  The topology we 
> use
> uses a bridge to plex all of the guests onto a specific outgoing 
> interface --
> so what are you bridging to require one per domain?

The main outcome I'm trying to achieve is isolation between the domains. 
Typically we would have this separation between our production servers (not 
all lumped together on one LAN).  Having the separation at layer 2 would 
seem like to be a positive thing to maintain.  Using the VLAN interfaces 
gives the impression that the machine has many network adapters [this would 
seem to me to be a good fit with a virtualisation technology].

>> In the trivial case the xen support is vif0.0->veth0, and vifN.0->eth0
>> (where N is a non-zero domain number).
> Do your arrows refer to incoming packets?  If so, I think that is right,
> though veth0 is renamed to eth0 now, so that dom0 sees a "normal" eth0
> interface.

Sorry, I wasn't very clear there. Yes, incoming packets, or 
backend<->frontend.


>> The scripts seem to treat xen0 as a special case, and I'm not sure why.
> Domain 0 is the privileged domain i.e. the one that manages the creation 
> of
> other domains.  It comes up on boot, and is where things like the script 
> in
> question are running.  That is why it is a special case.  Also, it is 
> arranged
> so that dom0's interface appears to have the MAC address of the underlying
> NIC -- you can't do this with the guest domains, of course.

Got that.  But could the part of the dom0 script that configures the dom0 
vif0.0<->veth0 part of the networking be extracted/pluggable just like the 
domU setup of the vifN.0 interface.

> I've no idea whether that makes sense, but why are you trying to do it?

I guess it comes from the desire not to configure an interface one way, and 
then moments later tear it all down and configure it another way.  I can see 
that the current way is great for development, and integrates well with an 
existing machine - not sure I'd want to head in that direction on a 
non-development system.

> The bottom line, I think, is that Xen does not try to do everything with
> network configuration.  We are happy to help, and happy to provide example

Thanks very much for your feedback. I lot of my comments are probably due to 
being a relative newbie.

Most of the configuration I have been doing has been with the fc4 init 
scripts. I'm looking for less, rather than more. However I am having 
problems getting packets forwarded/bridged correctly.  ICMP pings seem to 
work just fine, but the only way to get TCP/UDP packets to forward is to 
attach tcpdump to the VLAN interface. I probably missing something very 
obvious. I don't have the machine in front of me (and the networking isn't 
working), so can't post the current config.

I'm unclear as to why the physical/bridge/backend interfaces need to have a 
special mac address (fe:ff:ff:ff:ff:ff), and why they need arp disabled 
(since they have no IP address).  What is special about fe:ff:ff:ff:ff:ff? 
Could it be any locally assigned unicast mac address?

Greg.

Re: Network-bridge script with bonding and vlan

[Ted Kaczmarek]

On Sat, 2005-10-22 at 00:23 +1300, Greg Brackley wrote:
> ----- Original Message ----- 
> From: "Ewan Mellor" <ewan@xensource.com>
> 
> 
> > The new script allows for one bridge per outgoing interface, I think, so 
> > in
> > your case one bridge per VLAN ought to be possible.  Why do you need one
> > bridge per domain?  What are you bridging in this case?  The topology we 
> > use
> > uses a bridge to plex all of the guests onto a specific outgoing 
> > interface --
> > so what are you bridging to require one per domain?
> 
> The main outcome I'm trying to achieve is isolation between the domains. 
> Typically we would have this separation between our production servers (not 
> all lumped together on one LAN).  Having the separation at layer 2 would 
> seem like to be a positive thing to maintain.  Using the VLAN interfaces 
> gives the impression that the machine has many network adapters [this would 
> seem to me to be a good fit with a virtualisation technology].
> 
> >> In the trivial case the xen support is vif0.0->veth0, and vifN.0->eth0
> >> (where N is a non-zero domain number).
> > Do your arrows refer to incoming packets?  If so, I think that is right,
> > though veth0 is renamed to eth0 now, so that dom0 sees a "normal" eth0
> > interface.
> 
> Sorry, I wasn't very clear there. Yes, incoming packets, or 
> backend<->frontend.
> 
> 
> >> The scripts seem to treat xen0 as a special case, and I'm not sure why.
> > Domain 0 is the privileged domain i.e. the one that manages the creation 
> > of
> > other domains.  It comes up on boot, and is where things like the script 
> > in
> > question are running.  That is why it is a special case.  Also, it is 
> > arranged
> > so that dom0's interface appears to have the MAC address of the underlying
> > NIC -- you can't do this with the guest domains, of course.
> 
> Got that.  But could the part of the dom0 script that configures the dom0 
> vif0.0<->veth0 part of the networking be extracted/pluggable just like the 
> domU setup of the vifN.0 interface.
> 
> > I've no idea whether that makes sense, but why are you trying to do it?
> 
> I guess it comes from the desire not to configure an interface one way, and 
> then moments later tear it all down and configure it another way.  I can see 
> that the current way is great for development, and integrates well with an 
> existing machine - not sure I'd want to head in that direction on a 
> non-development system.
> 
> > The bottom line, I think, is that Xen does not try to do everything with
> > network configuration.  We are happy to help, and happy to provide example
> 
> Thanks very much for your feedback. I lot of my comments are probably due to 
> being a relative newbie.
> 
> Most of the configuration I have been doing has been with the fc4 init 
> scripts. I'm looking for less, rather than more. However I am having 
> problems getting packets forwarded/bridged correctly.  ICMP pings seem to 
> work just fine, but the only way to get TCP/UDP packets to forward is to 
> attach tcpdump to the VLAN interface. I probably missing something very 
> obvious. I don't have the machine in front of me (and the networking isn't 
> working), so can't post the current config.
> 
> I'm unclear as to why the physical/bridge/backend interfaces need to have a 
> special mac address (fe:ff:ff:ff:ff:ff), and why they need arp disabled 
> (since they have no IP address).  What is special about fe:ff:ff:ff:ff:ff? 
> Could it be any locally assigned unicast mac address?
> 
> Greg.
> 

I am still seeing very erratic network behavior in general, on first
reboot of the dom0 things will generally behave well, but after the dom0
has been up for a while, with domU's brought up and down along the way
all kinds of random failures start to happen.

DomU oops 
[<c02644b2>] xenbus_dev_ok+0x32/0x60

After domU is brought down
mac address for domU is still showing up on xen-br0

domU is brought up
vif is not mapped to domU

I am no longer seeing the issue with peth0 seeing packet with its own
address though.

xen_changeset : Wed Oct 19 06:53:00 2005 +0100 7425:7c951e3eb5ab

I am not seeing any issues like this at all on my UP P4, just my SMP
Athlon. None of these are easy to replicate :-(


xen console output, the issues correlate to the smp_send_stop
disable_local_APIC message.

(XEN) DOM1: (file=mm.c, line=460) Non-privileged attempt to map I/O
space 00000000
(XEN) DOM1: (file=mm.c, line=460) Non-privileged attempt to map I/O
space 00000000
smp_send_stop disable_local_APIC
smp_send_stop disable_local_APIC
(XEN) DOM4: (file=mm.c, line=460) Non-privileged attempt to map I/O
space 00000000
(XEN) DOM4: (file=mm.c, line=460) Non-privileged attempt to map I/O
space 00000000
smp_send_stop disable_local_APIC
(XEN) DOM5: (file=mm.c, line=460) Non-privileged attempt to map I/O
space 00000000
(XEN) DOM5: (file=mm.c, line=460) Non-privileged attempt to map I/O
space 00000000
(XEN) DOM22: (file=mm.c, line=460) Non-privileged attempt to map I/O
space 00000000
(XEN) DOM22: (file=mm.c, line=460) Non-privileged attempt to map I/O
space 00000000
(XEN) DOM23: (file=mm.c, line=460) Non-privileged attempt to map I/O
space 00000000
(XEN) DOM23: (file=mm.c, line=460) Non-privileged attempt to map I/O
space 00000000


Regards,
Ted

Re: Network-bridge script with bonding and vlan

[Greg Brackley]

----- Original Message ----- 
From: "Ted Kaczmarek" <tedkaz@optonline.net>
 > I am still seeing very erratic network behavior in general, on first
> reboot of the dom0 things will generally behave well, but after the dom0
> has been up for a while, with domU's brought up and down along the way
> all kinds of random failures start to happen.

Once the domains are up I haven't seen any additional problems.  I seem to 
be finally getting the networking under control by using standard FC4 init 
scripts. I'm not sure what I was getting wrong with the networking 
previously.

The current issue I have is that my test domainU fails to start every time 
after dom0 is started (log below). It starts on the second and subsequent 
restarts. I tried pulling the LVM/Raid support out of the XenU kernel, but 
it didn't help. The root being passed to the domU in the log is a LVM volume 
('phy:/dev/VolGroup00/root01xenu,sda1,w').

Greg :-)



Linux version 2.6.12-xenU (xxxx@blue.lucidsolutions.co.nz) (gcc version 
4.0.1 20050727 (Red Hat 4.0.1-5)) #3 SMP Sat Oct 22 22:38:14 NZDT 2005
kernel direct mapping tables upto 10000000 @ 42e000-4b0000
Built 1 zonelists
Kernel command line:  root=/dev/sda1 ro 3
Initializing CPU#0
PID hash table entries: 2048 (order: 11, 65536 bytes)
Xen reported: 1593.798 MHz processor.
Dentry cache hash table entries: 65536 (order: 7, 524288 bytes)
Inode-cache hash table entries: 32768 (order: 6, 262144 bytes)
Memory: 253184k/262144k available (1660k kernel code, 8496k reserved, 551k 
data, 128k init)
Mount-cache hash table entries: 256
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 1024K (64 bytes/line)
Brought up 1 CPUs
NET: Registered protocol family 16
xen_mem: Initialising balloon driver.
Grant table initialized
IA32 emulation $Id: sys_ia32.c,v 1.32 2002/03/24 13:02:28 ak Exp $
audit: initializing netlink socket (disabled)
audit(1129974155.344:0): initialized
VFS: Disk quotas dquot_6.5.1
Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
Initializing Cryptographic API
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered
RAMDISK driver initialized: 16 RAM disks of 16384K size 1024 blocksize
Xen virtual console successfully installed as tty1
Event-channel device installed.
xen_blk: Initialising virtual block device driver
xen_net: Initialising virtual ethernet driver.
NET: Registered protocol family 2
IP: routing cache hash table of 1024 buckets, 16Kbytes
TCP established hash table entries: 16384 (order: 6, 262144 bytes)
TCP bind hash table entries: 16384 (order: 6, 262144 bytes)
TCP: Hash tables configured (established 16384 bind 16384)
Initializing IPsec netlink socket
NET: Registered protocol family 1
NET: Registered protocol family 17
VFS: Cannot open root device "sda1" or unknown-block(0,0)
Please append a correct "root=" boot option
Kernel panic - not syncing: VFS: Unable to mount root fs on 
unknown-block(0,0)

Re: Network-bridge script with bonding and vlan

[Greg Brackley]

----- Original Message ----- 
From: "Ted Kaczmarek" <tedkaz@optonline.net>
 > I am still seeing very erratic network behavior in general, on first
> reboot of the dom0 things will generally behave well, but after the dom0
> has been up for a while, with domU's brought up and down along the way
> all kinds of random failures start to happen.

Once the domains are up I haven't seen any additional problems.  I seem to 
be finally getting the networking under control by using standard FC4 init 
scripts. I'm not sure what I was getting wrong with the networking 
previously.

The current issue I have is that my test domainU fails to start every time 
after dom0 is started (log below). It starts on the second and subsequent 
restarts. I tried pulling the LVM/Raid support out of the XenU kernel, but 
it didn't help. The root being passed to the domU in the log is a LVM volume 
('phy:/dev/VolGroup00/root01xenu,sda1,w').

Greg :-)



Linux version 2.6.12-xenU (xxxx@blue.lucidsolutions.co.nz) (gcc version 
4.0.1 20050727 (Red Hat 4.0.1-5)) #3 SMP Sat Oct 22 22:38:14 NZDT 2005
kernel direct mapping tables upto 10000000 @ 42e000-4b0000
Built 1 zonelists
Kernel command line:  root=/dev/sda1 ro 3
Initializing CPU#0
PID hash table entries: 2048 (order: 11, 65536 bytes)
Xen reported: 1593.798 MHz processor.
Dentry cache hash table entries: 65536 (order: 7, 524288 bytes)
Inode-cache hash table entries: 32768 (order: 6, 262144 bytes)
Memory: 253184k/262144k available (1660k kernel code, 8496k reserved, 551k 
data, 128k init)
Mount-cache hash table entries: 256
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 1024K (64 bytes/line)
Brought up 1 CPUs
NET: Registered protocol family 16
xen_mem: Initialising balloon driver.
Grant table initialized
IA32 emulation $Id: sys_ia32.c,v 1.32 2002/03/24 13:02:28 ak Exp $
audit: initializing netlink socket (disabled)
audit(1129974155.344:0): initialized
VFS: Disk quotas dquot_6.5.1
Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
Initializing Cryptographic API
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered
RAMDISK driver initialized: 16 RAM disks of 16384K size 1024 blocksize
Xen virtual console successfully installed as tty1
Event-channel device installed.
xen_blk: Initialising virtual block device driver
xen_net: Initialising virtual ethernet driver.
NET: Registered protocol family 2
IP: routing cache hash table of 1024 buckets, 16Kbytes
TCP established hash table entries: 16384 (order: 6, 262144 bytes)
TCP bind hash table entries: 16384 (order: 6, 262144 bytes)
TCP: Hash tables configured (established 16384 bind 16384)
Initializing IPsec netlink socket
NET: Registered protocol family 1
NET: Registered protocol family 17
VFS: Cannot open root device "sda1" or unknown-block(0,0)
Please append a correct "root=" boot option
Kernel panic - not syncing: VFS: Unable to mount root fs on 
unknown-block(0,0)

Re: Network-bridge script with bonding and vlan

[Ewan Mellor]

On Sat, Oct 22, 2005 at 11:17:21PM +1300, Greg Brackley wrote:

> ----- Original Message ----- 
> From: "Ted Kaczmarek" <tedkaz@optonline.net>
> > I am still seeing very erratic network behavior in general, on first
> >reboot of the dom0 things will generally behave well, but after the dom0
> >has been up for a while, with domU's brought up and down along the way
> >all kinds of random failures start to happen.
> 
> Once the domains are up I haven't seen any additional problems.  I seem to 
> be finally getting the networking under control by using standard FC4 init 
> scripts. I'm not sure what I was getting wrong with the networking 
> previously.
> 
> The current issue I have is that my test domainU fails to start every time 
> after dom0 is started (log below). It starts on the second and subsequent 
> restarts. I tried pulling the LVM/Raid support out of the XenU kernel, but 
> it didn't help. The root being passed to the domU in the log is a LVM 
> volume ('phy:/dev/VolGroup00/root01xenu,sda1,w').

When this happens, could you please run

PYTHONPATH=/usr/lib/python python /usr/lib/python/xen/util/diagnose.py

and tell me what you get?

You might also find useful information in your syslog (on Debian in
/var/log/syslog and /var/log/debug but it will depend on your syslog config).
It looks like a failure in the hotplug scripts, so knowing whether they are
running at all and what they are trying to write would be good.

Ewan.

Re: Network-bridge script with bonding and vlan

[Greg Brackley]

[-- Attachment #1: Type: text/plain, Size: 3613 bytes --]

----- Original Message ----- 
From: "Ewan Mellor" <ewan@xensource.com>
> Maybe we'll accept your script into the repository if it's clean and 
> general
> enough, but whether you maintain it or we do, I don't think that it
> (necessarily) belongs in network-bridge.

Well I have managed to get a trivial example configuration up and running. 
It uses the Fedora initscripts to initialise the networking (c.f. using 
network-bridge script). It still uses the vif-bridge script for domU's. It 
might be of some use to others running FC4.

The setup is two GbE ports (eth0 and eth1) bonded together using 802.3ad. 
Over that I have setup a single 802.1q VLAN (id 10) - with the intention of 
expanding this to one VLAN per domU. I configured a bridge (br0). I have 
created a config file to add vif0.0 to the bridge. Lastly I have provided a 
configuration file for veth0 which provides a MAC address and an IP address 
for dom0.

I notice that the dom0 vif interface uses zero based indexes (i.e vif0.0), 
however the domU network interface numbers are one based (i.e. 
vif<domain#>.1). Is there a reason for this?

I'd appreciate any feedback, especially if I'm doing bad things.

Greg.

--

# ip addr show
1: lo: <LOOPBACK,UP> mtu 16436 qdisc noqueue
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
2: vif0.0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue
    link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
3: veth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue
    link/ether aa:de:ad:be:ef:01 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.20/24 brd 192.168.0.255 scope global veth0
4: bond0: <BROADCAST,MULTICAST,MASTER,UP> mtu 1500 qdisc noqueue
    link/ether 00:e0:81:2e:13:14 brd ff:ff:ff:ff:ff:ff
5: eth0: <BROADCAST,MULTICAST,SLAVE,UP> mtu 1500 qdisc pfifo_fast master 
bond0 qlen 1000
    link/ether 00:e0:81:2e:13:14 brd ff:ff:ff:ff:ff:ff
6: eth1: <BROADCAST,MULTICAST,SLAVE,UP> mtu 1500 qdisc pfifo_fast master 
bond0 qlen 1000
    link/ether 00:e0:81:2e:13:14 brd ff:ff:ff:ff:ff:ff
7: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000
    link/ether 00:e0:81:2e:12:ca brd ff:ff:ff:ff:ff:ff
8: br0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue
    link/ether 00:e0:81:2e:13:14 brd ff:ff:ff:ff:ff:ff
9: bond0.10: <BROADCAST,MULTICAST,MASTER,UP> mtu 1500 qdisc noqueue
    link/ether 00:e0:81:2e:13:14 brd ff:ff:ff:ff:ff:ff
11: vif2.1: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue
    link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff


# brctl show
bridge name     bridge id               STP enabled     interfaces
br0             8000.00e0812e1314       no              bond0.10
                                                        vif0.0
                                                        vif2.1


--/etc/modprobe.conf--
alias bond0 bonding
options bond0 mode=802.3ad lacp_rate=fast miimon=100

alias eth0 tg3
alias eth1 tg3
alias eth2 e100

---domU config---
kernel = "/boot/vmlinuz-2.6.12-xenU"
memory = 256
name = "purple"
nics=1
vif = [ 'mac=aa:de:ad:be:ef:02, bridge=br0' ]
disk = [ 'phy:/dev/VolGroup00/root01xenu,sda1,w',
         'phy:/dev/VolGroup00/swap01,sdb1,w',
         'phy:/dev/md0,sdc1,w' ]
root = "/dev/sda1 ro"
extra = "3"

---xend-config.sxp---
(xend-port         8000)
(xend-event-port   8001)
(xend-address      'localhost')
(console-port-base 9600)
(console-address   'localhost')
(network-script   network-bridge-nop  )
(vif-bridge        br0)
(vif-netdev        veth0)
(vif-script        vif-bridge)
(vif-antispoof     no)
(block-file block-file)
(block-enbd block-enbd)
(dom0-min-mem 0)
(dom0-cpus 0)


[-- Attachment #2: route-veth0 --]
[-- Type: application/octet-stream, Size: 104 bytes --]

192.168.2.0/24 via 192.168.0.2
192.168.128.0/17 via 192.168.0.2
default via 192.168.0.254 mtu 1500


[-- Attachment #3: ifcfg-bond0 --]
[-- Type: application/octet-stream, Size: 80 bytes --]

DEVICE=bond0
BOOTPROTO=none
ONBOOT=yes
TYPE=Bonding
NOZEROCONF=yes
ARP=no

[-- Attachment #4: ifcfg-bond0.10 --]
[-- Type: application/octet-stream, Size: 123 bytes --]

DEVICE=bond0.10
TYPE=Ethernet
BOOTPROTO=none
ONBOOT=yes
NOZEROCONF=yes
BRIDGE=br0
MACADDR=fe:ff:ff:ff:ff:ff
ARP=no

[-- Attachment #5: ifcfg-br0 --]
[-- Type: application/octet-stream, Size: 73 bytes --]

DEVICE=br0
TYPE=Bridge
BOOTPROTO=static
ONBOOT=yes
DELAY=0
STP=off

[-- Attachment #6: ifcfg-eth0 --]
[-- Type: application/octet-stream, Size: 105 bytes --]

DEVICE=eth0
BOOTPROTO=none
ONBOOT=yes
TYPE=Ethernet
MASTER=bond0
SLAVE=yes
NOZEROCONF=yes
ARP=no

[-- Attachment #7: ifcfg-eth1 --]
[-- Type: application/octet-stream, Size: 105 bytes --]

DEVICE=eth1
BOOTPROTO=none
ONBOOT=yes
TYPE=Ethernet
MASTER=bond0
SLAVE=yes
NOZEROCONF=yes
ARP=no

[-- Attachment #8: ifcfg-eth2 --]
[-- Type: application/octet-stream, Size: 55 bytes --]

DEVICE=eth2
BOOTPROTO=dhcp
ONBOOT=no
TYPE=Ethernet

[-- Attachment #9: ifcfg-veth0 --]
[-- Type: application/octet-stream, Size: 201 bytes --]

DEVICE=veth0
ONBOOT=yes
TYPE=Ethernet
BOOTPROTO=none
BROADCAST=192.168.0.255
IPADDR=192.168.0.20
NETMASK=255.255.255.0
NETWORK=192.168.0.0
NOZEROCONF=yes
MACADDR=aa:de:ad:be:ef:01
ARP=yes


[-- Attachment #10: ifcfg-vif0.0 --]
[-- Type: application/octet-stream, Size: 121 bytes --]

DEVICE=vif0.0
BOOTPROTO=none
ONBOOT=yes
TYPE=Ethernet
NOZEROCONF=yes
MACADDR=fe:ff:ff:ff:ff:ff
ARP=no
BRIDGE=br0

[-- Attachment #11: network --]
[-- Type: application/octet-stream, Size: 84 bytes --]

NETWORKING=yes
HOSTNAME=blue.lucidsolutions.co.nz
VLAN=yes
NETWORKING_IPV6=no


[-- Attachment #12: network-bridge-nop --]
[-- Type: application/octet-stream, Size: 1073 bytes --]

#!/bin/sh


# Exit if anything goes wrong.
set -e 

# First arg is the operation.
OP=$1
shift

# Pull variables in args in to environment.
for arg ; do export "${arg}" ; done

bridge=${bridge:-xen-br0}
netdev=${netdev:-eth0}
antispoof=${antispoof:-no}

echo "*network $OP bridge=$bridge netdev=$netdev antispoof=$antispoof" >&2

# Usage: show_status dev bridge
# Print ifconfig and routes.
show_status () {
    local dev=$1
    local bridge=$2
    
    echo '============================================================'
    ifconfig ${dev}
    ifconfig ${bridge}
    echo ' '
    ip route list
    echo ' '
    route -n
    echo '============================================================'
}

op_start () {
}

op_stop () {
}

case ${OP} in
    start)
        op_start
        ;;
    
    stop)
        op_stop
        ;;

    status)
        show_status ${netdev} ${bridge}
       ;;

    *)
       echo 'Unknown command: ' ${OP} >&2
       echo 'Valid commands are: start, stop, status' >&2
       exit 1
esac

[-- Attachment #13: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Network-bridge script with bonding and vlan - WORKAROUND

[Greg Brackley]

I've found I can get my VLAN configuration working by disabling tx checksum 
offload support on the domU eth0 interface (not tested, but I hope that this 
is going to also apply to the veth0 interface in dom0).  I had incorrectly 
focused my efforts on the dom0 networking. A small comment in 
http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=143 by Keir let me to 
this.

There still appears to be an underlying issue.  Is there any information 
that I could provide to help resolve this?

Now that I have basic connectivity working, I would be keen to increase the 
MTU up from 1500. Previously I read that there was a 4k page DMA limit for 
the MTU.  What is the current maximum MTU for the frontend/backend driver 
please?

Greg :-)