xen 2.0.1, 2.4.27, 2.6.9, non-bridge

xen 2.0.1, 2.4.27, 2.6.9, non-bridge

[Adam Heath]

We use nfsroot for all our xen-based machines.  Using pxegrub, and dhcp.

For xen 1.0 and 1.2, the built-in bridge configured by xen worked well for us,
and network had no problems.

For 2.0, the way the bridge is configured breaks the existing networking, upon
which nfsroot is running.

So, I've been attempting to configure the xenU instances by other means.

I've done similiar things with uml, by using proxy_arp and ip_forwarding.  I
turn on both options in /procy/sys(proxy_arp on all).  I then modified
/etc/xen/scripts/vif-bridge, to hard-code a few settings.  The vifX.X
interface is given a netmask of /32, a host-route for the ip address of the
domain(which I have hard-coded for now).

However, with *both* 2.4.27 *and* 2.6.9, I am getting kernel panics.  I have
not run the oops on 2.4 thru ksymoops.  The built in oops decode logic in
2.6.9 shows a null pointer in the arp_send routine.

Is there any low-level problem in xen itself to cause this?  I can give more
info if needed(got a serial console on it).

As a side node, it'd be nice if the network backend allowed for a pointopoint
topology, or the existing method.  Ie, I'd like it switchable.



-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt

Re: xen 2.0.1, 2.4.27, 2.6.9, non-bridge

[Derrik Pates]

Adam Heath wrote:
> For 2.0, the way the bridge is configured breaks the existing networking, upon
> which nfsroot is running.

I don't see how. The only thing I've run into is the fact that the 
included script that's supposed to configure up xen-br0 doesn't transfer 
the config properly to xen-br0 from eth0, and doesn't change eth0's IP 
to 0.0.0.0; I'm intending to just remove the script from my Xen systems, 
and just configure the bridge device the right way from the word go.

> However, with *both* 2.4.27 *and* 2.6.9, I am getting kernel panics.  I have
> not run the oops on 2.4 thru ksymoops.  The built in oops decode logic in
> 2.6.9 shows a null pointer in the arp_send routine.

Well, that's an interesting bug; while I'm not a developer, I'd suggest 
running the Oops text through ksymoops (with an appropriate System.map), 
and sending the decoded Oops to the list along with anything else relevant.

> As a side node, it'd be nice if the network backend allowed for a pointopoint
> topology, or the existing method.  Ie, I'd like it switchable.

Hack the script; there's nothing magical about the bridge setup, you can 
use traditional IP routing if you want. As pointed out in the 
documentation, the virtual network interfaces are simply like having 
NICs in between real systems, with a crossover cable interconnecting; 
you can do anything network-wise from there, the default assumption is 
simply that you'll want to start with bridging, because it's simple, and 
just like what most people are expecting.

-- 
Derrik Pates
dpates@dsdk12.net


-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt

Re: xen 2.0.1, 2.4.27, 2.6.9, non-bridge

[Adam Heath]

On Sat, 1 Jan 2005, Derrik Pates wrote:

> Adam Heath wrote:
> > For 2.0, the way the bridge is configured breaks the existing networking, upon
> > which nfsroot is running.
>
> I don't see how. The only thing I've run into is the fact that the
> included script that's supposed to configure up xen-br0 doesn't transfer
> the config properly to xen-br0 from eth0, and doesn't change eth0's IP
> to 0.0.0.0; I'm intending to just remove the script from my Xen systems,
> and just configure the bridge device the right way from the word go.

Configuring a bridge is a multi-step process.  At one point during the
process, normal communication is severed over eth0, while the bridge itself is
not yet fully functional.

At this point, the root filesystem is no longer available, and the machine
falls over.

I'd prefer to not do it with an initrd, as that's an added step, and extra
complexity.

> > However, with *both* 2.4.27 *and* 2.6.9, I am getting kernel panics.  I have
> > not run the oops on 2.4 thru ksymoops.  The built in oops decode logic in
> > 2.6.9 shows a null pointer in the arp_send routine.
>
> Well, that's an interesting bug; while I'm not a developer, I'd suggest
> running the Oops text through ksymoops (with an appropriate System.map),
> and sending the decoded Oops to the list along with anything else relevant.

I'll do that tomorrow, when I'm more resting.

For reference, here is the combinations I've tried.

xen 2.0.1(tarball)
2.4.27-xen0: both 2.4.27 and 2.6.9 xenU
2.6.9-xen0: 2.6.9 xenU

I haven't tried 2.4 U with 2.6 0, it's late, will try tomorrow.  I doubt that
it will have much affect, however.

> > As a side node, it'd be nice if the network backend allowed for a pointopoint
> > topology, or the existing method.  Ie, I'd like it switchable.
>
> Hack the script; there's nothing magical about the bridge setup, you can
> use traditional IP routing if you want. As pointed out in the
> documentation, the virtual network interfaces are simply like having
> NICs in between real systems, with a crossover cable interconnecting;
> you can do anything network-wise from there, the default assumption is
> simply that you'll want to start with bridging, because it's simple, and
> just like what most people are expecting.

Yeah, I've hacked the script.  As I said, it's doing a /32 netmask, and a host
route over the vifX.X interface.  If I turn off proxy_arp(but leave
ip_forwarding turned on), I can ping the hard-coded ip address(of the new
xenU) from dom0.  It's not until I enable proxy_arp that it falls over.


-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt

Re: xen 2.0.1, 2.4.27, 2.6.9, non-bridge

[Derrik Pates]

Adam Heath wrote:
> Configuring a bridge is a multi-step process.  At one point during the
> process, normal communication is severed over eth0, while the bridge itself is
> not yet fully functional.
> 
> At this point, the root filesystem is no longer available, and the machine
> falls over.

Okay, I see what you're doing. I think. Though I don't understand why it 
is that you'd be setting up a bridge in an unprivileged domain - once 
you're to the point of booting an unprivileged domain, the bridge setup 
should already be long done on the domain 0 side.

> I'd prefer to not do it with an initrd, as that's an added step, and extra
> complexity.

If you're doing something that requires that level of network 
reconfiguration while being booted via NFS root, that may be the only 
way to do it.

> Yeah, I've hacked the script.  As I said, it's doing a /32 netmask, and a host
> route over the vifX.X interface.  If I turn off proxy_arp(but leave
> ip_forwarding turned on), I can ping the hard-coded ip address(of the new
> xenU) from dom0.  It's not until I enable proxy_arp that it falls over.

I've used proxy ARP on Linux before (not with Xen specifically), so I'd 
say excluding a case where you're hitting a major bug, I'd have to see 
what commands you're running to set up the proxy ARP. This doesn't sound 
right to me. It shouldn't affect the ability of the system doing the 
proxy ARP to resolve the target IP address to the target MAC.

-- 
Derrik Pates
dpates@dsdk12.net


-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt

Re: xen 2.0.1, 2.4.27, 2.6.9, non-bridge

[Keir Fraser]

> However, with *both* 2.4.27 *and* 2.6.9, I am getting kernel panics.  I have
> not run the oops on 2.4 thru ksymoops.  The built in oops decode logic in
> 2.6.9 shows a null pointer in the arp_send routine.

Sounds like a bug to me. We're interested in seeing the backtraces and
getting access to your xen-syms/vmlinux-syms files.

> As a side node, it'd be nice if the network backend allowed for a pointopoint
> topology, or the existing method.  Ie, I'd like it switchable.

The network backend driver knows nothing about the bridge. It's all
encoded in the two xend scripts, so just hack those. If you come up
with something generically useful perhaps we can integrate the
functionality into the main scripts.

 Cheers,
 Keir


-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt

RE: xen 2.0.1, 2.4.27, 2.6.9, non-bridge

[James Harper]

> Configuring a bridge is a multi-step process.  At one point during the
> process, normal communication is severed over eth0, while the bridge
> itself is
> not yet fully functional.
> At this point, the root filesystem is no longer available, and the
machine
> falls over.

I wonder if you could do this:

1. mount a tmpfs somewhere
2. copy in enough libs etc to make #4 work
3. chroot to the new tmpfs
4. set up the bridge and transfer the address across
5. leave the chroot environment

You wouldn't want anything much else to be running at this point though
so it would have to be done pretty early in the boot process.

> I'd prefer to not do it with an initrd, as that's an added step, and
extra
> complexity.

I've set up an initrd that boots dom0 over iSCSI, which suffers from the
same problem as nfs only more so as iSCSI is connection based and
downing eth0 cuts the connections. My initrd sets up the first bridge
(my setup has 3 bridges) from the outset so the network disappearing
never becomes an issue.

Under iSCSI an initrd is the only option but for nfs it would be nifty
to be able to do without.

If anyone's interested in the script to create the initrd (debian - not
sure how it would translate to another dist) I can post it here,
shouldn't be too hard to modify it to boot from nfs... in fact my home
pc's boot off nfs using almost the same setup.

James



-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt

Re: xen 2.0.1, 2.4.27, 2.6.9, non-bridge

[Ronald G. Minnich]

On Fri, 31 Dec 2004, Adam Heath wrote:

> For 2.0, the way the bridge is configured breaks the existing
> networking, upon which nfsroot is running.

how did 2.0 break the existing networking? Sounds like I might be having 
the same problem you are. All my bridging was fine on 1.3 and won't work 
at all on 2.0.

ron


-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt

Re: xen 2.0.1, 2.4.27, 2.6.9, non-bridge

[Adam Heath]

On Sat, 1 Jan 2005, Derrik Pates wrote:

> Okay, I see what you're doing. I think. Though I don't understand why it
> is that you'd be setting up a bridge in an unprivileged domain - once
> you're to the point of booting an unprivileged domain, the bridge setup
> should already be long done on the domain 0 side.

I'm not setting up a bridge in an unprivledged domain.  There is no bridge
anywhere.

> I've used proxy ARP on Linux before (not with Xen specifically), so I'd
> say excluding a case where you're hitting a major bug, I'd have to see
> what commands you're running to set up the proxy ARP. This doesn't sound
> right to me. It shouldn't affect the ability of the system doing the
> proxy ARP to resolve the target IP address to the target MAC.

I'll be trying plain 2.4 and 2.6 nfsroot as well in a bit.



-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt

RE: xen 2.0.1, 2.4.27, 2.6.9, non-bridge

[Adam Heath]

On Sat, 1 Jan 2005, James Harper wrote:

> > Configuring a bridge is a multi-step process.  At one point during the
> > process, normal communication is severed over eth0, while the bridge
> > itself is
> > not yet fully functional.
> > At this point, the root filesystem is no longer available, and the
> machine
> > falls over.
>
> I wonder if you could do this:
>
> 1. mount a tmpfs somewhere
> 2. copy in enough libs etc to make #4 work
> 3. chroot to the new tmpfs
> 4. set up the bridge and transfer the address across
> 5. leave the chroot environment

I  had a version that did that.

nfsroot a small bootstrap system.
copy root to tmpfs.
pivot_root
setup bridge
start portmap
mount real nfsroot
kill portmap
pivot_root
continue boot

All steps worked, except that after continuing the boot, I kept getting odd
issues out no free locks.  I got fustrated, and trying the ip routing
approach.

Note that the above does not use an initrd.  Using a real nfsroot system makes
it easier to update.



-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt

Re: xen 2.0.1, 2.4.27, 2.6.9, non-bridge

[Adam Heath]

On Sat, 1 Jan 2005, Ronald G. Minnich wrote:

>
>
> On Fri, 31 Dec 2004, Adam Heath wrote:
>
> > For 2.0, the way the bridge is configured breaks the existing
> > networking, upon which nfsroot is running.
>
> how did 2.0 break the existing networking? Sounds like I might be having
> the same problem you are. All my bridging was fine on 1.3 and won't work
> at all on 2.0.

The bridging for pre 2.0 was done in xen, outside of our control.  For 2.0,
the networking is working before the bridge is configured.


-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt

Re: xen 2.0.1, 2.4.27, 2.6.9, non-bridge

[Adam Heath]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 612 bytes --]

On Fri, 31 Dec 2004, Adam Heath wrote:

> However, with *both* 2.4.27 *and* 2.6.9, I am getting kernel panics.  I have
> not run the oops on 2.4 thru ksymoops.  The built in oops decode logic in
> 2.6.9 shows a null pointer in the arp_send routine.

Attached you'll find the oops and ksymoops output for a 2.4 dom0 kernel.

The oops for 2.6 is very similiar.

I have to enable *both* proxy_arp and ip_forward for this to happen.  If I
only enable ip_forward, then gateway route the ip of xenU, the pings enter
vifX.X(as shown by tcpdump), but they don't return, as dom0 is not arping the
machine doing the ping.

[-- Attachment #2: Type: TEXT/PLAIN, Size: 1106 bytes --]

Unable to handle kernel NULL pointer dereference at virtual address 00000000
 printing eip:
c01f2fbd
*pde=00000000(55555000)
Oops: 0000
CPU:    0
EIP:    0819:[<c01f2fbd>]    Not tainted
EFLAGS: 00211282
eax: 00000000   ebx: c1973740   ecx: 00000004   edx: c1441560
esi: 00000000   edi: c1441400   ebp: c02a3c64   esp: c02a3c40
ds: 0821   es: 0821   ss: 0821
Process swapper (pid: 0, stackpage=c02a3000)<1>
Stack: c02a3c7c c022dd9a 00000001 00000806 0502a8c0 c1441560 c1441400 c1973740 
       c1441400 c02a3c7c c01fc71d c1973740 c1441400 00000806 00000000 c02a3cbc 
       c022d5e8 c1973740 00000806 0502a8c0 c1441400 0402a8c0 00000000 c144147c 
Call Trace: [<c022dd9a>] [<c01fc71d>] [<c022d5e8>] [<c020192d>] [<c02021ec>] 
   [<c02105f0>] [<c0210530>] [<c0204b77>] [<c0210530>] [<c021050a>] [<c0210530>] 
   [<c0204805>] [<c020d768>] [<c0204b77>] [<c020d720>] [<c020d667>] [<c020d720>] 
   [<c020c6a9>] [<c020c500>] [<c0204b77>] [<c020c500>] [<c020c314>] [<c020c500>] 
   [<c01fce29>] [<c01fcf96>] [<c01fd0d3>] [<c010d449>] [<c0194db9>] [<c0198dce>] 
   [<c01938c7>] [<c0191b22>] 


[-- Attachment #3: Type: TEXT/PLAIN, Size: 3417 bytes --]

ksymoops 2.4.9 on i686 2.4.27-xen0.  Options used
     -V (default)
     -k /proc/ksyms (default)
     -l /proc/modules (default)
     -o /lib/modules/2.4.27-xen0/ (default)
     -m /boot/System.map-2.4.27-xen0 (default)

Warning: You did not tell me where to find symbol information.  I will
assume that the log matches the kernel and modules that are running
right now and I'll use the default options above for symbol resolution.
If the current kernel and/or modules do not match the log, you can get
more accurate output by telling me the kernel version and where to find
map, modules, ksyms etc.  ksymoops -h explains the options.

No modules in ksyms, skipping objects
Warning (read_lsmod): no symbols in lsmod, is /proc/modules a valid lsmod file?
Unable to handle kernel NULL pointer dereference at virtual address 00000000
c01f2fbd
Oops: 0000
CPU:    0
EIP:    0819:[<c01f2fbd>]    Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00211282
eax: 00000000   ebx: c1973740   ecx: 00000004   edx: c1441560
esi: 00000000   edi: c1441400   ebp: c02a3c64   esp: c02a3c40
ds: 0821   es: 0821   ss: 0821
Process swapper (pid: 0, stackpage=c02a3000)<1>
Stack: c02a3c7c c022dd9a 00000001 00000806 0502a8c0 c1441560 c1441400 c1973740 
       c1441400 c02a3c7c c01fc71d c1973740 c1441400 00000806 00000000 c02a3cbc 
       c022d5e8 c1973740 00000806 0502a8c0 c1441400 0402a8c0 00000000 c144147c 
Call Trace: [<c022dd9a>] [<c01fc71d>] [<c022d5e8>] [<c020192d>] [<c02021ec>] 
   [<c02105f0>] [<c0210530>] [<c0204b77>] [<c0210530>] [<c021050a>] [<c0210530>] 
   [<c0204805>] [<c020d768>] [<c0204b77>] [<c020d720>] [<c020d667>] [<c020d720>] 
   [<c020c6a9>] [<c020c500>] [<c0204b77>] [<c020c500>] [<c020c314>] [<c020c500>] 
   [<c01fce29>] [<c01fcf96>] [<c01fd0d3>] [<c010d449>] [<c0194db9>] [<c0198dce>] 
   [<c01938c7>] [<c0191b22>] 
Warning (Oops_read): Code line not seen, dumping what data is available


>>EIP; c01f2fbd <netif_be_start_xmit+1d/200>   <=====

>>ebp; c02a3c64 <init_task_union+1c64/2000>
>>esp; c02a3c40 <init_task_union+1c40/2000>

Trace; c022dd9a <arp_send+6a/90>
Trace; c01fc71d <dev_queue_xmit+9d/280>
Trace; c022d5e8 <arp_solicit+e8/1d0>
Trace; c020192d <__neigh_event_send+ed/1f0>
Trace; c02021ec <neigh_resolve_output+1bc/1f0>
Trace; c02105f0 <ip_finish_output2+c0/100>
Trace; c0210530 <ip_finish_output2+0/100>
Trace; c0204b77 <nf_hook_slow+b7/190>
Trace; c0210530 <ip_finish_output2+0/100>
Trace; c021050a <ip_finish_output+14a/150>
Trace; c0210530 <ip_finish_output2+0/100>
Trace; c0204805 <nf_iterate+65/a0>
Trace; c020d768 <ip_forward_finish+48/60>
Trace; c0204b77 <nf_hook_slow+b7/190>
Trace; c020d720 <ip_forward_finish+0/60>
Trace; c020d667 <ip_forward+1b7/270>
Trace; c020d720 <ip_forward_finish+0/60>
Trace; c020c6a9 <ip_rcv_finish+1a9/200>
Trace; c020c500 <ip_rcv_finish+0/200>
Trace; c0204b77 <nf_hook_slow+b7/190>
Trace; c020c500 <ip_rcv_finish+0/200>
Trace; c020c314 <ip_rcv+384/3f0>
Trace; c020c500 <ip_rcv_finish+0/200>
Trace; c01fce29 <netif_receive_skb+139/210>
Trace; c01fcf96 <process_backlog+96/150>
Trace; c01fd0d3 <net_rx_action+83/160>
Trace; c010d449 <do_softirq+e9/100>
Trace; c0194db9 <do_IRQ+b9/e0>
Trace; c0198dce <evtchn_do_upcall+ae/110>
Trace; c01938c7 <hypervisor_callback+33/49>
Trace; c0191b22 <cpu_idle+72/c0>


3 warnings issued.  Results may not be reliable.

Re: xen 2.0.1, 2.4.27, 2.6.9, non-bridge

[Adam Heath]

On Sat, 1 Jan 2005, Adam Heath wrote:

> On Fri, 31 Dec 2004, Adam Heath wrote:
>
> > However, with *both* 2.4.27 *and* 2.6.9, I am getting kernel panics.  I have
> > not run the oops on 2.4 thru ksymoops.  The built in oops decode logic in
> > 2.6.9 shows a null pointer in the arp_send routine.
>
> Attached you'll find the oops and ksymoops output for a 2.4 dom0 kernel.
>
> The oops for 2.6 is very similiar.
>
> I have to enable *both* proxy_arp and ip_forward for this to happen.  If I
> only enable ip_forward, then gateway route the ip of xenU, the pings enter
> vifX.X(as shown by tcpdump), but they don't return, as dom0 is not arping the
> machine doing the ping.

00000160 <netif_be_start_xmit>:
     160:       55                      push   %ebp
     161:       89 e5                   mov    %esp,%ebp
     163:       57                      push   %edi
     164:       56                      push   %esi
     165:       53                      push   %ebx
     166:       83 ec 18                sub    $0x18,%esp
     169:       8b 45 0c                mov    0xc(%ebp),%eax
     16c:       8b 5d 08                mov    0x8(%ebp),%ebx
     16f:       8b 40 6c                mov    0x6c(%eax),%eax
     172:       89 c2                   mov    %eax,%edx
     174:       89 45 f0                mov    %eax,0xfffffff0(%ebp)
     177:       8b 40 1c                mov    0x1c(%eax),%eax
     17a:       8b 72 20                mov    0x20(%edx),%esi
     17d:       3b 30                   cmp    (%eax),%esi
     17f:       0f 84 6b 01 00 00       je     2f0 <netif_be_start_xmit+0x190>
     185:       8b 7a 24                mov    0x24(%edx),%edi
     188:       89 f0                   mov    %esi,%eax
     18a:       29 f8                   sub    %edi,%eax
     18c:       3d 00 01 00 00          cmp    $0x100,%eax
     191:       0f 84 59 01 00 00       je     2f0 <netif_be_start_xmit+0x190>
     197:       8b 43 74                mov    0x74(%ebx),%eax
     19a:       48                      dec    %eax
     19b:       0f 85 a2 01 00 00       jne    343 <netif_be_start_xmit+0x1e3>
     1a1:       80 7b 6d 00             cmpb   $0x0,0x6d(%ebx)
     1a5:       0f 84 68 01 00 00       je     313 <netif_be_start_xmit+0x1b3>
     1ab:       8b 83 8c 00 00 00       mov    0x8c(%ebx),%eax
     1b1:       8b 00                   mov    (%eax),%eax
     1b3:       48                      dec    %eax
     1b4:       0f 84 59 01 00 00       je     313 <netif_be_start_xmit+0x1b3>
     1ba:       8b 83 80 00 00 00       mov    0x80(%ebx),%eax
     1c0:       89 45 ec                mov    %eax,0xffffffec(%ebp)
     1c3:       8d b6 00 00 00 00       lea    0x0(%esi),%esi

17d is the faulting instruction, so eax is null.

>From looking at the source and other lines(esp. the je 313, which is the first
condition that does a goto drop), eax contains dev->priv, which this function
assumes is a netif_t.

So, something is not setting ->priv.


-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt

Re: xen 2.0.1, 2.4.27, 2.6.9, non-bridge

[Adam Heath]

On Sat, 1 Jan 2005, Adam Heath wrote:

> On Sat, 1 Jan 2005, Adam Heath wrote:
>
> > On Fri, 31 Dec 2004, Adam Heath wrote:
> >
> > > However, with *both* 2.4.27 *and* 2.6.9, I am getting kernel panics.  I have
> > > not run the oops on 2.4 thru ksymoops.  The built in oops decode logic in
> > > 2.6.9 shows a null pointer in the arp_send routine.
> >
> > Attached you'll find the oops and ksymoops output for a 2.4 dom0 kernel.
> >
> > The oops for 2.6 is very similiar.
> >
> > I have to enable *both* proxy_arp and ip_forward for this to happen.  If I
> > only enable ip_forward, then gateway route the ip of xenU, the pings enter
> > vifX.X(as shown by tcpdump), but they don't return, as dom0 is not arping the
> > machine doing the ping.

Hrm.  If I turn on proxy_arp for all interfaces, I get this error.  If I turn
it on only for eth0, then in vif-bridge(the name of which I haven't changed)
turn it on for vifX.X, then this system works for me.

I can now nfsroot a xen0 and xenU kernel, without using bridging.

I still think there is a bug tho.  Something is not setting dev->priv.



-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt

Re: xen 2.0.1, 2.4.27, 2.6.9, non-bridge

[Adam Heath]

On Sat, 1 Jan 2005, Adam Heath wrote:

> On Sat, 1 Jan 2005, Adam Heath wrote:
>
> > On Sat, 1 Jan 2005, Adam Heath wrote:
> >
> > > On Fri, 31 Dec 2004, Adam Heath wrote:
> > >
> > > > However, with *both* 2.4.27 *and* 2.6.9, I am getting kernel panics.  I have
> > > > not run the oops on 2.4 thru ksymoops.  The built in oops decode logic in
> > > > 2.6.9 shows a null pointer in the arp_send routine.
> > >
> > > Attached you'll find the oops and ksymoops output for a 2.4 dom0 kernel.
> > >
> > > The oops for 2.6 is very similiar.
> > >
> > > I have to enable *both* proxy_arp and ip_forward for this to happen.  If I
> > > only enable ip_forward, then gateway route the ip of xenU, the pings enter
> > > vifX.X(as shown by tcpdump), but they don't return, as dom0 is not arping the
> > > machine doing the ping.
>
> Hrm.  If I turn on proxy_arp for all interfaces, I get this error.  If I turn
> it on only for eth0, then in vif-bridge(the name of which I haven't changed)
> turn it on for vifX.X, then this system works for me.
>
> I can now nfsroot a xen0 and xenU kernel, without using bridging.
>
> I still think there is a bug tho.  Something is not setting dev->priv.

Ok, more info.

With proxy_arp turned off on eth0, but on for vifX.X, and using a gateway
route for the ip address of xenU, I can ping xenU.

If I turn on proxy_arp for eth0, so other machines on the lan can ping xenU,
then I get a panic.


-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt

Re: xen 2.0.1, 2.4.27, 2.6.9, non-bridge

[Keir Fraser]

> Ok, more info.
> 
> With proxy_arp turned off on eth0, but on for vifX.X, and using a gateway
> route for the ip address of xenU, I can ping xenU.
> 
> If I turn on proxy_arp for eth0, so other machines on the lan can ping xenU,
> then I get a panic.

I think I see the problem. I'll try to clean up netback and fix the
bug before I go on holiday tomorrow.

 -- Keir


-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt

RE: xen 2.0.1, 2.4.27, 2.6.9, non-bridge

[Ian Pratt]

> As a side node, it'd be nice if the network backend allowed 
> for a pointopoint topology, or the existing method.  Ie, I'd 
> like it switchable.

Using routing rather than bridging works just fine. It's arguable that
this would be a better default. (Other than the need for a DHCP relay).

We should probably include alternatives to /etc/xen/scripts/vif-bridge
and /etc/xen/scripts/network that demonstrate how to set this up.

Basically, in /etc/xen/scripts/vif-router : 

 enable proxy_arp on vifX
 give vifX some (/32) IP addr e.g. 10.10.10.vifX
 ip route add <ip_dom1> dev vifX

Your /etc/xen/scripts/network-routing script just needs to enable ip
forwarding (if it isn't already).

[It may be possible to configure the VIF as a point-to-point interface,
and hence avoid the need for the 10.x address at all. If anyone has this
working I'd be interested to hear.]

If anyone has any decent example scripts I'd like to see them documented
and go into the repo.

Cheers,
Ian


-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt

RE: xen 2.0.1, 2.4.27, 2.6.9, non-bridge

[Adam Heath]

On Wed, 5 Jan 2005, Ian Pratt wrote:

> > As a side node, it'd be nice if the network backend allowed
> > for a pointopoint topology, or the existing method.  Ie, I'd
> > like it switchable.
>
> Using routing rather than bridging works just fine. It's arguable that
> this would be a better default. (Other than the need for a DHCP relay).
>
> We should probably include alternatives to /etc/xen/scripts/vif-bridge
> and /etc/xen/scripts/network that demonstrate how to set this up.
>
> Basically, in /etc/xen/scripts/vif-router :
>
>  enable proxy_arp on vifX
>  give vifX some (/32) IP addr e.g. 10.10.10.vifX
>  ip route add <ip_dom1> dev vifX
>
> Your /etc/xen/scripts/network-routing script just needs to enable ip
> forwarding (if it isn't already).
>
> [It may be possible to configure the VIF as a point-to-point interface,
> and hence avoid the need for the 10.x address at all. If anyone has this
> working I'd be interested to hear.]
>
> If anyone has any decent example scripts I'd like to see them documented
> and go into the repo.

xen-0:/etc/xen# tail -n 3 /etc/xen/scripts/vif-route
ifconfig $vif 10.100.2.0 netmask 255.255.255.255
echo 1 > /proc/sys/net/ipv4/conf/$vif/proxy_arp
route add -host $domain dev $vif
--
The first part of the file is the standard boiler plate.

10.100.2.0 is the ip address of eth0.
$domain is an entry in our dns.

Note, that one must *always* specify the bridge parameter in the config, and
you can't specifiy additional parameters to be passed to this script.  That's
why I was forced to use the $domain.

If one wanted to use a dhcp relay on dom0, then the mac addresses of the
vifX.X interfaces *must* be unique.  With the above setup, that is not a
requirement.



-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt

RE: xen 2.0.1, 2.4.27, 2.6.9, non-bridge

[Ian Pratt]

> Note, that one must *always* specify the bridge parameter in 
> the config, and
> you can't specifiy additional parameters to be passed to this 
> script.  That's
> why I was forced to use the $domain.

Can't you replace the bridge-name parameter with the hostname or IP
address of the domain? (I haven't tried this). Adding a means of passing
extra parameters shouldn't be too hard anyhow.
 
> If one wanted to use a dhcp relay on dom0, then the mac 
> addresses of the
> vifX.X interfaces *must* be unique.  With the above setup, 
> that is not a
> requirement.

Do you use DHCP to assign (static) addresses in your configuration, or
configure the address on the domain's command line? 

Assignment of dynamic DHCP addresses is always going to be messy, but
fortunately its not a very useful thing to support. 

Ian


-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt

Re: xen 2.0.1, 2.4.27, 2.6.9, non-bridge

[Jody Belka]

On Wed, Jan 05, 2005 at 12:28:45AM -0000, Ian Pratt wrote:
> Using routing rather than bridging works just fine. It's arguable that
> this would be a better default. (Other than the need for a DHCP relay).
> 
> We should probably include alternatives to /etc/xen/scripts/vif-bridge
> and /etc/xen/scripts/network that demonstrate how to set this up.

I use a mix of routing and bridging myself.

I treat dom0 as a router connected to a switch. eth0, my physical nic,
is on subnet A, and xen-br0 (which i create at boot, so i disable the
xend network script) is on subnet B. I then just enable ip forwarding,
and hook new domains up to xen-br0 as normal. Works really well.


J

-- 
Jody Belka
knew (at) pimb (dot) org


-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt

RE: xen 2.0.1, 2.4.27, 2.6.9, non-bridge

[Ian Pratt]

> I use a mix of routing and bridging myself.
> 
> I treat dom0 as a router connected to a switch. eth0, my physical nic,
> is on subnet A, and xen-br0 (which i create at boot, so i disable the
> xend network script) is on subnet B. I then just enable ip forwarding,
> and hook new domains up to xen-br0 as normal. Works really well.

Interesting. Any particular reason why you do this? Using iptables
netfilter I guess you could probably get the same effect with just a
bridge.

Since the hosts main IP addresses are presumably on subnet A, I don't
see why the bridge couldn't be brought up by the network script -- you
just don't want to do the IP address transfer, so remove the
transfer_addresses and transfer_routes lines.

Cheers,
Ian


-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt

Re: xen 2.0.1, 2.4.27, 2.6.9, non-bridge

[Jody Belka]

On Wed, Jan 05, 2005 at 01:30:55AM -0000, Ian Pratt wrote:
> > I use a mix of routing and bridging myself.
> > 
> > I treat dom0 as a router connected to a switch. eth0, my physical nic,
> > is on subnet A, and xen-br0 (which i create at boot, so i disable the
> > xend network script) is on subnet B. I then just enable ip forwarding,
> > and hook new domains up to xen-br0 as normal. Works really well.
> 
> Interesting. Any particular reason why you do this? Using iptables
> netfilter I guess you could probably get the same effect with just a
> bridge.

Well, although it's not running yet, i'm intending on running OSPF on
both my xen boxes and my router. Unsure if that would play well or not
with using netfilter for it, whereas i know this should work.

> Since the hosts main IP addresses are presumably on subnet A, I don't
> see why the bridge couldn't be brought up by the network script -- you
> just don't want to do the IP address transfer, so remove the
> transfer_addresses and transfer_routes lines.

Well, it's nice to manage the networking through the system scripts.
Keeps the xen-br0 configuration together with that of eth0. Also i'm
assigning an IP to the bridge, as I need an IP on subnet B to act as the
gateway addr anyway, and it also leads to the routing "just working".


J

-- 
Jody Belka
knew (at) pimb (dot) org


-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt

RE: xen 2.0.1, 2.4.27, 2.6.9, non-bridge

[Adam Heath]

On Wed, 5 Jan 2005, Ian Pratt wrote:

> Do you use DHCP to assign (static) addresses in your configuration, or
> configure the address on the domain's command line?

Static.  With 1.0 and 1.2, we used dhcp to assign static addresses, but that
meant we had static mac addresses in the config.


-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt