domU networking trouble in Xen 3.0-unstable.hg

domU networking trouble in Xen 3.0-unstable.hg

[Bryan D. Payne]

[-- Attachment #1.1: Type: text/plain, Size: 7604 bytes --]

I sent this message to xen-users, but haven't received any replies.   
Perhaps since it is an issue with the pre-3.0 code, this is a better  
forum for my question / problem.  Thanks!

----------

I'm just getting started with Xen and am having some trouble getting  
networking up and running in domU.  The symptoms are that packets can  
leave domU and go through xen-br0, but packets are not getting back  
into domU.  Specifically, this is what I'm seeing when sending a ping  
from within domU:

Note: domU is 192.168.0.12
Note: dom0 is 192.168.0.11

[root@domU ~]# tcpdump -i eth0
tcpdump: verbose output suppressed, use -v or -vv for full protocol  
decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
19:48:48.173992 arp who-has 192.168.0.11 tell 192.168.0.12
19:48:49.173981 arp who-has 192.168.0.11 tell 192.168.0.12
19:48:50.173981 arp who-has 192.168.0.11 tell 192.168.0.12
19:48:52.183980 arp who-has 192.168.0.11 tell 192.168.0.12
19:48:53.183991 arp who-has 192.168.0.11 tell 192.168.0.12

Watching this ping from dom0 shows a different story:

[root@dom0 ~]# tcpdump -i vif1.1
tcpdump: WARNING: vif1.1: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol  
decode
listening on vif1.1, link-type EN10MB (Ethernet), capture size 96 bytes
19:50:41.780777 arp who-has 192.168.0.11 tell 192.168.0.12
19:50:41.969751 arp reply 192.168.0.11 is-at 00:a0:c9:e3:75:5e
19:50:42.900698 arp who-has 192.168.0.11 tell 192.168.0.12
19:50:42.900728 arp reply 192.168.0.11 is-at 00:a0:c9:e3:75:5e
19:50:43.900690 arp who-has 192.168.0.11 tell 192.168.0.12
19:50:43.900717 arp reply 192.168.0.11 is-at 00:a0:c9:e3:75:5e
19:50:44.900677 arp who-has 192.168.0.11 tell 192.168.0.12
19:50:44.900705 arp reply 192.168.0.11 is-at 00:a0:c9:e3:75:5e
19:50:46.910656 arp who-has 192.168.0.11 tell 192.168.0.12
19:50:46.910685 arp reply 192.168.0.11 is-at 00:a0:c9:e3:75:5e

It looks like domU is sending out the arp request, but never getting  
the arp reply.  However, clearly the arp reply is being sent by  
dom0...it seems to be getting lost somewhere in the connection  
between vif1.1 and the domU/eth0.  I'm including some additional  
configuration information about my setup below.  If additional  
information would be helpful, just let me know.

Does anyone have thoughts about how to solve this?

Thanks,
bryan


========== DOMU Information ====================
[root@domU ~]# ifconfig
eth0      Link encap:Ethernet  HWaddr AA:00:00:3D:B1:DD
           inet addr:192.168.0.12  Bcast:192.168.0.255  Mask: 
255.255.255.0
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
           TX packets:44 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:1000
           RX bytes:0 (0.0 b)  TX bytes:1848 (1.8 KiB)

lo        Link encap:Local Loopback
           inet addr:127.0.0.1  Mask:255.0.0.0
           UP LOOPBACK RUNNING  MTU:16436  Metric:1
           RX packets:39 errors:0 dropped:0 overruns:0 frame:0
           TX packets:39 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:4368 (4.2 KiB)  TX bytes:4368 (4.2 KiB)

[root@domU ~]# netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window   
irtt Iface
192.168.0.0     0.0.0.0         255.255.255.0   U         0  
0          0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U         0  
0          0 eth0
0.0.0.0         192.168.0.254   0.0.0.0         UG        0  
0          0 eth0
[root@domU ~]# ping 192.168.0.11
PING 192.168.0.11 (192.168.0.11) 56(84) bytes of data.
 From 192.168.0.12 icmp_seq=0 Destination Host Unreachable
 From 192.168.0.12 icmp_seq=1 Destination Host Unreachable
 From 192.168.0.12 icmp_seq=2 Destination Host Unreachable
 From 192.168.0.12 icmp_seq=3 Destination Host Unreachable
 From 192.168.0.12 icmp_seq=4 Destination Host Unreachable
 From 192.168.0.12 icmp_seq=5 Destination Host Unreachable

--- 192.168.0.11 ping statistics ---
7 packets transmitted, 0 received, +6 errors, 100% packet loss, time  
6055ms
, pipe 4
[root@domU ~]#


========== DOM0 Information ====================
[root@dom0 ~]# xm list
Name              ID  Mem(MiB)  CPU  VCPUs  State   Time(s)
Domain-0           0      1500    0      1  r-----     85.7
domU               1       255    0      1  -b----      8.7
[root@dom0 ~]# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:A0:C9:E3:75:5E
           inet addr:192.168.0.11  Bcast:192.168.0.255  Mask: 
255.255.255.0
           inet6 addr: fe80::2a0:c9ff:fee3:755e/64 Scope:Link
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:221 errors:0 dropped:0 overruns:0 frame:0
           TX packets:224 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:66104 (64.5 KiB)  TX bytes:34473 (33.6 KiB)

lo        Link encap:Local Loopback
           inet addr:127.0.0.1  Mask:255.0.0.0
           inet6 addr: ::1/128 Scope:Host
           UP LOOPBACK RUNNING  MTU:16436  Metric:1
           RX packets:135 errors:0 dropped:0 overruns:0 frame:0
           TX packets:135 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:10210 (9.9 KiB)  TX bytes:10210 (9.9 KiB)

peth0     Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
           inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
           UP BROADCAST RUNNING NOARP MULTICAST  MTU:1500  Metric:1
           RX packets:215 errors:0 dropped:0 overruns:0 frame:0
           TX packets:308 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:1000
           RX bytes:71438 (69.7 KiB)  TX bytes:41704 (40.7 KiB)

vif0.0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
           inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:224 errors:0 dropped:0 overruns:0 frame:0
           TX packets:222 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:34473 (33.6 KiB)  TX bytes:66194 (64.6 KiB)

vif1.1    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
           inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:53 errors:0 dropped:0 overruns:0 frame:0
           TX packets:0 errors:0 dropped:89 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:2226 (2.1 KiB)  TX bytes:0 (0.0 b)

xen-br0   Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
           inet6 addr: fe80::200:ff:fe00:0/64 Scope:Link
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:102 errors:0 dropped:0 overruns:0 frame:0
           TX packets:5 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:9297 (9.0 KiB)  TX bytes:378 (378.0 b)

[root@dom0 ~]# netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window   
irtt Iface
192.168.0.0     0.0.0.0         255.255.255.0   U         0  
0          0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U         0  
0          0 eth0
0.0.0.0         192.168.0.254   0.0.0.0         UG        0  
0          0 eth0
[root@dom0 ~]# cat /etc/xen/domU
kernel = "/boot/vmlinuz-2.6.12-xenU"
memory = 256
name = "domU"
nics=1
disk = ['file:/opt/xen_images/domU.img,sda1,w' ]
root = "/dev/sda1"
extra = "ro selinux=0 3"
[root@dom0 ~]#
====================



[-- Attachment #1.2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 2365 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: domU networking trouble in Xen 3.0-unstable.hg

[Nivedita Singhvi]

Bryan D. Payne wrote:
> I sent this message to xen-users, but haven't received any replies.   
> Perhaps since it is an issue with the pre-3.0 code, this is a better  
> forum for my question / problem.  Thanks!

I presume you are downloading from Xen directly and using their
default config scripts? Can you turn IPV6 off?

thanks,
Nivedita

Re: domU networking trouble in Xen 3.0-unstable.hg

[Ewan Mellor]

On Thu, Oct 27, 2005 at 09:03:09AM -0400, Bryan D. Payne wrote:

> [Snip]
>
> It looks like domU is sending out the arp request, but never getting  
> the arp reply.  However, clearly the arp reply is being sent by  
> dom0...it seems to be getting lost somewhere in the connection  
> between vif1.1 and the domU/eth0.

That's pretty peculiar, Bryan.  We're having problems with the network
configuration scripts at the moment, but it looks to me like your network is
intact.  If the packet has reached vif1.1, then it really ought to reach the
domU, because that is a pretty simple connection, and well tested.  Do you
have any firewalling / iptables rules set up in the guest domain?  Can it
receive pings from elsewhere on the network?  If it's not firewall rules, I'm
flummoxed, and you ought to file a bug.

Cheers,

Ewan.

Re: domU networking trouble in Xen 3.0-unstable.hg

[Bryan D. Payne]

[-- Attachment #1.1.1: Type: text/plain, Size: 1668 bytes --]

Interesting.  Thanks for this reply and thanks to the people that  
have replied off the list thus far.  I had to step away from this for  
a couple of days, but I'm going to be working on it again tomorrow.   
Based on the feedback that I have received, here's the plan of attack:

* I'll provide some additional info to the list (e.g., ip addr show,  
brctl show, ip route show...each for both dom0 and domU)

* I'll double check that IPv6 is disabled...but I believe that it is

* I'll update to the latest changeset and try everything again

And, if it's still not working at that point, then I'll go ahead with  
filing the bug report.  I agree that it seemed like an unusual place  
to have problems...so hopefully we can get to the bottom of it.  And  
I'm happy to pull any other information out of the machine that may  
prove useful for debugging and such as well.  Just let me know if  
there's anything in particular that would be useful.

Cheers,
bryan



> That's pretty peculiar, Bryan.  We're having problems with the network
> configuration scripts at the moment, but it looks to me like your  
> network is
> intact.  If the packet has reached vif1.1, then it really ought to  
> reach the
> domU, because that is a pretty simple connection, and well tested.   
> Do you
> have any firewalling / iptables rules set up in the guest domain?   
> Can it
> receive pings from elsewhere on the network?  If it's not firewall  
> rules, I'm
> flummoxed, and you ought to file a bug.
>
> Cheers,
>
> Ewan.
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>


[-- Attachment #1.1.2: Type: text/html, Size: 4852 bytes --]

[-- Attachment #1.2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 2365 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: domU networking trouble in Xen 3.0-unstable.hg

[Bryan D. Payne]

[-- Attachment #1.1: Type: text/plain, Size: 3263 bytes --]

I have just tried to upgrade to changeset 7572.  I discovered that  
doing a 'make install' after my update blew away my dom0 and domU  
kernel configs!  Unfortunately, I didn't have those saved off  
anywhere (my mistake), so I have spent the past four hours (and  
counting...) trying to recreate a working config.  I shall save this  
one when I finally get it working, but I'd like to suggest that  
perhaps a make install doesn't blow away .config* in the kernel  
source directories ;-)

Anyway, in the mean time, here's some additional information about my  
setup before attempting the upgrade.

* When I first posted about this problem I was using changeset 7398.

* The additional network config information that people requested is  
below:

====================
[root@domU ~]# ip addr show
1: lo: <LOOPBACK,UP> mtu 16436 qdisc noqueue
     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
     inet 127.0.0.1/8 scope host lo
2: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 1000
     link/ether aa:00:00:3d:b1:dd brd ff:ff:ff:ff:ff:ff
     inet 192.168.0.12/24 brd 192.168.0.255 scope global eth0
[root@domU ~]# ip route show
192.168.0.0/24 dev eth0  proto kernel  scope link  src 192.168.0.12
169.254.0.0/16 dev eth0  scope link
default via 192.168.0.254 dev eth0
[root@domU ~]#

====================
[root@dom0 ~]# ip addr show
1: lo: <LOOPBACK,UP> mtu 16436 qdisc noqueue
     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
     inet 127.0.0.1/8 scope host lo
     inet6 ::1/128 scope host
        valid_lft forever preferred_lft forever
2: vif0.0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue
     link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
     inet6 fe80::fcff:ffff:feff:ffff/64 scope link
        valid_lft forever preferred_lft forever
3: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue
     link/ether 00:a0:c9:e3:75:5e brd ff:ff:ff:ff:ff:ff
     inet 192.168.0.11/24 brd 192.168.0.255 scope global eth0
     inet6 fe80::2a0:c9ff:fee3:755e/64 scope link
        valid_lft forever preferred_lft forever
4: peth0: <BROADCAST,MULTICAST,NOARP,UP> mtu 1500 qdisc pfifo_fast  
qlen 1000
     link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
     inet6 fe80::fcff:ffff:feff:ffff/64 scope link
        valid_lft forever preferred_lft forever
5: sit0: <NOARP> mtu 1480 qdisc noop
     link/sit 0.0.0.0 brd 0.0.0.0
6: xen-br0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue
     link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
     inet6 fe80::200:ff:fe00:0/64 scope link
        valid_lft forever preferred_lft forever
7: vif1.1: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue
     link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
     inet6 fe80::fcff:ffff:feff:ffff/64 scope link
        valid_lft forever preferred_lft forever
[root@dom0 ~]# ip route show
192.168.0.0/24 dev eth0  proto kernel  scope link  src 192.168.0.11
169.254.0.0/16 dev eth0  scope link
default via 192.168.0.254 dev eth0
[root@dom0 ~]# brctl show
bridge name     bridge id               STP enabled     interfaces
xen-br0         8000.feffffffffff       no              peth0
                                                         vif0.0
                                                         vif1.1
[root@dom0 ~]#


[-- Attachment #1.2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 2365 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: domU networking trouble in Xen 3.0-unstable.hg

[Bryan D. Payne]

[-- Attachment #1.1: Type: text/plain, Size: 1109 bytes --]

Ok, it's working!  I'm now up and running with changeset 7572.  My  
domU config is the default provided with this changeset.  My dom0  
config is a *very* slightly modified version of the kernel-2.6.12- 
i686-xen0.config provided in the fedora core 4 updates.  In case  
other people are trying to get something similar working, here's my  
grub.conf:

default=1
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title Fedora Core (2.6.13-1.1526_FC4)
         root (hd0,0)
         kernel /vmlinuz-2.6.13-1.1526_FC4 ro root=LABEL=/ rhgb quiet
         initrd /initrd-2.6.13-1.1526_FC4.img

title Xen 3.0-unstable.hg (2.6.12-xen0)
         root (hd0,0)
         kernel /xen.gz
         module /vmlinuz-2.6.12-xen0 ro root=/dev/sda3
         module /initrd-2.6.12.6-xen0.img

And you can find my kernel config for dom0 at the link below:

http://www.cc.gatech.edu/~bdpayne/notes/kernel-2.6.12.6-xen0.config

The only difference between this and the one from FC4 updates is that  
I have disabled SMP and enabled APIC.  On my system, that was  
required for a successful boot into dom0.

Cheers,
bryan

[-- Attachment #1.2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 2365 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel