ARP problems in -testing?

ARP problems in -testing?

[Ron Watkins]

I'm seeing something rather unusual that I thought might be an ARP problem, 
but further testing appears to have ruled that out.

Scenario:  slave domains with random ARP addresses.  On a fresh start of a 
domain, it is unreachable from outside.  Pinging from Domain 0 to the slave 
domain work fine, but changes nothing.

Here's the interesting part: pinging from the slave domain to any external 
host, including Domain 0, also works... and immediately 'unsticks' the net 
connection, so that everything works as I expect it to.

This seems to happen with or without my firewalling rules.  (I disable the 
antispoof section of the 'network' script, so that the firewall rules there 
don't interfere with mine.)

My initial theory was that it's an ARP problem.  I thought the outbound 
packet was being bridged properly to the outside world, the router saw the 
arp address, and started working.   But this does not appear to be correct. 
If I add a secondary IP to the eth0 inside the virtual domain, I do indeed 
see arp requests and arp replies.

When it is in 'stuck' mode, running a tcpdump from the SLAVE domain shows 
the echo requests arriving: [ips changed to protect the morally 
questionable]:

16:36:54.694003 IP 24.0.0.10 > 69.0.0.76: icmp 40: echo request seq 2344

But there are no replies issued.    After I ping the outside world, which 
instantly 'wakes up' the connection:

16:38:57.212284 IP 24.0.0.10 > 69.0.0.76: icmp 40: echo request seq 11816
16:38:57.212314 IP 69.0.0.76 > 24.0.0.10: icmp 40: echo reply seq 11816

This is from a brand-new download today, btw.

I am really mystified.  Any suggestions? 



-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt

RE: ARP problems in -testing?

[Ian Pratt]

> Here's the interesting part: pinging from the slave domain to 
> any external 
> host, including Domain 0, also works... and immediately 
> 'unsticks' the net 
> connection, so that everything works as I expect it to.

This may be an issue with the bridge, though its hard to see why its
causing you problems and not anyone else. 

If you ping the guest domain from dom0, do you see arp requests arriving
inside the guest domain?  The bridge won't learn the guest's MAC address
until the guest sends its first packet. This shouldn't be a problem as
the incoming ARP should be a broadcast packet, and hence get forwarded
anyway.

What happens if you use a routed setup as per Keir's recent checkin?

Cheers,
Ian


-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt

Re: ARP problems in -testing?

[Felipe Alfaro Solana]

On 16 Jan 2005, at 17:42, Ron Watkins wrote:

> I'm seeing something rather unusual that I thought might be an ARP 
> problem, but further testing appears to have ruled that out.
>
> Scenario:  slave domains with random ARP addresses.  On a fresh start 
> of a domain, it is unreachable from outside.  Pinging from Domain 0 to 
> the slave domain work fine, but changes nothing.
>
> Here's the interesting part: pinging from the slave domain to any 
> external host, including Domain 0, also works... and immediately 
> 'unsticks' the net connection, so that everything works as I expect it 
> to.

I'm experiencing this problem too... The workaround I'm using is to 
include a "ping -c2 some_outside_IP" inside the init scripts of the 
slave xenU domain, and it works, but it's an ugly hack.

> This seems to happen with or without my firewalling rules.  (I disable 
> the antispoof section of the 'network' script, so that the firewall 
> rules there don't interfere with mine.)
>
> My initial theory was that it's an ARP problem.  I thought the 
> outbound packet was being bridged properly to the outside world, the 
> router saw the arp address, and started working.   But this does not 
> appear to be correct. If I add a secondary IP to the eth0 inside the 
> virtual domain, I do indeed see arp requests and arp replies.

I don't know what the problem is, but running tcpdump inside the slave 
U domain, shows that packets are received by then xenU kernel, but no 
packets are sent back from the xenU kernel in response. Weird. However, 
pinging some external host from the xenU domain, seems to resolve this 
misterious problem.

> When it is in 'stuck' mode, running a tcpdump from the SLAVE domain 
> shows the echo requests arriving: [ips changed to protect the morally 
> questionable]:
>
> 16:36:54.694003 IP 24.0.0.10 > 69.0.0.76: icmp 40: echo request seq 
> 2344
>
> But there are no replies issued.    After I ping the outside world, 
> which instantly 'wakes up' the connection:
>
> 16:38:57.212284 IP 24.0.0.10 > 69.0.0.76: icmp 40: echo request seq 
> 11816
> 16:38:57.212314 IP 69.0.0.76 > 24.0.0.10: icmp 40: echo reply seq 11816
>
> This is from a brand-new download today, btw.

Yep! I run bk pull every day or so from xeno-unstable.

> I am really mystified.  Any suggestions?

It seems we are having the same problems :-)



-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt

Re: ARP problems in -testing?

[Ron Watkins]

Well, I could certainly be doing something wrong.  That's why I'm asking if 
it's a bug, not saying definitely that it is. :-)

If I ping the guest domain from dom0, while running tcpdump in the slave 
domain, I do see arp requests and replies working.  I can be pinging from 
dom0 to the slave and getting answers back at the EXACT SAME TIME that I am 
pinging from outside... the dom0 pings work, the outside pings arrive and go 
unanswered.  As soon as I ping something from within the slave domain, the 
connection wakes up and works normally.

I have attached a redacted tcpdump text file... I have replaced the IP 
addresses, but it is otherwise exact.  You will see that I am pinging from 
69.0.0.74 to 69.0.0.77 (the master to the slave), which is working, while 
simultaneous requests from 24.0.0.10 go unanswered.   There are correct arp 
request/reply packets, it's working, at least, over the virtual link.

The way this colocated server is configured:

network (wire) address 69.0.0.72/29
Router:  69.0.0.73
Domain0: 69.0.0.74
slave1: .75
slave2: .76
slave 3: .77  (this is the one I was using for testing)
slave 4: .78 (not yet active)
broadcast: .79

I just started on the list today and don't have any idea what you're talking 
about wrt a routed setup...  but if it's just now been checked in, it's 
probably not in my code anyway.  I'm in -testing, not -unstable.

(I won't bother redacting the server ips anymore if I post anything else, 
anyone with half an ounce of brains is going to figure it out anyway.  I 
will continue to change my client IP, however.)



-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt

Re: ARP problems in -testing?

[Ron Watkins]

Yep, that is precisely what I am seeing, and I was considering doing the 
exact same workaround.  Glad it's not just me!

Next question is, why isn't this happening to everyone?   What kind of 
hardware are you running?   Mine is colocated and I don't own it, so I can't 
just look at the manuals.  I'm running the e100 driver, and I'll paste the 
output from lspci at the end of the message.

<<RON>>

lspci output:

0000:00:00.0 Host bridge: Intel Corp. 82865G/PE/P DRAM Controller/Host-Hub 
Interface (rev 02)
0000:00:02.0 VGA compatible controller: Intel Corp. 82865G Integrated 
Graphics Device (rev 02)
0000:00:1d.0 USB Controller: Intel Corp. 82801EB/ER (ICH5/ICH5R) USB UHCI #1 
(rev 02)
0000:00:1d.1 USB Controller: Intel Corp. 82801EB/ER (ICH5/ICH5R) USB UHCI #2 
(rev 02)
0000:00:1d.2 USB Controller: Intel Corp. 82801EB/ER (ICH5/ICH5R) USB UHCI #3 
(rev 02)
0000:00:1d.3 USB Controller: Intel Corp. 82801EB/ER (ICH5/ICH5R) USB UHCI #4 
(rev 02)
0000:00:1d.7 USB Controller: Intel Corp. 82801EB/ER (ICH5/ICH5R) USB2 EHCI 
Controller (rev 02)
0000:00:1f.0 ISA bridge: Intel Corp. 82801EB/ER (ICH5/ICH5R) LPC Bridge (rev 
02)
0000:00:1f.2 IDE interface: Intel Corp. 82801EB (ICH5) Serial ATA 150 
Storage Controller (rev 02)
0000:00:1f.3 SMBus: Intel Corp. 82801EB/ER (ICH5/ICH5R) SMBus Controller 
(rev 02)
0000:00:1f.5 Multimedia audio controller: Intel Corp. 82801EB/ER 
(ICH5/ICH5R) AC'97 Audio Controller (rev 02)
0000:01:08.0 Ethernet controller: Intel Corp. 82562EZ 10/100 Ethernet 
Controller (rev 02)



-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt

Re: ARP problems in -testing?

[Ron Watkins]

[-- Attachment #1: Type: text/plain, Size: 50 bytes --]

Idiot me, I forgot the tcpdump file.  Here it is. 

[-- Attachment #2: tcpdump.out --]
[-- Type: application/octet-stream, Size: 5101 bytes --]

18:08:29.523212 802.1d config 823d.00:0d:bd:c3:d3:c0.8006 root 823d.00:0d:bd:c3:d3:c0 pathcost 0 age 0 max 20 hello 2 fdelay 15 
18:08:31.523272 802.1d config 823d.00:0d:bd:c3:d3:c0.8006 root 823d.00:0d:bd:c3:d3:c0 pathcost 0 age 0 max 20 hello 2 fdelay 15 
18:08:31.588858 IP 24.0.0.10 > 69.0.0.77: icmp 40: echo request seq 35626
18:08:33.523321 802.1d config 823d.00:0d:bd:c3:d3:c0.8006 root 823d.00:0d:bd:c3:d3:c0 pathcost 0 age 0 max 20 hello 2 fdelay 15 
18:08:35.384127 IP 69.0.0.74 > 69.0.0.77: icmp 64: echo request seq 0
18:08:35.384184 IP 69.0.0.77 > 69.0.0.74: icmp 64: echo reply seq 0
18:08:35.523483 802.1d config 823d.00:0d:bd:c3:d3:c0.8006 root 823d.00:0d:bd:c3:d3:c0 pathcost 0 age 0 max 20 hello 2 fdelay 15 
18:08:36.382307 IP 69.0.0.74 > 69.0.0.77: icmp 64: echo request seq 256
18:08:36.382344 IP 69.0.0.77 > 69.0.0.74: icmp 64: echo reply seq 256
18:08:37.088582 IP 24.0.0.10 > 69.0.0.77: icmp 40: echo request seq 35882
18:08:37.382309 IP 69.0.0.74 > 69.0.0.77: icmp 64: echo request seq 512
18:08:37.382346 IP 69.0.0.77 > 69.0.0.74: icmp 64: echo reply seq 512
18:08:37.523466 802.1d config 823d.00:0d:bd:c3:d3:c0.8006 root 823d.00:0d:bd:c3:d3:c0 pathcost 0 age 0 max 20 hello 2 fdelay 15 
18:08:38.382370 IP 69.0.0.74 > 69.0.0.77: icmp 64: echo request seq 768
18:08:38.382411 IP 69.0.0.77 > 69.0.0.74: icmp 64: echo reply seq 768
18:08:39.382340 IP 69.0.0.74 > 69.0.0.77: icmp 64: echo request seq 1024
18:08:39.382377 IP 69.0.0.77 > 69.0.0.74: icmp 64: echo reply seq 1024
18:08:39.523193 802.1d config 823d.00:0d:bd:c3:d3:c0.8006 root 823d.00:0d:bd:c3:d3:c0 pathcost 0 age 0 max 20 hello 2 fdelay 15 
18:08:40.382003 arp who-has 69.0.0.74 tell 69.0.0.77
18:08:40.382149 arp who-has 69.0.0.77 tell 69.0.0.74
18:08:40.382192 arp reply 69.0.0.77 is-at aa:00:00:3d:d5:1b
18:08:40.382389 arp reply 69.0.0.74 is-at 00:0d:61:31:29:8e
18:08:40.382727 IP 69.0.0.74 > 69.0.0.77: icmp 64: echo request seq 1280
18:08:40.382756 IP 69.0.0.77 > 69.0.0.74: icmp 64: echo reply seq 1280
18:08:41.382273 IP 69.0.0.74 > 69.0.0.77: icmp 64: echo request seq 1536
18:08:41.382306 IP 69.0.0.77 > 69.0.0.74: icmp 64: echo reply seq 1536
18:08:41.524493 802.1d config 823d.00:0d:bd:c3:d3:c0.8006 root 823d.00:0d:bd:c3:d3:c0 pathcost 0 age 0 max 20 hello 2 fdelay 15 
18:08:42.382354 IP 69.0.0.74 > 69.0.0.77: icmp 64: echo request seq 1792
18:08:42.382392 IP 69.0.0.77 > 69.0.0.74: icmp 64: echo reply seq 1792
18:08:42.589639 IP 24.0.0.10 > 69.0.0.77: icmp 40: echo request seq 36138
18:08:43.382282 IP 69.0.0.74 > 69.0.0.77: icmp 64: echo request seq 2048
18:08:43.382318 IP 69.0.0.77 > 69.0.0.74: icmp 64: echo reply seq 2048
18:08:43.523135 802.1d config 823d.00:0d:bd:c3:d3:c0.8006 root 823d.00:0d:bd:c3:d3:c0 pathcost 0 age 0 max 20 hello 2 fdelay 15 
18:08:44.382287 IP 69.0.0.74 > 69.0.0.77: icmp 64: echo request seq 2304
18:08:44.382325 IP 69.0.0.77 > 69.0.0.74: icmp 64: echo reply seq 2304
18:08:45.382311 IP 69.0.0.74 > 69.0.0.77: icmp 64: echo request seq 2560
18:08:45.382346 IP 69.0.0.77 > 69.0.0.74: icmp 64: echo reply seq 2560
18:08:45.523173 802.1d config 823d.00:0d:bd:c3:d3:c0.8006 root 823d.00:0d:bd:c3:d3:c0 pathcost 0 age 0 max 20 hello 2 fdelay 15 
18:08:46.382569 IP 69.0.0.74 > 69.0.0.77: icmp 64: echo request seq 2816
18:08:46.382605 IP 69.0.0.77 > 69.0.0.74: icmp 64: echo reply seq 2816
18:08:47.382292 IP 69.0.0.74 > 69.0.0.77: icmp 64: echo request seq 3072
18:08:47.382323 IP 69.0.0.77 > 69.0.0.74: icmp 64: echo reply seq 3072
18:08:47.523205 802.1d config 823d.00:0d:bd:c3:d3:c0.8006 root 823d.00:0d:bd:c3:d3:c0 pathcost 0 age 0 max 20 hello 2 fdelay 15 
18:08:48.087258 IP 24.0.0.10 > 69.0.0.77: icmp 40: echo request seq 36394
18:08:48.382288 IP 69.0.0.74 > 69.0.0.77: icmp 64: echo request seq 3328
18:08:48.382321 IP 69.0.0.77 > 69.0.0.74: icmp 64: echo reply seq 3328
18:08:49.382261 IP 69.0.0.74 > 69.0.0.77: icmp 64: echo request seq 3584
18:08:49.382298 IP 69.0.0.77 > 69.0.0.74: icmp 64: echo reply seq 3584
18:08:49.523177 802.1d config 823d.00:0d:bd:c3:d3:c0.8006 root 823d.00:0d:bd:c3:d3:c0 pathcost 0 age 0 max 20 hello 2 fdelay 15 
18:08:50.382302 IP 69.0.0.74 > 69.0.0.77: icmp 64: echo request seq 3840
18:08:50.382336 IP 69.0.0.77 > 69.0.0.74: icmp 64: echo reply seq 3840
18:08:51.382284 IP 69.0.0.74 > 69.0.0.77: icmp 64: echo request seq 4096
18:08:51.382316 IP 69.0.0.77 > 69.0.0.74: icmp 64: echo reply seq 4096
18:08:51.523115 802.1d config 823d.00:0d:bd:c3:d3:c0.8006 root 823d.00:0d:bd:c3:d3:c0 pathcost 0 age 0 max 20 hello 2 fdelay 15 
18:08:53.523100 802.1d config 823d.00:0d:bd:c3:d3:c0.8006 root 823d.00:0d:bd:c3:d3:c0 pathcost 0 age 0 max 20 hello 2 fdelay 15 
18:08:53.587209 IP 24.0.0.10 > 69.0.0.77: icmp 40: echo request seq 36650
18:08:55.523138 802.1d config 823d.00:0d:bd:c3:d3:c0.8006 root 823d.00:0d:bd:c3:d3:c0 pathcost 0 age 0 max 20 hello 2 fdelay 15 
18:08:57.523262 802.1d config 823d.00:0d:bd:c3:d3:c0.8006 root 823d.00:0d:bd:c3:d3:c0 pathcost 0 age 0 max 20 hello 2 fdelay 15 
18:08:58.535887 00:0d:bd:c3:d3:c6 > 01:00:0c:cc:cc:cc sap aa ui/C len=35

Re: ARP problems in -testing?

[Jan Kundrát]

Ron Watkins wrote:
> Next question is, why isn't this happening to everyone?

What MAC address have you assigned to the domU's interface? Isn't it 
something reserved for broadcasts?

-jkt

-- 
cd /local/pub && more beer > /dev/mouth



-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt

Re: ARP problems in -testing?

[Keir Fraser]

> Ron Watkins wrote:
> > Next question is, why isn't this happening to everyone?
> 
> What MAC address have you assigned to the domU's interface? Isn't it 
> something reserved for broadcasts?
> 
> -jkt

The MAC addresses from the tcpdump log look sane. The problem must lie
elsewhere. 

 -- Keir


-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt

Re: ARP problems in -testing?

[Felipe Alfaro Solana]

On 16 Jan 2005, at 19:49, Ron Watkins wrote:

> Next question is, why isn't this happening to everyone?   What kind of 
> hardware are you running?   Mine is colocated and I don't own it, so I 
> can't just look at the manuals.  I'm running the e100 driver, and I'll 
> paste the output from lspci at the end of the message.

I'm running Xen unstable on a Pentium IV with a 3Com 3c59x (eth0) and a 
Realtek RTL8139 (eth1). I have configured Xen to attach xen-br0 to 
interface eth1.

# /sbin/ifconfig -a
eth0      Link encap:Ethernet  HWaddr 00:04:75:75:97:65
           inet addr:192.168.0.2  Bcast:192.168.0.127  
Mask:255.255.255.128
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:524791 errors:0 dropped:0 overruns:0 frame:0
           TX packets:311542 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:1000
           RX bytes:705155122 (672.4 MiB)  TX bytes:22699822 (21.6 MiB)
           Interrupt:20 Base address:0xa400

eth1      Link encap:Ethernet  HWaddr 00:02:44:7B:B9:53
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:1548286 errors:0 dropped:0 overruns:0 frame:0
           TX packets:768111 errors:0 dropped:0 overruns:0 carrier:0
           collisions:14416 txqueuelen:1000
           RX bytes:1242525048 (1.1 GiB)  TX bytes:408321930 (389.4 MiB)
           Interrupt:18

lo        Link encap:Local Loopback
           inet addr:127.0.0.1  Mask:255.0.0.0
           UP LOOPBACK RUNNING  MTU:16436  Metric:1
           RX packets:1028 errors:0 dropped:0 overruns:0 frame:0
           TX packets:1028 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:105464 (102.9 KiB)  TX bytes:105464 (102.9 KiB)

vif1.0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:768111 errors:0 dropped:0 overruns:0 frame:0
           TX packets:726411 errors:0 dropped:116 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:406930344 (388.0 MiB)  TX bytes:515444106 (491.5 MiB)

xen-br0   Link encap:Ethernet  HWaddr 00:02:44:7B:B9:53
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:106 errors:0 dropped:0 overruns:0 frame:0
           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:4858 (4.7 KiB)  TX bytes:0 (0.0 b)

# lspci
00:00.0 Host bridge: Intel Corp. 82845 845 (Brookdale) Chipset Host 
Bridge (rev 11)
00:1f.0 ISA bridge: Intel Corp. 82801BA ISA Bridge (LPC) (rev 05)
00:1f.1 IDE interface: Intel Corp. 82801BA IDE U100 (rev 05)
00:1f.2 USB Controller: Intel Corp. 82801BA/BAM USB (Hub #1) (rev 05)
00:1f.4 USB Controller: Intel Corp. 82801BA/BAM USB (Hub #2) (rev 05)
00:1f.5 Multimedia audio controller: Intel Corp. 82801BA/BAM AC'97 
Audio (rev 05)
01:01.0 VGA compatible controller: ATI Technologies Inc Radeon RV100 QY 
[Radeon 7000/VE]
02:06.0 Ethernet controller: Realtek Semiconductor Co., Ltd. 
RTL-8139/8139C/8139C+ (rev 10)
02:0c.0 Ethernet controller: 3Com Corporation 3c905C-TX/TX-M [Tornado] 
(rev 30)



-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt

Re: ARP problems in -testing?

[Felipe Alfaro Solana]

On 16 Jan 2005, at 20:18, Jan Kundrát wrote:

> Ron Watkins wrote:
>> Next question is, why isn't this happening to everyone?
>
> What MAC address have you assigned to the domU's interface? Isn't it 
> something reserved for broadcasts?

Don't know how to change that...
However, I see the xenU kernel is invoked with a somewhat funny command 
line:

# cat /proc/cmdline
  ip=192.168.0.130:1.2.3.4::::eth0:off root=/dev/sda1 ro

Here's the xenU config file:

kernel = "/boot/vmlinuz-2.6.10-xenU"
memory = 100
name = "Debian"
nics = 1
ip = "192.168.0.130"
disk = [ 'file:/path/to/file,sda1,w' ]
root = "/dev/sda1 ro

Where's 1.2.3.4 coming from?


-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt

Re: ARP problems in -testing?

[Derrik Pates]

Keir Fraser wrote:
> The MAC addresses from the tcpdump log look sane. The problem must lie
> elsewhere. 

I just tried this on a dedicated server located at thePlanet in Texas; I 
think this is an issue that combines the following:

   - No MAC address specified causes a random MAC address to be selected 
at domain create time
   - The upstream router has a local ARP cache
   - The host system's ARP cache for addresses on the virtual NIC for 
the instance is blown out when the virtual interface disappears when the 
domain is destroyed

This seems to explain why tcpdump shows packets going to the VM's IP, 
but TCP sessions and ICMP packets don't get a response - the switch 
still knows the "old" MAC address, but the first outbound packet to the 
outside world implicitly eradicates the old ARP-cache entry. I don't 
think this is Xen's fault at all, just a property of the implementation 
of ARP resolution. I just happened to notice that the MAC shown in 
incoming packets didn't match the current MAC address (based on 
ifconfig's output), so this is my current theory on what's happening.

-- 
Derrik Pates
demon@devrandom.net


-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt

Re: ARP problems in -testing?

[Felipe Alfaro Solana]

On 17 Jan 2005, at 01:27, Derrik Pates wrote:

> Keir Fraser wrote:
>> The MAC addresses from the tcpdump log look sane. The problem must lie
>> elsewhere.
>
> I just tried this on a dedicated server located at thePlanet in Texas; 
> I think this is an issue that combines the following:
>
>   - No MAC address specified causes a random MAC address to be 
> selected at domain create time

Well, I can confirm this was the culprit of my problems: Xen was 
selecting random MAC addresses at startup. I have fixed the problem by 
manually and statically setting a predefined MAC address in the domain 
config file:

vif = [ 'mac=XX:XX:XX:XX:XX:XX' ]

Now, the MAC address is always the same and thus I don't need the ugly 
ping hack.
Thanks.



-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt

RE: ARP problems in -testing?

[Ian Pratt]

 > >
> > vif = [ 'mac=XX:XX:XX:XX:XX:XX' ]
> >
> > Now, the MAC address is always the same and thus I don't 
> need the ugly ping 
> > hack.
> 
> this is good, but if I start multiple domains, like this
> 
>  	xm create vm4-bproc vmid=1
>  	xm create vm4-bproc vmid=2
>  	xm create vm4-bproc vmid=3
>  	xm create vm4-bproc vmid=4
> 
> then I'll end up with 4 copies of the same mac. Is there some 
> way to tie 
> mac generation to vmid ?

The following will work, though has obvious limitations (256 domains):
vif = [ 'mac=XX:XX:XX:XX:%02x:XX' % vmid ]

The netback vif needs a MAC addr too, and this is picked as the frontend's MAC addr+1. This was probably a bad choice as people tend to pick consecutive MAC addrs for their domains. We should probably move the front/back bit to a mor esignificant bit.

Hence, the above vif= line sets vmid into the 2nd octet.

Ian



Ian


-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt

Re: ARP problems in -testing?

[Adam Sulmicki]

>> I just tried this on a dedicated server located at thePlanet in Texas; I 
>> think this is an issue that combines the following:
>> 
>>   - No MAC address specified causes a random MAC address to be selected at 
>> domain create time
>
> Well, I can confirm this was the culprit of my problems: Xen was selecting 
> random MAC addresses at startup. I have fixed the problem by manually and 
> statically setting a predefined MAC address in the domain config file:
>
> vif = [ 'mac=XX:XX:XX:XX:XX:XX' ]
>
> Now, the MAC address is always the same and thus I don't need the ugly ping 
> hack.

this is good, but if I start multiple domains, like this

 	xm create vm4-bproc vmid=1
 	xm create vm4-bproc vmid=2
 	xm create vm4-bproc vmid=3
 	xm create vm4-bproc vmid=4

then I'll end up with 4 copies of the same mac. Is there some way to tie 
mac generation to vmid ?


-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt

RE: ARP problems in -testing?

[Ian Pratt]

> ah, cool. it makes me wonder though.
> 
> in the IP the 10.x.x.x and 192.168.x.x are reserved for private use?
> is there such MAC address space reserved as well?
> 
> looking a bit around the web I was unable to find such guideliness.

>From xen/xm/create.py: 

def randomMAC():
    """Generate a random MAC address.

    Uses OUI (Organizationally Unique Identifier) AA:00:00, an
    unassigned one that used to belong to DEC. The OUI list is
    available at 'standards.ieee.org'.

    The remaining 3 fields are random, with the first bit of the first
    random field set 0.

    @return: MAC address string
    """
    random.seed()
    mac = [ 0xaa, 0x00, 0x00,
            random.randint(0x00, 0x7f),
            random.randint(0x00, 0xff),
            random.randint(0x00, 0xff) ]
    return ':'.join(map(lambda x: "%02x" % x, mac))


-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt

RE: ARP problems in -testing?

[Adam Sulmicki]

>
> The following will work, though has obvious limitations (256 domains):
> vif = [ 'mac=XX:XX:XX:XX:%02x:XX' % vmid ]
>
> The netback vif needs a MAC addr too, and this is picked as the frontend's MAC addr+1. This was probably a bad choice as people tend to pick consecutive MAC addrs for their domains. We should probably move the front/back bit to a mor esignificant bit.
>
> Hence, the above vif= line sets vmid into the 2nd octet.

ah, cool. it makes me wonder though.

in the IP the 10.x.x.x and 192.168.x.x are reserved for private use?
is there such MAC address space reserved as well?

looking a bit around the web I was unable to find such guideliness.

 	http://standards.ieee.org/regauth/oui/index.shtml
 	http://standards.ieee.org/regauth/oui/oui.txt

 	http://en.wikipedia.org/wiki/MAC_address


-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt

Re: ARP problems in -testing?

[Ron Watkins]

I. Am. An. Idiot.

Thanks, Derrick.

<<RON>>

----- Original Message ----- 
From: "Derrik Pates" <demon@devrandom.net>
To: "Keir Fraser" <Keir.Fraser@cl.cam.ac.uk>
Cc: "Jan Kundrát" <jan.kundrat@fzu.cz>; "Ron Watkins" <xen-devel@malor.com>; 
<xen-devel@lists.sourceforge.net>
Sent: Sunday, January 16, 2005 7:27 PM
Subject: Re: [Xen-devel] ARP problems in -testing?


> Keir Fraser wrote:
>> The MAC addresses from the tcpdump log look sane. The problem must lie
>> elsewhere.
>
> I just tried this on a dedicated server located at thePlanet in Texas; I 
> think this is an issue that combines the following:
>
>   - No MAC address specified causes a random MAC address to be selected at 
> domain create time
>   - The upstream router has a local ARP cache
>   - The host system's ARP cache for addresses on the virtual NIC for the 
> instance is blown out when the virtual interface disappears when the 
> domain is destroyed
>
> This seems to explain why tcpdump shows packets going to the VM's IP, but 
> TCP sessions and ICMP packets don't get a response - the switch still 
> knows the "old" MAC address, but the first outbound packet to the outside 
> world implicitly eradicates the old ARP-cache entry. I don't think this is 
> Xen's fault at all, just a property of the implementation of ARP 
> resolution. I just happened to notice that the MAC shown in incoming 
> packets didn't match the current MAC address (based on ifconfig's output), 
> so this is my current theory on what's happening.
>
> -- 
> Derrik Pates
> demon@devrandom.net
>
>
> -------------------------------------------------------
> The SF.Net email is sponsored by: Beat the post-holiday blues
> Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
> It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel
> 



-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt

Re: ARP problems in -testing?

[Keir Fraser]

> I just tried this on a dedicated server located at thePlanet in Texas; I 
> think this is an issue that combines the following:
>
>   - No MAC address specified causes a random MAC address to be selected at 
> domain create time
>   - The upstream router has a local ARP cache
>   - The host system's ARP cache for addresses on the virtual NIC for the 
> instance is blown out when the virtual interface disappears when the 
> domain is destroyed
>
> This seems to explain why tcpdump shows packets going to the VM's IP, but 
> TCP sessions and ICMP packets don't get a response - the switch still 
> knows the "old" MAC address, but the first outbound packet to the outside 
> world implicitly eradicates the old ARP-cache entry. I don't think this is 
> Xen's fault at all, just a property of the implementation of ARP 
> resolution. I just happened to notice that the MAC shown in incoming 
> packets didn't match the current MAC address (based on ifconfig's output), 
> so this is my current theory on what's happening.

Thanks for the detailed answer -- I've added it to the Xen FAQ
(http://www.cl.cam.ac.uk/netos/xen/faq.html). I guessed it would
probably be something like this... 

 Cheers,
 Keir


-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt

Re: ARP problems in -testing?

[Keir Fraser]

> > Keir Fraser wrote:
> >> The MAC addresses from the tcpdump log look sane. The problem must lie
> >> elsewhere.
> >
> > I just tried this on a dedicated server located at thePlanet in Texas; 
> > I think this is an issue that combines the following:
> >
> >   - No MAC address specified causes a random MAC address to be 
> > selected at domain create time
> 
> Well, I can confirm this was the culprit of my problems: Xen was 
> selecting random MAC addresses at startup. I have fixed the problem by 
> manually and statically setting a predefined MAC address in the domain 
> config file:
> 
> vif = [ 'mac=XX:XX:XX:XX:XX:XX' ]
> 
> Now, the MAC address is always the same and thus I don't need the ugly 
> ping hack.
> Thanks.

I've fixed the netfront driver to send an unsolicited ARP reply
whenever an IP address gets brought up on it. This fix is in the
-testing and -unstable trees, and will end up in the forthcoming 2.0.4
stable release.

Perhaps someone could build the repository, return to using random MAC
addresses, and confirm that the problem has gone away?

 Thanks,
 Keir


-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt

RE: ARP problems in -testing?

[Tim Durack]

There are four types of MAC addresses:

Globally Unique
*0-**-**-**-**-**
*4-**-**-**-**-**
*8-**-**-**-**-**
*C-**-**-**-**-**

Locally Administered
*2-**-**-**-**-**
*6-**-**-**-**-**
*A-**-**-**-**-**
*E-**-**-**-**-**

Multicast
*1-**-**-**-**-**
*3-**-**-**-**-**
*5-**-**-**-**-**
*7-**-**-**-**-**
*9-**-**-**-**-**
*B-**-**-**-**-**
*D-**-**-**-**-**
*F-**-**-**-**-**

Broadcast
FF-FF-FF-FF-FF-FF (Could be considered an all-stations multicast)

Locally administered addresses can be considered private ethernet:

(LAA) are MAC addresses which have the second least significant bit
(LSB) of the first octect is set to '1' (for example, 'xxxxxx1x'. LAA
enable administrators to assign MAC addresses using their own scheme.
When an LAA is assigned, it overrides the in-memory copy of the Global
address during driver initialization. So the burned-in address on the
Ethernet adapter is never actually changed, only the in-memory copy.
LAA create the possibility of assigning duplicate MAC addresses, which
makes for some very interesting network problems. Fortunately, they
are rarely used in Ethernet environments, but are fairly popular in
Token Ring networks.


The Xen addresses are in the LAA range, so scheme wise you are safe to
do whatever you want with them.

Alternatively XenSource could spring for an OUI or IAB, not really
that expensive:

    The OUI is $1,650.00 (US); the IAB is $550.00 (US)

IAB = Individual Address Block, 12 bits of address space instead of 24.

Tim:>


-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt

Re: ARP problems in -testing?

[Felipe Alfaro Solana]

On 17 Jan 2005, at 14:41, Keir Fraser wrote:

>> Well, I can confirm this was the culprit of my problems: Xen was
>> selecting random MAC addresses at startup. I have fixed the problem by
>> manually and statically setting a predefined MAC address in the domain
>> config file:
>>
>> vif = [ 'mac=XX:XX:XX:XX:XX:XX' ]
>>
>> Now, the MAC address is always the same and thus I don't need the ugly
>> ping hack.
>> Thanks.
>
> I've fixed the netfront driver to send an unsolicited ARP reply
> whenever an IP address gets brought up on it. This fix is in the
> -testing and -unstable trees, and will end up in the forthcoming 2.0.4
> stable release.
>
> Perhaps someone could build the repository, return to using random MAC
> addresses, and confirm that the problem has gone away?

I will be away for 8 hours, but will test those changes ASAP...
Thanks!



-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt