Problem with VLANs on tg3 NICs

[Evan Bigall <evan.bigall@scalent.com>]

I have 3 machines,
    T60 thinkpad with e1000 NIC
    Dell PowerEdge 2950 with bnx2 NIC
    HP DL360 G4P with tg3 NIC
All running xen 3.03 / 2.6.16.29-xen

I have a netbootable image with 2.6.16.29-xenU.  This image when it 
boots creates a vlan eth0.4002, and then tries to communicate on that 
interface to a central server.

On the machines with e1000 an bnx2 NICs, this works fine.  On the 
machine with tg3 NICs, this does not work.

It's a little bit hard to tell, but I think whats happening is that the 
packets are going out tagged ok.  But, on ingress in dom0, the tg3 
driver is stripping the vlan tag:

I'm not sure I trust tcpdump 100% but it seems to support that the arp 
request is going out tagged, but the response is coming back untagged:

09:41:43.126497 00:16:3e:5a:4e:d8 > Broadcast, ethertype 802.1Q 
(0x8100), length 46: vlan 4002, p 0, ethertype ARP, arp who-has 
10.21.1.1 tell 10.21.1.99
        0x0000:  0fa2 0806 0001 0800 0604 0001 0016 3e5a  ..............>Z
        0x0010:  4ed8 0a15 0163 0000 0000 0000 0a15 0101  N....c..........
09:41:43.126591 00:0e:aa:04:00:01 > 00:16:3e:5a:4e:d8, ethertype ARP 
(0x0806), length 60: arp reply 10.21.1.1 is-at 00:0e:aa:04:00:01
        0x0000:  0001 0800 0604 0002 000e aa04 0001 0a15  ................
        0x0010:  0101 0016 3e5a 4ed8 0a15 0163 0000 0000  ....>ZN....c....
        0x0020:  0000 0000 0000 0000 0000 0000 0000       ..............

I did a quick experiment using xen 3.0.4_1 and got the same results.

Any ideas?

Evan