domU network data payload corruption

domU network data payload corruption

[Saku Ytti]

 neutral observing machine running 'ping 10.10.10.101 -p aa', where
10.10.10.101 is latest xen/unstable domU (dom0 also xen/unstable, kernel,
hypervisor, tools)

 tethereal ran in domU for packet leaving:
 0000  6d 6d eb 43 88 0c 0f 00 aa aa aa aa aa aa aa aa   mm.C............
 0010  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa   ................
 0020  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa   ................
 0030  aa aa aa aa aa aa aa aa                           ........

 tethereal ran in neutral observing machine for packet arriving:
 0000  6d 6d eb 43 88 0c 0f 00 aa aa aa aa aa aa aa aa   mm.C............
 0010  aa aa aa aa aa aa ff ff ff ff ff ff ff ff ff ff   ................
 0020  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   ................
 0030  ff ff ff ff ff ff ff ff                           ........


Also 'ping' command reporting:
64 bytes from 10.10.10.101: icmp_seq=13 ttl=64 time=0.268 ms
wrong data byte #22 should be 0xaa but was 0xff
#8      aa aa aa aa aa aa aa aa aa aa aa aa aa aa ff ff ff ff ff ff ff ff
ff ff ff ff ff ff ff ff ff ff 
#40     ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff 


now if I run 'ping 10.10.10.101 -p ff', there is no problem, packets
are received without errors. To further prove it 'ping 10.10.10.101 -s 22
-p aa' is also ok, but 'ping 10.10.10.101 -s 23 -p aa' not anymore, as 
all bits after that are set '1'.

Any clue what might be causing this? Reloading domU doesn't help, but
if I reload dom0 domU can be ok for long time (upgraded debian/unstable
etc) but after some time and especially after adding more domU it
seems always trigger at least after adding 3rd domU and then hits
also to the earlier domU's.

I'm happy to provide any information required, I'm using forcedeth
NIC driver in dom0.

Thanks,
-- 
  ++ytti

Re: domU network data payload corruption

[Saku Ytti]

On (2006-02-09 18:34 +0200), Saku Ytti wrote:

 
> Also 'ping' command reporting:
> 64 bytes from 10.10.10.101: icmp_seq=13 ttl=64 time=0.268 ms
> wrong data byte #22 should be 0xaa but was 0xff
> #8      aa aa aa aa aa aa aa aa aa aa aa aa aa aa ff ff ff ff ff ff ff ff
> ff ff ff ff ff ff ff ff ff ff 
> #40     ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff 
> 
> now if I run 'ping 10.10.10.101 -p ff', there is no problem, packets
> are received without errors. To further prove it 'ping 10.10.10.101 -s 22
> -p aa' is also ok, but 'ping 10.10.10.101 -s 23 -p aa' not anymore, as 
> all bits after that are set '1'.

Bit more information.

 1) changing domU kernel to xen/stable (2.6.12) does not help
 2) it's always 3rd domU where it beings triggered immediately.
    It can be any domU, just it has to be one that is started 3rd.
    Network doesn't work for a second in the 3rd domU.
     2nd and 1st domU continue to work for quite long time, but
    issue can be triggered there too, not sure what's the trigger.

-- 
  ++ytti

Re: domU network data payload corruption

[Keir Fraser]

On 9 Feb 2006, at 16:34, Saku Ytti wrote:

> Any clue what might be causing this? Reloading domU doesn't help, but
> if I reload dom0 domU can be ok for long time (upgraded debian/unstable
> etc) but after some time and especially after adding more domU it
> seems always trigger at least after adding 3rd domU and then hits
> also to the earlier domU's.
>
> I'm happy to provide any information required, I'm using forcedeth
> NIC driver in dom0.

How much RAM does your system have? Are you running 32-bit, PAE, or 
64-bit? What does the Ethernet driver print out during boot (may tell 
us what sub-version forcedeth you have)?

  -- Keir

Re: domU network data payload corruption

[Saku Ytti]

On (2006-02-09 17:03 +0000), Keir Fraser wrote:

Hello Keir,
 
> How much RAM does your system have? Are you running 32-bit, PAE, or 
> 64-bit? What does the Ethernet driver print out during boot (may tell 
> us what sub-version forcedeth you have)?

4GB, dual core AMD athalon, 64b (no 32b emulation support compiled in
in domU or dom0). No SMP support compiled in in domU or dom0.

forcedeth.c: Reverse Engineered nForce ethernet driver. Version 0.49.
eth0: forcedeth.c: subsystem: 01043:8141 bound to 0000:00:0a.0

0000:00:0a.0 Bridge: nVidia Corporation CK804 Ethernet Controller (rev a3)
        Subsystem: ASUSTeK Computer Inc. K8N4-E Mainboard IRQ 16
        Flags: bus master, 66MHz, fast devsel, latency 0, IRQ 16
        Memory at d5400000 (32-bit, non-prefetchable) [size=4K]
        I/O ports at b000 [size=8]anagement version 2 Bridge (rev a3)
        Capabilities: [44] Power Management version 2 Bridge (rev a3)

Last entry in forcedeth.c changelog
 *      0.49: 10 Dec 2005: Fix tso for large buffers.

I'll try to see if I have extra PCI NIC's at home to see if it's NIC
specific issue. If not need to loan one from work tomorrow.

Thanks for quick reply,

-- 
  ++ytti

Re: domU network data payload corruption

[Saku Ytti]

On (2006-02-09 17:13 +0000), Keir Fraser wrote:

> >How much RAM does your system have? Are you running 32-bit, PAE, or 
> >64-bit? What does the Ethernet driver print out during boot (may tell 
> >us what sub-version forcedeth you have)?
> 
> Also please post the output of 'lspci -n'.

Sorry, already replied toi the earlier email, included pci information
but just in case you need it from others too, here goes:

[root@herra ~]# lspci -n
0000:00:00.0 0580: 10de:005e (rev a3)
0000:00:01.0 0601: 10de:0050 (rev a3)
0000:00:01.1 0c05: 10de:0052 (rev a2)
0000:00:02.0 0c03: 10de:005a (rev a2)
0000:00:02.1 0c03: 10de:005b (rev a3)
0000:00:04.0 0401: 10de:0059 (rev a2)
0000:00:06.0 0101: 10de:0053 (rev f2)
0000:00:07.0 0101: 10de:0054 (rev f3)
0000:00:08.0 0101: 10de:0055 (rev f3)
0000:00:09.0 0604: 10de:005c (rev a2)
0000:00:0a.0 0680: 10de:0057 (rev a3)
0000:00:0b.0 0604: 10de:005d (rev a3)
0000:00:0c.0 0604: 10de:005d (rev a3)
0000:00:0d.0 0604: 10de:005d (rev a3)
0000:00:0e.0 0604: 10de:005d (rev a3)
0000:00:18.0 0600: 1022:1100
0000:00:18.1 0600: 1022:1101
0000:00:18.2 0600: 1022:1102
0000:00:18.3 0600: 1022:1103
0000:01:00.0 0604: 8086:0330 (rev 07)
0000:01:00.2 0604: 8086:0332 (rev 07)
0000:02:0e.0 0104: 17d3:1210
0000:07:08.0 0300: 102b:0519 (rev 01)
[root@herra ~]# 

-- 
  ++ytti

Re: domU network data payload corruption

[Keir Fraser]

On 9 Feb 2006, at 17:03, Keir Fraser wrote:

>> Any clue what might be causing this? Reloading domU doesn't help, but
>> if I reload dom0 domU can be ok for long time (upgraded 
>> debian/unstable
>> etc) but after some time and especially after adding more domU it
>> seems always trigger at least after adding 3rd domU and then hits
>> also to the earlier domU's.
>>
>> I'm happy to provide any information required, I'm using forcedeth
>> NIC driver in dom0.
>
> How much RAM does your system have? Are you running 32-bit, PAE, or 
> 64-bit? What does the Ethernet driver print out during boot (may tell 
> us what sub-version forcedeth you have)?

Also please post the output of 'lspci -n'.

  -- Keir

Re: domU network data payload corruption

[Saku Ytti]

On (2006-02-09 17:26 +0000), Keir Fraser wrote:
 
> Hopefully you are building your own kernels. If so, try modifying the 

I am, due to need for areca support.

> Linux file drivers/net/forcedeth.c. There is a line like this:
> define DEV_HAS_HIGH_DMA 0x0008
> Try changing the 0x0008 to 0
> 
> I think this will probably work better for you. :-)

Will try. Btw. just found vortex and the issue is not present there,
will try installing 4th domU while three domU's are running and
after that I'll try the change your suggesting and report back.

Thanks,
-- 
  ++ytti

Re: domU network data payload corruption

[Keir Fraser]

On 9 Feb 2006, at 17:07, Saku Ytti wrote:

> I'll try to see if I have extra PCI NIC's at home to see if it's NIC
> specific issue. If not need to loan one from work tomorrow.

Your NIC is supposed to support DMA to high memory (above 4GB) but I 
don't believe it is working.

Hopefully you are building your own kernels. If so, try modifying the 
Linux file drivers/net/forcedeth.c. There is a line like this:
define DEV_HAS_HIGH_DMA 0x0008
Try changing the 0x0008 to 0

I think this will probably work better for you. :-)

  -- Keir

Re: domU network data payload corruption

[Saku Ytti]

On (2006-02-09 17:26 +0000), Keir Fraser wrote:
 
> Hopefully you are building your own kernels. If so, try modifying the 
> Linux file drivers/net/forcedeth.c. There is a line like this:
> define DEV_HAS_HIGH_DMA 0x0008
> Try changing the 0x0008 to 0
> 
> I think this will probably work better for you. :-)

 I can confirm that it fixes the issue, thanks. I'll contact
upstream and refer them to this thread.

Thanks again,
-- 
  ++ytti

Re: domU network data payload corruption

[Keir Fraser]

On 9 Feb 2006, at 17:59, Saku Ytti wrote:

>> Hopefully you are building your own kernels. If so, try modifying the
>> Linux file drivers/net/forcedeth.c. There is a line like this:
>> define DEV_HAS_HIGH_DMA 0x0008
>> Try changing the 0x0008 to 0
>>
>> I think this will probably work better for you. :-)
>
>  I can confirm that it fixes the issue, thanks. I'll contact
> upstream and refer them to this thread.

FYI:

The correct fix is to remove DEV_HAS_HIGH_DMA from the feature list for 
the appropriate device subtype (your device subtype is 
NVIDIA_NVENET_9).

It's surprising that noone else with a 64-bit system has seen this 
issue. Perhaps the NIC only has problems with packet fragments residing 
above 4GB -- that case would almost never be tested in native Linux. Or 
perhaps very few people with 64-bit systems use this type of card. Even 
PAE systems would typically not pass highmem pages to a network driver.

  -- Keir