From: Dmitriy Novikov <dimetrios@gmail.com>
To: xen-devel@lists.xensource.com
Cc: xen-devel@sce.pridelands.org
Subject: Conntrack checksumming problem with pv_ops dom0
Date: Wed, 28 Apr 2010 16:08:24 +0300 [thread overview]
Message-ID: <r2p5f2b45831004280608h9850b718n7449f908c9b696a3@mail.gmail.com> (raw)
Hello.
I have 2.6.32.11 pv_ops dom0 kernel, xen 3.4.3.
dom0 has 2 x Broadcom NICs:
02:05.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5704
Gigabit Ethernet (rev 10)
02:05.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5704
Gigabit Ethernet (rev 10)
On second interface 4,6 tagged vlan configured and connected to
corresponding bridges:
br4 8000.0030485a308f no eth1.4
vif1.1
br6 8000.0030485a308f no eth1.6
vif1.0
domU configured with 2 interfeces (WAN,LAN) which connected to
corresponding bridges:
vif = ['mac=00:16:3e:00:06:11,bridge=br6','mac=00:16:3e:00:04:11,bridge=br4']
domU used for OpenVPN. OpenVPN runs over tun1 interface. OpenVPN
traffic comes from eth0
and forwarded to local network via eth1 interface. All checksumming
disabled on xen-netfront interfaces:
# ethtool -k eth0
Offload parameters for eth0:
rx-checksumming: off
tx-checksumming: off
scatter-gather: off
tcp-segmentation-offload: off
udp-fragmentation-offload: off
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off
# ethtool -k eth1
Offload parameters for eth1:
rx-checksumming: off
tx-checksumming: off
scatter-gather: off
tcp-segmentation-offload: off
udp-fragmentation-offload: off
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off
All checksumming disabled on real and vif/dom0 interfaces:
# ethtool -k eth1
Offload parameters for eth1:
rx-checksumming: off
tx-checksumming: off
scatter-gather: off
tcp-segmentation-offload: off
udp-fragmentation-offload: off
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off
# ethtool -k vif1.1
Offload parameters for vif1.1:
rx-checksumming: off
tx-checksumming: off
scatter-gather: off
tcp-segmentation-offload: off
udp-fragmentation-offload: off
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off
# ethtool -k vif1.0
Offload parameters for vif1.0:
rx-checksumming: off
tx-checksumming: off
scatter-gather: off
tcp-segmentation-offload: off
udp-fragmentation-offload: off
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off
Broadcom offloading firmware removed from /lib/firmware:
mv /lib/firmware/2.6.32.11-xen-dom0/tigon ~/fw_backup
Apr 27 18:26:09 xenon kernel: [ 0.562660] tg3.c:v3.102 (September 1, 2009)
Apr 27 18:26:09 xenon kernel: [ 0.562800] xen: registering gsi 26
triggering 0 polarity 1
Apr 27 18:26:09 xenon kernel: [ 0.562817] alloc irq_desc for 26 on node 0
Apr 27 18:26:09 xenon kernel: [ 0.562821] alloc kstat_irqs on node 0
Apr 27 18:26:09 xenon kernel: [ 0.562831] xen: --> irq=26
Apr 27 18:26:09 xenon kernel: [ 0.562841] tg3 0000:02:05.0: PCI INT
A -> GSI 26 (level, low) -> IRQ 26
...
Apr 27 18:26:09 xenon kernel: [ 0.607550] eth0: Tigon3
[partno(BCM95704A6) rev 2100] (PCIX:100MHz:64-bit) MAC address
00:30:48:5a:30:8e
Apr 27 18:26:09 xenon kernel: [ 0.607708] eth0: attached PHY is
5704 (10/100/1000Base-T Ethernet) (WireSpeed[1])
Apr 27 18:26:09 xenon kernel: [ 0.607856] eth0: RXcsums[1]
LinkChgREG[0] MIirq[0] ASF[1] TSOcap[0] <---------------RXcsums[1]
???
Apr 27 18:26:09 xenon kernel: [ 0.607965] eth0:
dma_rwctrl[769f4000] dma_mask[64-bit]
Apr 27 18:26:09 xenon kernel: [ 0.608111] xen: registering gsi 19
triggering 0 polarity 1
Apr 27 18:26:09 xenon kernel: [ 0.608128] alloc irq_desc for 19 on node 0
Apr 27 18:26:09 xenon kernel: [ 0.608132] alloc kstat_irqs on node 0
Apr 27 18:26:09 xenon kernel: [ 0.608142] xen: --> irq=19
...
Apr 27 18:26:09 xenon kernel: [ 0.615909] tg3 0000:02:05.1: PCI INT
B -> GSI 27 (level, low) -> IRQ 27
Apr 27 18:26:09 xenon kernel: [ 0.636546] eth1: Tigon3
[partno(BCM95704A6) rev 2100] (PCIX:100MHz:64-bit) MAC address
00:30:48:5a:30:8f
Apr 27 18:26:09 xenon kernel: [ 0.636704] eth1: attached PHY is
5704 (10/100/1000Base-T Ethernet) (WireSpeed[1])
Apr 27 18:26:09 xenon kernel: [ 0.636851] eth1: RXcsums[1]
LinkChgREG[0] MIirq[0] ASF[0] TSOcap[1]
Apr 27 18:26:09 xenon kernel: [ 0.636961] eth1:
dma_rwctrl[769f4000] dma_mask[64-bit]
...
Apr 27 18:26:09 xenon kernel: [ 7.340398] tg3 0000:02:05.1:
firmware: requesting tigon/tg3_tso.bin
Apr 27 18:26:09 xenon kernel: [ 7.369999] eth1: Failed to load
firmware "tigon/tg3_tso.bin"
Apr 27 18:26:09 xenon kernel: [ 7.370123] eth1: TSO capability disabled.
Before modprobing nf_conntrack_ipv4 forwarding works fine:
# tcpdump -ni tun1 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tun1, link-type RAW (Raw IP), capture size 65535 bytes
15:13:19.525889 IP 192.168.96.4 > 10.10.16.211: ICMP echo request, id
2401, seq 17, length 64
15:13:19.540332 IP 10.10.16.211 > 192.168.96.4: ICMP echo reply, id
2401, seq 17, length 64
15:13:20.526379 IP 192.168.96.4 > 10.10.16.211: ICMP echo request, id
2401, seq 18, length 64
15:13:20.540641 IP 10.10.16.211 > 192.168.96.4: ICMP echo reply, id
2401, seq 18, length 64
# tcpdump -ni eth1 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes
15:13:41.550283 IP 192.168.96.4 > 10.10.16.211: ICMP echo request, id
2401, seq 39, length 64
15:13:41.563788 IP 10.10.16.211 > 192.168.96.4: ICMP echo reply, id
2401, seq 39, length 64
15:13:42.551962 IP 192.168.96.4 > 10.10.16.211: ICMP echo request, id
2401, seq 40, length 64
15:13:42.565110 IP 10.10.16.211 > 192.168.96.4: ICMP echo reply, id
2401, seq 40, length 64
# tcpdump -ni vif1.1 icmp
tcpdump: WARNING: vif1.1: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vif1.1, link-type EN10MB (Ethernet), capture size 96 bytes
15:14:23.823167 IP 192.168.96.4 > 10.10.16.211: ICMP echo request, id
2401, seq 81, length 64
15:14:23.836664 IP 10.10.16.211 > 192.168.96.4: ICMP echo reply, id
2401, seq 81, length 64
15:14:24.823795 IP 192.168.96.4 > 10.10.16.211: ICMP echo request, id
2401, seq 82, length 64
15:14:24.836965 IP 10.10.16.211 > 192.168.96.4: ICMP echo reply, id
2401, seq 82, length 64
But after modprobing nf_conntrack_ipv4 traffic destinated to local
network disappears on vif1.1 while traffic is visible on domU's lan
interface.
A lot of "Attempting to checksum a non-TCP/UDP packet, dropping a
protocol 1 packet" messages seen on dom0.
# tcpdump -ni eth1 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes
15:15:25.603809 IP 192.168.96.4 > 10.10.16.211: ICMP echo request, id
2401, seq 143, length 64
15:15:25.618697 IP 10.10.16.211 > 192.168.96.4: ICMP echo reply, id
2401, seq 143, length 64
15:15:26.610258 IP 192.168.96.4 > 10.10.16.211: ICMP echo request, id
2401, seq 144, length 64
15:15:26.626101 IP 10.10.16.211 > 192.168.96.4: ICMP echo reply, id
2401, seq 144, length 64
# tcpdump -ni tun1 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tun1, link-type RAW (Raw IP), capture size 65535 bytes
15:15:41.680256 IP 192.168.96.4 > 10.10.16.211: ICMP echo request, id
2401, seq 159, length 64
15:15:41.694291 IP 10.10.16.211 > 192.168.96.4: ICMP echo reply, id
2401, seq 159, length 64
15:15:42.681677 IP 192.168.96.4 > 10.10.16.211: ICMP echo request, id
2401, seq 160, length 64
15:15:42.697395 IP 10.10.16.211 > 192.168.96.4: ICMP echo reply, id
2401, seq 160, length 64
# tcpdump -ni vif1.1 icmp
tcpdump: WARNING: vif1.1: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vif1.1, link-type EN10MB (Ethernet), capture size 96 bytes
15:15:58.036979 IP 192.168.96.4 > 10.10.16.211: ICMP echo request, id
2401, seq 175, length 64
15:15:59.043942 IP 192.168.96.4 > 10.10.16.211: ICMP echo request, id
2401, seq 176, length 64
15:16:00.052912 IP 192.168.96.4 > 10.10.16.211: ICMP echo request, id
2401, seq 177, length 64
15:16:01.053850 IP 192.168.96.4 > 10.10.16.211: ICMP echo request, id
2401, seq 178, length 64
BUT when I set 'sysctl -w net.netfilter.nf_conntrack_checksum=0'
traffic starts to run again.
Any ideas?
next reply other threads:[~2010-04-28 13:08 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2010-04-28 13:08 Dmitriy Novikov [this message]
2010-05-11 7:42 ` Conntrack checksumming problem with pv_ops dom0 Dmitry Novikov
2010-05-22 11:35 ` Pasi Kärkkäinen
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=r2p5f2b45831004280608h9850b718n7449f908c9b696a3@mail.gmail.com \
--to=dimetrios@gmail.com \
--cc=xen-devel@lists.xensource.com \
--cc=xen-devel@sce.pridelands.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).