* tg3 network stall in xen-3.4.x but not in xen-3.3.x
@ 2009-07-04 6:32 Teck Choon Giam
2009-07-04 6:41 ` Keir Fraser
0 siblings, 1 reply; 10+ messages in thread
From: Teck Choon Giam @ 2009-07-04 6:32 UTC (permalink / raw)
To: xen-devel
Hi,
I have experienced network stall when running in xen 3.4.x on all DELL
PE850/860/R200 which are using onboard broadcom driver tg3 driver.
I have done some testing on both xen-3.3.2-rc3 and xen-3.4.1-rc5 with
linux-2.6.18-xen.hg changeset 913 which in domU doing scp transfer of
couple 1MB/10MB/100MB files to another server in few instances
concurrently. Within an hour the network will stall in xen-3.4.1-rc5
but not in xen-3.3.2-rc3. ifconfig, route -n and ip link show normal
but unable to ping gateway. Sometimes, doing the following (in
crontab using custom script to check ping gateway and if 100% packet
lost will execute the below can bring back the network but not always
and needed a reboot):
1. xm shutdown all domUs
2. service xendomains stop
3. stop network-bridge
4. service xend stop
5. service xend start
6. xm create all domUs
However the above might cause some domU ext3 file system dirty and
e2fsck is required.
I have done many tests (at least more than 5 times on 3 DELL PE850/860
servers) and the results are the same. With xen-3.3.2-rc3 no issue
and network will not be down/stalled doing the scp transfer test to
other server. Whereby with xen-3.4.1-rc5, it will happen within an
hour if such test are carried out at least 5 instances running
concurrently. In fact from xen-3.4.0 to xen-3.4.1-rc1 to rc5 are the
same.
/var/log/messages will show the following when network stall:
tg3: peth0: transmit timed out, resetting
I have tried:
/sbin/ethtool -K eth0 tx off
/sbin/ethtool -K eth0 rx off
/sbin/ethtool -K eth0 gso off
/sbin/ethtool -K eth0 tso off
Is there any netfront/netback changes between xen-3.3.x and xen-3.4.x
which cause such issue? Anybody experience such network stall in your
tg3 in bridge network environment?
The above test also carried out in non tg3 servers such as with
e100/e1000 drivers do not cause such network stall problem.
All servers are running CentOS 5.3 with linux-2.6.18.8-xen for all
dom0s and domUs.
Any idea?
Thanks.
Kindest regards,
Giam Teck Choon
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: tg3 network stall in xen-3.4.x but not in xen-3.3.x
2009-07-04 6:32 tg3 network stall in xen-3.4.x but not in xen-3.3.x Teck Choon Giam
@ 2009-07-04 6:41 ` Keir Fraser
2009-07-04 6:59 ` Teck Choon Giam
0 siblings, 1 reply; 10+ messages in thread
From: Keir Fraser @ 2009-07-04 6:41 UTC (permalink / raw)
To: Teck Choon Giam, xen-devel@lists.xensource.com
On 04/07/2009 07:32, "Teck Choon Giam" <giamteckchoon@gmail.com> wrote:
> Is there any netfront/netback changes between xen-3.3.x and xen-3.4.x
> which cause such issue? Anybody experience such network stall in your
> tg3 in bridge network environment?
You are running exactly the same kernels on both 3.3 and 3.4? Then there can
be no netfront/back differences as you are running exactly the same drivers.
Certain things are on by default in 3.4 which were not in 3.3. One such
thing is MSI. You might try disabling CONFIG_PCI_MSI in your dom0 kernel
config. It'll be listed as "Message Signaled Interrupts (MSI and MSI-X)" in
the PCI section of the Linux config menus.
-- Keir
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: tg3 network stall in xen-3.4.x but not in xen-3.3.x
2009-07-04 6:41 ` Keir Fraser
@ 2009-07-04 6:59 ` Teck Choon Giam
2009-07-04 7:09 ` Keir Fraser
0 siblings, 1 reply; 10+ messages in thread
From: Teck Choon Giam @ 2009-07-04 6:59 UTC (permalink / raw)
To: xen-devel
Hi Keir,
On Sat, Jul 4, 2009 at 2:41 PM, Keir Fraser<keir.fraser@eu.citrix.com> wrote:
> On 04/07/2009 07:32, "Teck Choon Giam" <giamteckchoon@gmail.com> wrote:
>
>> Is there any netfront/netback changes between xen-3.3.x and xen-3.4.x
>> which cause such issue? Anybody experience such network stall in your
>> tg3 in bridge network environment?
>
> You are running exactly the same kernels on both 3.3 and 3.4? Then there can
> be no netfront/back differences as you are running exactly the same drivers.
Yes, same kernel version for both since compiling both xen versions
using the same linux-2.6.18-xen.hg.
>
> Certain things are on by default in 3.4 which were not in 3.3. One such
> thing is MSI. You might try disabling CONFIG_PCI_MSI in your dom0 kernel
> config. It'll be listed as "Message Signaled Interrupts (MSI and MSI-X)" in
> the PCI section of the Linux config menus.
I checked on both configs and both are showing me:
# CONFIG_PCI_MSI is not set
I also done a diff for both .config and they are the same.
Thanks.
Kindest regards,
Giam Teck Choon
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: tg3 network stall in xen-3.4.x but not in xen-3.3.x
2009-07-04 6:59 ` Teck Choon Giam
@ 2009-07-04 7:09 ` Keir Fraser
2009-07-04 7:30 ` Teck Choon Giam
0 siblings, 1 reply; 10+ messages in thread
From: Keir Fraser @ 2009-07-04 7:09 UTC (permalink / raw)
To: Teck Choon Giam, xen-devel@lists.xensource.com
On 04/07/2009 07:59, "Teck Choon Giam" <giamteckchoon@gmail.com> wrote:
>> Certain things are on by default in 3.4 which were not in 3.3. One such
>> thing is MSI. You might try disabling CONFIG_PCI_MSI in your dom0 kernel
>> config. It'll be listed as "Message Signaled Interrupts (MSI and MSI-X)" in
>> the PCI section of the Linux config menus.
>
> I checked on both configs and both are showing me:
> # CONFIG_PCI_MSI is not set
>
> I also done a diff for both .config and they are the same.
Power management is another difference between 3.3 and 3.4. You can disable
3.4 power management by adding Xen boot parameters: cpuidle=0 cpufreq=none
-- Keir
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: tg3 network stall in xen-3.4.x but not in xen-3.3.x
2009-07-04 7:09 ` Keir Fraser
@ 2009-07-04 7:30 ` Teck Choon Giam
2009-07-05 2:56 ` Teck Choon Giam
0 siblings, 1 reply; 10+ messages in thread
From: Teck Choon Giam @ 2009-07-04 7:30 UTC (permalink / raw)
To: xen-devel
Hi Keir,
> Power management is another difference between 3.3 and 3.4. You can disable
> 3.4 power management by adding Xen boot parameters: cpuidle=0 cpufreq=none
In xen-3.3.x I got:
# xenpm get-cpuidle-states
Xen cpuidle is not enabled!
Xen cpufreq is not enabled!
In xen-3.4.x I got:
# xenpm get-cpuidle-states
Max C-state: C1
cpu id : 0
total C-states : 2
idle time(ms) : 131588676
C0 : transition [00000000000019346170]
residency [00000000000003897999 ms]
C1 : transition [00000000000019346170]
residency [00000000000131507268 ms]
cpu id : 1
total C-states : 2
idle time(ms) : 131696919
C0 : transition [00000000000012247741]
residency [00000000000003766854 ms]
C1 : transition [00000000000012247741]
residency [00000000000131638414 ms]
cpu id : 2
total C-states : 2
idle time(ms) : 131540647
C0 : transition [00000000000013405442]
residency [00000000000003922680 ms]
C1 : transition [00000000000013405442]
residency [00000000000131482588 ms]
cpu id : 3
total C-states : 2
idle time(ms) : 131527968
C0 : transition [00000000000031194790]
residency [00000000000004030618 ms]
C1 : transition [00000000000031194790]
residency [00000000000131374650 ms]
I will disable and run the test tomorrow to see whether network stall
issue is there or not.
Thanks.
Kindest regards,
Giam Teck Choon
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: tg3 network stall in xen-3.4.x but not in xen-3.3.x
2009-07-04 7:30 ` Teck Choon Giam
@ 2009-07-05 2:56 ` Teck Choon Giam
2009-07-05 21:36 ` Ian Pratt
0 siblings, 1 reply; 10+ messages in thread
From: Teck Choon Giam @ 2009-07-05 2:56 UTC (permalink / raw)
To: xen-devel
On Sat, Jul 4, 2009 at 3:30 PM, Teck Choon Giam<giamteckchoon@gmail.com> wrote:
Hi Keir,
>> Power management is another difference between 3.3 and 3.4. You can disable
>> 3.4 power management by adding Xen boot parameters: cpuidle=0 cpufreq=none
> I will disable and run the test tomorrow to see whether network stall
> issue is there or not.
Using cpuidle=0 cpufreq=none seems to solve the network stall problem.
Thanks.
Kindest regards,
Giam Teck Choon
^ permalink raw reply [flat|nested] 10+ messages in thread
* RE: tg3 network stall in xen-3.4.x but not in xen-3.3.x
2009-07-05 2:56 ` Teck Choon Giam
@ 2009-07-05 21:36 ` Ian Pratt
2009-07-06 3:55 ` Teck Choon Giam
0 siblings, 1 reply; 10+ messages in thread
From: Ian Pratt @ 2009-07-05 21:36 UTC (permalink / raw)
To: Teck Choon Giam, xen-devel@lists.xensource.com; +Cc: Ian Pratt
> >> Power management is another difference between 3.3 and 3.4. You can
> disable
> >> 3.4 power management by adding Xen boot parameters: cpuidle=0
> cpufreq=none
>
> > I will disable and run the test tomorrow to see whether network stall
> > issue is there or not.
>
> Using cpuidle=0 cpufreq=none seems to solve the network stall problem.
Hmm, that's rather disturbing. Its presumably the cpuidle parameter which is having the effect. Quite why deeper sleep states can result in one particular device interrupt getting stuck (as opposed to all of them) is a mystery. It might be interesting to see the boot messages, and also to find out which of the C states is causing the problem (presumably C2 or C3).
In your tests, rather than rebooting the machine you may possibly be able to recover the machine by unloading and reloading the NIC module. (you may need to remove it from the bridge and ifconfig it down first).
Ian
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: tg3 network stall in xen-3.4.x but not in xen-3.3.x
2009-07-05 21:36 ` Ian Pratt
@ 2009-07-06 3:55 ` Teck Choon Giam
2009-07-06 7:19 ` Keir Fraser
0 siblings, 1 reply; 10+ messages in thread
From: Teck Choon Giam @ 2009-07-06 3:55 UTC (permalink / raw)
To: xen-devel
Hi Ian,
On Mon, Jul 6, 2009 at 5:36 AM, Ian Pratt<Ian.Pratt@eu.citrix.com> wrote:
>> >> Power management is another difference between 3.3 and 3.4. You can
>> disable
>> >> 3.4 power management by adding Xen boot parameters: cpuidle=0
>> cpufreq=none
>>
>> > I will disable and run the test tomorrow to see whether network stall
>> > issue is there or not.
>>
>> Using cpuidle=0 cpufreq=none seems to solve the network stall problem.
>
> Hmm, that's rather disturbing. Its presumably the cpuidle parameter which is having the effect. Quite why deeper sleep states can result in one particular device interrupt getting stuck (as opposed to all of them) is a mystery. It might be interesting to see the boot messages, and also to find out which of the C states is causing the problem (presumably C2 or C3).
If I do not add cpuidle and cpufreq in xen boot para. I got the below:
# xenpm get-cpuidle-states
Max C-state: C1
cpu id : 0
total C-states : 2
idle time(ms) : 131588676
C0 : transition [00000000000019346170]
residency [00000000000003897999 ms]
C1 : transition [00000000000019346170]
residency [00000000000131507268 ms]
cpu id : 1
total C-states : 2
idle time(ms) : 131696919
C0 : transition [00000000000012247741]
residency [00000000000003766854 ms]
C1 : transition [00000000000012247741]
residency [00000000000131638414 ms]
cpu id : 2
total C-states : 2
idle time(ms) : 131540647
C0 : transition [00000000000013405442]
residency [00000000000003922680 ms]
C1 : transition [00000000000013405442]
residency [00000000000131482588 ms]
cpu id : 3
total C-states : 2
idle time(ms) : 131527968
C0 : transition [00000000000031194790]
residency [00000000000004030618 ms]
C1 : transition [00000000000031194790]
residency [00000000000131374650 ms]
Sorry, I am unable to give you more details as currently all are
booted with cpuidle and cpufreq in xen boot para. I will try to
migrate one of the server VMs to another then use that to test without
cpuidle and cpufreq in xen boot para. then will report back my
findings. In fact now all are with:
kernel /xen.gz dom0_mem=256M loglvl=all guest_loglvl=all cpuidle=0 cpufreq=none
If you have any suggestion to add in xen boot para. or any other, feel
free to let me know ;)
> In your tests, rather than rebooting the machine you may possibly be able to recover the machine by unloading and reloading the NIC module. (you may need to remove it from the bridge and ifconfig it down first).
Yes, shutdown all xendomains, shutdown network-bridge and xend then
restart them without the need to restart network can bring back the
network most of the time but it is disturbing as all VMs will need to
shutdown clearly to prevent ext3 file system dirty.
I noticed for other servers that without the cpuidle=0 cpufreq=none in
xen-3.4.x, xenpm get-cpuidle-states showing:
# xenpm get-cpuidle-states
Max C-state: C7
Is this due to the processor type since they are not dual core and/or
quad core or multi-processors and whether is it a VT-d enabled system
type?
# cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 15
model : 4
model name : Intel(R) Xeon(TM) CPU 3.00GHz
stepping : 3
cpu MHz : 3000.112
cache size : 2048 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 5
wp : yes
flags : fpu de tsc msr pae mce cx8 apic mtrr mca cmov pat
clflush acpi mmx fxsr sse sse2 ss ht nx constant_tsc pni cid
bogomips : 6004.86
processor : 1
vendor_id : GenuineIntel
cpu family : 15
model : 4
model name : Intel(R) Xeon(TM) CPU 3.00GHz
stepping : 3
cpu MHz : 3000.112
cache size : 2048 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 5
wp : yes
flags : fpu de tsc msr pae mce cx8 apic mtrr mca cmov pat
clflush acpi mmx fxsr sse sse2 ss ht nx constant_tsc pni cid
bogomips : 6004.86
The above server is not DELL but is a Tyan server:
# lspci -vvv|grep -i ethernet
01:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5721
Gigabit Ethernet PCI Express (rev 11)
Subsystem: Broadcom Corporation NetXtreme BCM5721 Gigabit
Ethernet PCI Express
02:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5721
Gigabit Ethernet PCI Express (rev 11)
Subsystem: Broadcom Corporation NetXtreme BCM5721 Gigabit
Ethernet PCI Express
Doing test on this server is ok with no network stall however this
server will crash within a month time and when I plug in
monitor/keyboard can't see any output nor cltr+alt+delete got any
response. The only thing I can do is to reboot the server then this
cycle will repeat... sudden crash within a month and sometimes can
happen 2 or more times within a month. So this server is running a
backup domU and a mirror domU which are not so critical. Due to
sudden crash issue on this type of server(s) (I got two such server
having same issue), thus can't really run this in real production :(
Thanks.
Kindest regards,
Giam Teck Choon
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: tg3 network stall in xen-3.4.x but not in xen-3.3.x
2009-07-06 3:55 ` Teck Choon Giam
@ 2009-07-06 7:19 ` Keir Fraser
2009-07-07 1:29 ` Teck Choon Giam
0 siblings, 1 reply; 10+ messages in thread
From: Keir Fraser @ 2009-07-06 7:19 UTC (permalink / raw)
To: Teck Choon Giam, xen-devel@lists.xensource.com
On 06/07/2009 04:55, "Teck Choon Giam" <giamteckchoon@gmail.com> wrote:
>> Hmm, that's rather disturbing. Its presumably the cpuidle parameter which is
>> having the effect. Quite why deeper sleep states can result in one particular
>> device interrupt getting stuck (as opposed to all of them) is a mystery. It
>> might be interesting to see the boot messages, and also to find out which of
>> the C states is causing the problem (presumably C2 or C3).
>
> If I do not add cpuidle and cpufreq in xen boot para. I got the below:
>
> # xenpm get-cpuidle-states
> Max C-state: C1
>
> cpu id : 0
> total C-states : 2
That's interesting, since it appears the troublesome system does not even
support deep sleep states (e.g., C3). Just C0 and C1: which would normally
mean C0=running, C1=normal-HLT. I've cc'ed a couple of Intel guys to confirm
we couldn't be misreading the xenpm output.
If we're reading this correctly I think it really means that the special
acpi-cx idle handler has a bug in it somewhere. Actually one bug has been
found already, and I will forward the patch to you. It could be worth
applying it and rebuilding Xen and see if we're lucky enough for that to
solve your problem.
-- Keir
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: tg3 network stall in xen-3.4.x but not in xen-3.3.x
2009-07-06 7:19 ` Keir Fraser
@ 2009-07-07 1:29 ` Teck Choon Giam
0 siblings, 0 replies; 10+ messages in thread
From: Teck Choon Giam @ 2009-07-07 1:29 UTC (permalink / raw)
To: xen-devel
Hi keir,
> That's interesting, since it appears the troublesome system does not even
> support deep sleep states (e.g., C3). Just C0 and C1: which would normally
> mean C0=running, C1=normal-HLT. I've cc'ed a couple of Intel guys to confirm
> we couldn't be misreading the xenpm output.
Oh ok.
> If we're reading this correctly I think it really means that the special
> acpi-cx idle handler has a bug in it somewhere. Actually one bug has been
> found already, and I will forward the patch to you. It could be worth
> applying it and rebuilding Xen and see if we're lucky enough for that to
> solve your problem.
I can test the patch then report back. Please send me the patch.
Thanks.
Kindest regards,
Giam Teck Choon
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2009-07-07 1:29 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2009-07-04 6:32 tg3 network stall in xen-3.4.x but not in xen-3.3.x Teck Choon Giam
2009-07-04 6:41 ` Keir Fraser
2009-07-04 6:59 ` Teck Choon Giam
2009-07-04 7:09 ` Keir Fraser
2009-07-04 7:30 ` Teck Choon Giam
2009-07-05 2:56 ` Teck Choon Giam
2009-07-05 21:36 ` Ian Pratt
2009-07-06 3:55 ` Teck Choon Giam
2009-07-06 7:19 ` Keir Fraser
2009-07-07 1:29 ` Teck Choon Giam
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.