qemu-devel.nongnu.org archive mirror
 help / color / mirror / Atom feed
* Re: [Qemu-devel] virtio-net: network stops responding in Win2k3 server
@ 2014-02-06  8:05 ing. Mario De Chenno
  0 siblings, 0 replies; 6+ messages in thread
From: ing. Mario De Chenno @ 2014-02-06  8:05 UTC (permalink / raw)
  To: Yan Vugenfirer; +Cc: qemu-devel list


[-- Attachment #1.1: Type: text/plain, Size: 5697 bytes --]

Still having problems, even after upgrading to latest virtio-win
driver: DriverVer=11/20/2013,52.65.104.7400

NIC stops responding, and I often need to power off and back on the vm,
since disabling the affected nic within Windows does not always solve the
problem.
tap setting are the same as before.
ethtool -k tap-cineca-serv
Offload parameters for tap-cineca-serv:
rx-checksumming: off
tx-checksumming: off
scatter-gather: off
tcp-segmentation-offload: off
udp-fragmentation-offload: off
generic-segmentation-offload: off
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: off
tx-vlan-offload: off
ntuple-filters: off
receive-hashing: off

Debugview logs are attached end are always the same (now with debug level
set to 4)

Please tell me how I can further give information. The issue is very
annoying.

Mario


2013-12-12 ing. Mario De Chenno <mario.dechenno@unina2.it>:

> Hi all, it happened again, but not on the win32 machine I was talking
> before. In that I changed the driver to e1000 and it's up from more than 18
> days, never seen that before!
>
> Today the affected machine is a Win2k3 R2 SP2 x64 server, with all updates
> (before today's) applied. Virtio-net driver is 52.65.104.6500 19/06/2013,
> loglevel is set to 6.
>
> Qemu startup parameters are as before (one nic this time):
> qemu-system-x86_64 -enable-kvm -machine type=pc,accel=kvm -cpu kvm64 -rtc
> base=localtime -drive file=/vmstore/vm_disks/cineca-service.img,if=virtio
> -net nic,model=virtio,macaddr=CE:DA:0C:00:00:55 -net
> tap,script=br224-ifup,vhost=on,ifname=tap-cineca-service -vnc :55 -m 6144
> -boot c -k it -usbdevice tablet -name cineca-service -daemonize
>
> Offload parameters for tap-cineca-serv:
> rx-checksumming: off
> tx-checksumming: off
> scatter-gather: off
> tcp-segmentation-offload: off
> udp-fragmentation-offload: off
> generic-segmentation-offload: off
> generic-receive-offload: on
> large-receive-offload: off
> rx-vlan-offload: off
> tx-vlan-offload: off
> ntuple-filters: off
> receive-hashing: off
>
> The network interface (IP: 10.224.4.55) has just stopped responding. I now
> run ping -t 10.224.4.1 (to the local gateway) from inside the virtual
> machine and I get "request timeout".
>
> on the tap interface of the host server:
>
> > tcpdump -n -i tap-cineca-serv
> tcpdump: WARNING: tap-cineca-serv: no IPv4 address assigned
> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
> listening on tap-cineca-serv, link-type EN10MB (Ethernet), capture size
> 65535 bytes
> 10:54:06.986752 ARP, Request who-has 10.224.4.1 tell 10.224.4.55, length 28
> 10:54:12.312851 ARP, Request who-has 10.224.4.1 tell 10.224.4.55, length 28
> 10:54:14.736914 ARP, Request who-has 10.224.4.197 tell 10.224.4.55, length
> 28
> 10:54:17.781544 ARP, Request who-has 10.224.4.197 tell 10.224.4.55, length
> 28
> 10:54:17.812755 ARP, Request who-has 10.224.4.1 tell 10.224.4.55, length 28
> 10:54:23.312894 ARP, Request who-has 10.224.4.1 tell 10.224.4.55, length 28
> 10:54:23.797236 ARP, Request who-has 10.224.4.197 tell 10.224.4.55, length
> 28
> 10:54:28.812902 ARP, Request who-has 10.224.4.1 tell 10.224.4.55, length 28
> 10:54:34.312949 ARP, Request who-has 10.224.4.1 tell 10.224.4.55, length 28
> 10:54:39.812811 ARP, Request who-has 10.224.4.1 tell 10.224.4.55, length 28
> 10:54:45.312855 ARP, Request who-has 10.224.4.1 tell 10.224.4.55, length 28
> 10:54:50.813006 ARP, Request who-has 10.224.4.1 tell 10.224.4.55, length 28
>
>
> Debugview log attached.
>
> You can see, until row 00000214, a lot of "[PrepareSendEntry] Sending
> packet of 42 with NO offload" entries, while I was pinging the gateway.
> Then, from inside Windows' device manager, I disable and re-enable the
> virtio network interface and network return working as usual.
> This machine had network working continously for about 10-12 days.
>
> Hope to see news from you. I will make more checks if you ask me what to
> do.
>
> Regards,
> Mario
>
>
>
> 2013/11/26 ing. Mario De Chenno <mario.dechenno@unina2.it>
>
> Since it's a production machine, I had to try a remedy first: I changed
>> one network card (pub, of course) to e1000 and now it's up from 62 hours
>> (maybe a record!)
>>
>> Here is the output for the other card (virtio): I guess parameters should
>> have been the same for the affected interface too (they are now the same
>> for the tap interface connected to e1000 driver, and reflect the underlying
>> physical interface).
>>
>> Offload parameters for tap-solariconsi:
>> rx-checksumming: off
>> tx-checksumming: off
>> scatter-gather: off
>> tcp-segmentation-offload: off
>> udp-fragmentation-offload: off
>> generic-segmentation-offload: off
>> generic-receive-offload: on
>> large-receive-offload: off
>>
>> I cannot switch back the virtual nic just now. I will try to increase
>> debug on other machines, but most are 64bit Win2k3 OSs, so the virtio-net
>> driver is not the same (it's a 64bit version at least...).
>> Soon or later I will have some debug data to report to the list. In the
>> meantime if you have some configuration advises, feel free to post them.
>>
>> Thanks,
>>
>> Mario
>>
>>
>> 2013/11/24 Yan Vugenfirer <yvugenfi@redhat.com>
>>
>>> Hi Mario,
>>>
>>> Can you check the offload settings of the tap device that is connected
>>> to guest?
>>>
>>> Run "ethtool -k tap-solaripub".
>>>
>>> On the guest. Raise the log verbosity by going to device manager ->
>>> NetKVM device -> Advanced tab -> Logging.Level and changing it to 4. Use
>>> DebugView to record the driver tracing (enable kernel trace):
>>> http://technet.microsoft.com/en-us/sysinternals/bb896647.aspx
>>>
>>>
>>> Best regards,
>>> Yan.
>>>
>>>
>>>
>>>
>
>
> --
>
>

[-- Attachment #1.2: Type: text/html, Size: 9086 bytes --]

[-- Attachment #2: log1.log --]
[-- Type: text/x-log, Size: 1277 bytes --]

00000001	8.18.31	[PrepareSendEntry] Sending packet of 257 with NO offload	
00000002	8.19.02	[PrepareSendEntry] Sending packet of 42 with NO offload	
00000003	8.19.07	[PrepareSendEntry] Sending packet of 42 with NO offload	
00000004	8.19.08	[PrepareSendEntry] Sending packet of 42 with NO offload	
00000005	8.19.13	[PrepareSendEntry] Sending packet of 42 with NO offload	
00000006	8.19.16	[PrepareSendEntry] Sending packet of 42 with NO offload	
00000007	8.19.19	[PrepareSendEntry] Sending packet of 42 with NO offload	
00000008	8.19.28	[PrepareSendEntry] Sending packet of 42 with NO offload	
00000009	8.19.31	[PrepareSendEntry] Sending packet of 257 with NO offload	
00000010	8.19.32	[PrepareSendEntry] Sending packet of 42 with NO offload	
00000011	8.19.38	[PrepareSendEntry] Sending packet of 42 with NO offload	
00000012	8.19.48	[PrepareSendEntry] Sending packet of 42 with NO offload	
00000013	8.19.50	[PrepareSendEntry] Sending packet of 42 with NO offload	
00000014	8.19.51	[PrepareSendEntry] Sending packet of 42 with NO offload	
00000015	8.19.53	[PrepareSendEntry] Sending packet of 42 with NO offload	
00000016	8.19.57	[PrepareSendEntry] Sending packet of 42 with NO offload	
00000017	8.19.59	[PrepareSendEntry] Sending packet of 42 with NO offload	

[-- Attachment #3: log2.log --]
[-- Type: text/x-log, Size: 1953 bytes --]

00000001	8.20.56	[PrepareSendEntry] Sending packet of 257 with NO offload	
00000002	8.21.08	[PrepareSendEntry] Sending packet of 42 with NO offload	
00000003	8.21.08	[PrepareSendEntry] Sending packet of 92 with NO offload	
00000004	8.21.09	[PrepareSendEntry] Sending packet of 92 with NO offload	
00000005	8.21.10	[PrepareSendEntry] Sending packet of 92 with NO offload	
00000006	8.21.12	[PrepareSendEntry] Sending packet of 42 with NO offload	
00000007	8.21.16	[PrepareSendEntry] Sending packet of 42 with NO offload	
00000008	8.21.48	[PrepareSendEntry] Sending packet of 42 with NO offload	
00000009	8.21.52	[PrepareSendEntry] Sending packet of 42 with NO offload	
00000010	8.21.55	[PrepareSendEntry] Sending packet of 42 with NO offload	
00000011	8.21.56	[PrepareSendEntry] Sending packet of 257 with NO offload	
00000012	8.21.57	[PrepareSendEntry] Sending packet of 42 with NO offload	
00000013	8.21.59	[PrepareSendEntry] Sending packet of 42 with NO offload	
00000014	8.22.03	[PrepareSendEntry] Sending packet of 42 with NO offload	
00000015	8.22.05	[PrepareSendEntry] Sending packet of 42 with NO offload	
00000016	8.23.30	[PrepareSendEntry] Sending packet of 42 with NO offload	
00000017	8.23.32	[PrepareSendEntry] Sending packet of 42 with NO offload	
00000018	8.23.33	[PrepareSendEntry] Sending packet of 42 with NO offload	
00000019	8.23.35	[PrepareSendEntry] Sending packet of 42 with NO offload	
00000020	8.23.39	[PrepareSendEntry] Sending packet of 42 with NO offload	
00000021	8.23.41	[PrepareSendEntry] Sending packet of 42 with NO offload	
00000022	8.26.56	[PrepareSendEntry] Sending packet of 257 with NO offload	
00000023	8.28.51	[PrepareSendEntry] Sending packet of 42 with NO offload	
00000024	8.28.54	[PrepareSendEntry] Sending packet of 42 with NO offload	
00000025	8.28.56	[PrepareSendEntry] Sending packet of 42 with NO offload	
00000026	8.29.00	[PrepareSendEntry] Sending packet of 42 with NO offload	

^ permalink raw reply	[flat|nested] 6+ messages in thread
* [Qemu-devel] virtio-net: network stops responding in Win2k3 server
@ 2013-11-22 17:28 Mario De Chenno
  2013-11-24 13:59 ` Yan Vugenfirer
  0 siblings, 1 reply; 6+ messages in thread
From: Mario De Chenno @ 2013-11-22 17:28 UTC (permalink / raw)
  To: qemu-devel

[-- Attachment #1: Type: text/plain, Size: 1679 bytes --]

Hi all.
We are facing some network issues on some Windows Server 2003 machines.
In short, network seems to lock up and stops responding even to ping
requests. From TCPdump on the tap interface on the server I only see arp
request to the gateway, without replies. Increased tx overruns on the some
interface too. No errors at all on windows side. Most of the time we can
disable the interface from within windows and re-enable it to make things
work again. Rarely we have to shut down the virtual machine (Windows reboot
does not solve the problem).
The issue appears randomly without apparent relation with server activity.
One server makes a lot of small outbond connections and locks up about
every 12-24 hours. It has two nic defined and only one is affected (the
heaviest loaded). Other servers have just one nic and lock less often.
We run Qemu-kvm 1.4.0 and latest virtio-win drivers. On the same host we
did run a Linux web server with no issue in months of activity, with
traffic peaks up to 80Mb/s.

Here's is the startup command:

qemu-system-x86_64 -enable-kvm -machine type=pc,accel=kvm -cpu kvm64
-pidfile /vmstore/vm_pids/solari.pid -rtc base=localtime -drive
file=/vmstore/vm_disks/solari.img,if=virtio -netdev
tap,id=nic01,ifname=tap-solaripub,script=pub102-ifup -device
virtio-net-pci,netdev=nic01,mac=CE:DA:01:00:17:16 -netdev
tap,id=nic02,ifname=tap-solariconsip,script=consip-ifup -device
virtio-net-pci,netdev=nic02,mac=CE:DA:01:00:16:16 -vnc :16 -m 4096 -boot c
-k it -usbdevice tablet -name solari -daemonize

We tried also the following options without result
,vhost=off,vnet_hdr=on
,ioeventfd=on,event_idx=off


How can we troubleshoot the issue?
Thanks

[-- Attachment #2: Type: text/html, Size: 2073 bytes --]

^ permalink raw reply	[flat|nested] 6+ messages in thread
* [Qemu-devel] virtio-net: network stops responding in Win2k3 server
@ 2013-11-20 16:11 ing. Mario De Chenno
  0 siblings, 0 replies; 6+ messages in thread
From: ing. Mario De Chenno @ 2013-11-20 16:11 UTC (permalink / raw)
  To: qemu-devel

[-- Attachment #1: Type: text/plain, Size: 1696 bytes --]

Hi all.
We are facing some network issues on some Windows Server 2003 machines.
In short, network seems to lock up and stops responding even to ping
requests. From TCPdump on the tap interface on the server I only see arp
request to the gateway, without replies. Increased tx overruns on the some
interface too. No errors at all on windows side. Most of the time we can
disable the interface from within windows and re-enable it to make things
work again. Rarely we have to shut down the virtual machine (Windows reboot
does not solve the problem).
The issue appears randomly without apparent relation with server activity.
One server makes a lot of small outbond connections and locks up about
every 12-24 hours. It has two nic defined and only one is affected (the
heaviest loaded). Other servers have just one nic and lock less often.
We run Qemu-kvm 1.4.0 and latest virtio-win drivers. On the same host we
did run a Linux web server with no issue in months of activity, with
traffic peaks up to 80Mb/s.

Here's is the startup command:

qemu-system-x86_64 -enable-kvm -machine type=pc,accel=kvm -cpu kvm64
-pidfile /vmstore/vm_pids/solari.pid -rtc base=localtime -drive
file=/vmstore/vm_disks/solari.img,if=virtio -netdev
tap,id=nic01,ifname=tap-solaripub,script=pub102-ifup -device
virtio-net-pci,netdev=nic01,mac=CE:DA:01:00:17:16 -netdev
tap,id=nic02,ifname=tap-solariconsip,script=consip-ifup -device
virtio-net-pci,netdev=nic02,mac=CE:DA:01:00:16:16 -vnc :16 -m 4096 -boot c
-k it -usbdevice tablet -name solari -daemonize

We tried also the following options without result
,vhost=off,vnet_hdr=on
,ioeventfd=on,event_idx=off


How can we troubleshoot the issue?
Thanks

Mario De Chenno

[-- Attachment #2: Type: text/html, Size: 1965 bytes --]

^ permalink raw reply	[flat|nested] 6+ messages in thread
end of thread, other threads:[~2014-02-06  8:05 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2014-02-06  8:05 [Qemu-devel] virtio-net: network stops responding in Win2k3 server ing. Mario De Chenno
  -- strict thread matches above, loose matches on Subject: below --
2013-11-22 17:28 Mario De Chenno
2013-11-24 13:59 ` Yan Vugenfirer
2013-11-26 13:01   ` ing. Mario De Chenno
2013-12-12 10:36     ` ing. Mario De Chenno
2013-11-20 16:11 ing. Mario De Chenno

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).