massive performance drop after a while when using virtio nics

massive performance drop after a while when using virtio nics

[Fabian Deutsch]

Hey.

When running a guest in the following setup, I observ a "massive"
performance decrease after a couple of gigabytes of transfrerred data.

other guests keep running fine (no regressions) and nothing unusual is
observed on the host.

fedora 8 host, kvm-72 (user and module)

fedora 9 guest, all updates.
        nic 0, vtio:
        subnet a
        samba listening

        nic 1, vtio:
        subnet b
        iscsi initiator running

        samba shares a mount point, pointing to an iscsi target.


now pulling/pushing about 80GB over samba results in a drop from about
60/70Mbit to about 1/2Mbit of troughput.

The problem is reproducible.
The problem is solved after a reboot of the guest. Just restarting
networking doesn't help.
Guest's dmesg doesn't say anything about problems. No errors in
ifconfig.

Has someone observed something similar?


Greetings
- fabian

Re: massive performance drop after a while when using virtio nics

[Yann Dupont]

Fabian Deutsch a écrit :
> Hey.
>
> When running a guest in the following setup, I observ a "massive"
> performance decrease after a couple of gigabytes of transfrerred data.
>
> other guests keep running fine (no regressions) and nothing unusual is
> observed on the host.
>
> fedora 8 host, kvm-72 (user and module)
>
> fedora 9 guest, all updates.
>         nic 0, vtio:
>         subnet a
>         samba listening
>
>         nic 1, vtio:
>         subnet b
>         iscsi initiator running
>
>         samba shares a mount point, pointing to an iscsi target.
>
>
> now pulling/pushing about 80GB over samba results in a drop from about
> 60/70Mbit to about 1/2Mbit of troughput.
>
> The problem is reproducible.
> The problem is solved after a reboot of the guest. Just restarting
> networking doesn't help.
> Guest's dmesg doesn't say anything about problems. No errors in
> ifconfig.
>
> Has someone observed something similar?
>
>
> Greetings
> - fabian
>
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>   
We just saw the very same issue.
guest & host :  2.6.25.3, debian etch 64 bits

KVM 70 on this machine.

As fabian said,  rebooting the  kernel doesn't help, but rebooting the 
whole kvm guest  does.

No error messages as far as I can tell. The kvm guest is a clonezilla 
server and send lots of data.
It began to crawl after 2/3 weeks of usage.

Greetings,

-- 
Yann Dupont - Service IRTS, DSI Université de Nantes
Tel : 02.51.12.53.91 - Mail/Jabber : Yann.Dupont@univ-nantes.fr

Re: massive performance drop after a while when using virtio nics

[Fabian Deutsch]

Am Montag, den 08.09.2008, 14:07 +0200 schrieb Yann Dupont:
> Fabian Deutsch a écrit :
> > Hey.
> >
> > When running a guest in the following setup, I observ a "massive"
> > performance decrease after a couple of gigabytes of transfrerred data.
> >
> > other guests keep running fine (no regressions) and nothing unusual is
> > observed on the host.
> >
> > fedora 8 host, kvm-72 (user and module)
> >
> > fedora 9 guest, all updates.
> >         nic 0, vtio:
> >         subnet a
> >         samba listening
> >
> >         nic 1, vtio:
> >         subnet b
> >         iscsi initiator running
> >
> >         samba shares a mount point, pointing to an iscsi target.
> >
> >
> > now pulling/pushing about 80GB over samba results in a drop from about
> > 60/70Mbit to about 1/2Mbit of troughput.
> >
> > The problem is reproducible.
> > The problem is solved after a reboot of the guest. Just restarting
> > networking doesn't help.
> > Guest's dmesg doesn't say anything about problems. No errors in
> > ifconfig.
> >
> > Has someone observed something similar?
> >
> >
> > Greetings
> > - fabian
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe kvm" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >
> >   
> We just saw the very same issue.
> guest & host :  2.6.25.3, debian etch 64 bits
> 
> KVM 70 on this machine.
> 
> As fabian said,  rebooting the  kernel doesn't help, but rebooting the 
> whole kvm guest  does.
> 
> No error messages as far as I can tell. The kvm guest is a clonezilla 
> server and send lots of data.
> It began to crawl after 2/3 weeks of usage.

Are you also using model=vortio?
If so, you might want to switch to e1000. Less performance but stable.

The issues seems to be related to virtio/tap.
And a little bit more to the tap itnerface/implementation, because of
the reboot-observation.

fabian

Re: massive performance drop after a while when using virtio nics

[Yann Dupont]

Fabian Deutsch a écrit :
>> We just saw the very same issue.
>> guest & host :  2.6.25.3, debian etch 64 bits
>>
>>     
...
>
> Are you also using model=vortio?
> If so, you might want to switch to e1000. Less performance but stable.
>
> The issues seems to be related to virtio/tap.
> And a little bit more to the tap itnerface/implementation, because of
> the reboot-observation.
>
> fabian
>
>   
Yes it is virtio.
e1000 works but it is pure emulation, so :
Less performance
More  CPU usage.

This server is in pre-prod right now, So I'd rather try to help 
debugging the issue for the moment.

greetings,

-- 
Yann Dupont - Service IRTS, DSI Université de Nantes
Tel : 02.51.12.53.91 - Mail/Jabber : Yann.Dupont@univ-nantes.fr

Re: massive performance drop after a while when using virtio nics

[Mark McLoughlin]

Hi,

On Sat, 2008-08-23 at 21:21 +0200, Fabian Deutsch wrote:
> Hey.
> 
> When running a guest in the following setup, I observ a "massive"
> performance decrease after a couple of gigabytes of transfrerred data.
> 
> other guests keep running fine (no regressions) and nothing unusual is
> observed on the host.

If you do "killall -ALRM qemu-kvm" after the slowdown has occurred, does
it make things speed up again for a while?

If so, the issue is fixed by:

  http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=ba661292a2bc6ddd305a212b0526e5dc22195fe7

It also got applied in 2.6.26.3 and 2.6.25.16.

Cheers,
Mark.

Re: massive performance drop after a while when using virtio nics

[Fabian Deutsch]

Am Dienstag, den 09.09.2008, 11:22 +0100 schrieb Mark McLoughlin:
> Hi,
> 
> On Sat, 2008-08-23 at 21:21 +0200, Fabian Deutsch wrote:
> > Hey.
> > 
> > When running a guest in the following setup, I observ a "massive"
> > performance decrease after a couple of gigabytes of transfrerred data.
> > 
> > other guests keep running fine (no regressions) and nothing unusual is
> > observed on the host.
> 
> If you do "killall -ALRM qemu-kvm" after the slowdown has occurred, does
> it make things speed up again for a while?
> 

It seems so! 
Fantastic.

> If so, the issue is fixed by:
> 
>   http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=ba661292a2bc6ddd305a212b0526e5dc22195fe7
> 
> It also got applied in 2.6.26.3 and 2.6.25.16.
> 

So it was some signaling-bug on the .. client side?

Cheers!
fabian

Re: massive performance drop after a while when using virtio nics

[Mark McLoughlin]

On Tue, 2008-09-09 at 12:32 +0200, Fabian Deutsch wrote:
> Am Dienstag, den 09.09.2008, 11:22 +0100 schrieb Mark McLoughlin:
> > Hi,
> > 
> > On Sat, 2008-08-23 at 21:21 +0200, Fabian Deutsch wrote:
> > > Hey.
> > > 
> > > When running a guest in the following setup, I observ a "massive"
> > > performance decrease after a couple of gigabytes of transfrerred data.
> > > 
> > > other guests keep running fine (no regressions) and nothing unusual is
> > > observed on the host.
> > 
> > If you do "killall -ALRM qemu-kvm" after the slowdown has occurred, does
> > it make things speed up again for a while?
> > 
> 
> It seems so! 
> Fantastic.
> 
> > If so, the issue is fixed by:
> > 
> >   http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=ba661292a2bc6ddd305a212b0526e5dc22195fe7
> > 
> > It also got applied in 2.6.26.3 and 2.6.25.16.
> > 
> 
> So it was some signaling-bug on the .. client side?

No, it was a kernel race condition causing the qemu-kvm process to stop
receiving SIGALRM signals.

The original thread might explain it a bit better:

  http://thread.gmane.org/gmane.comp.emulators.kvm.devel/19582

Cheers,
Mark.

Re: massive performance drop after a while when using virtio nics

[Henrik Holst]

tis 2008-09-09 klockan 11:47 +0100 skrev Mark McLoughlin:
> On Tue, 2008-09-09 at 12:32 +0200, Fabian Deutsch wrote:
> > Am Dienstag, den 09.09.2008, 11:22 +0100 schrieb Mark McLoughlin:
> > > Hi,
> > > 
> > > On Sat, 2008-08-23 at 21:21 +0200, Fabian Deutsch wrote:
> > > > Hey.
> > > > 
> > > > When running a guest in the following setup, I observ a "massive"
> > > > performance decrease after a couple of gigabytes of transfrerred data.
> > > > 
> > > > other guests keep running fine (no regressions) and nothing unusual is
> > > > observed on the host.
> > > 
> > > If you do "killall -ALRM qemu-kvm" after the slowdown has occurred, does
> > > it make things speed up again for a while?
> > > 
> > 
> > It seems so! 
> > Fantastic.
> > 
> > > If so, the issue is fixed by:
> > > 
> > >   http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=ba661292a2bc6ddd305a212b0526e5dc22195fe7
> > > 
> > > It also got applied in 2.6.26.3 and 2.6.25.16.
> > > 
> > 
> > So it was some signaling-bug on the .. client side?
> 
> No, it was a kernel race condition causing the qemu-kvm process to stop
> receiving SIGALRM signals.

So is it the host kernel that should be patched?

/Henrik Holst
Witsbits AB

Re: massive performance drop after a while when using virtio nics

[Mark McLoughlin]

On Tue, 2008-09-09 at 13:11 +0200, Henrik Holst wrote:
> tis 2008-09-09 klockan 11:47 +0100 skrev Mark McLoughlin:
> > On Tue, 2008-09-09 at 12:32 +0200, Fabian Deutsch wrote:
> > > Am Dienstag, den 09.09.2008, 11:22 +0100 schrieb Mark McLoughlin:
> > > > Hi,
> > > > 
> > > > On Sat, 2008-08-23 at 21:21 +0200, Fabian Deutsch wrote:
> > > > > Hey.
> > > > > 
> > > > > When running a guest in the following setup, I observ a "massive"
> > > > > performance decrease after a couple of gigabytes of transfrerred data.
> > > > > 
> > > > > other guests keep running fine (no regressions) and nothing unusual is
> > > > > observed on the host.
> > > > 
> > > > If you do "killall -ALRM qemu-kvm" after the slowdown has occurred, does
> > > > it make things speed up again for a while?
> > > > 
> > > 
> > > It seems so! 
> > > Fantastic.
> > > 
> > > > If so, the issue is fixed by:
> > > > 
> > > >   http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=ba661292a2bc6ddd305a212b0526e5dc22195fe7
> > > > 
> > > > It also got applied in 2.6.26.3 and 2.6.25.16.
> > > > 
> > > 
> > > So it was some signaling-bug on the .. client side?
> > 
> > No, it was a kernel race condition causing the qemu-kvm process to stop
> > receiving SIGALRM signals.
> 
> So is it the host kernel that should be patched?

Yes.

(Or build qemu-kvm to not use signalfd)

Cheers,
Mark.

Re: massive performance drop after a while when using virtio nics

[Yann Dupont]

Mark McLoughlin a écrit :
> Hi,
>
> On Sat, 2008-08-23 at 21:21 +0200, Fabian Deutsch wrote:
>   
>> Hey.
>>
>> When running a guest in the following setup, I observ a "massive"
>> performance decrease after a couple of gigabytes of transfrerred data.
>>
>> other guests keep running fine (no regressions) and nothing unusual is
>> observed on the host.
>>     
>
> If you do "killall -ALRM qemu-kvm" after the slowdown has occurred, does
> it make things speed up again for a while?
>
>   
Too soon to say if it works for us as it seems the problem is long to 
trigger in our case ; Anyway as the problem seems similar, it's good to 
know it is probably fixed.

> If so, the issue is fixed by:
>
>   http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=ba661292a2bc6ddd305a212b0526e5dc22195fe7
>
> It also got applied in 2.6.26.3 and 2.6.25.16.
>
>   
Ok, so upgrading the kernel is sufficient ?



> Cheers,
> Mark.
>
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>   
Thanks for the answer,
greetings,

-- 
Yann Dupont - Service IRTS, DSI Université de Nantes
Tel : 02.51.12.53.91 - Mail/Jabber : Yann.Dupont@univ-nantes.fr