Sending packet at fixed rate really bad performance

Sending packet at fixed rate really bad performance

[Marco Tizzoni]

Hi all,
I'm doing some tests about i/o performance and cpu overhead in dom0.
I've coded a small tool that can sends UDP packets at fixed rate or
flood. While sending packet at fixed rate I'm experiencing performance
problem, related to how Xen  handle signal, I guess.

On my laptop (Core2 duo P8700), with no Xen running, I can send
packets at a rate of 20k, 40k, 60k, 80k/s achieving an actual packet
rate per second pretty near to the requested rate.

On Xen (dom0 with no guest running and no load - Athlon 64 dual core
4000+ ), the performance are really bad.
Here what happen:

Requested | Actual
10             | 10.03
100           | 100.05
200           | 200.04
500           | 250.04
1000         | 250.18
10000        | 247.49
20000        | 250.26
40000        | 250.24

It seems impossibile to go over 250 packets/second, but in flood
configuration (send packet as fast as possible) the actual rate is
quite higher, 39k packets/second about.

The main difference is in the code. Flood loop as been implemented like this:
------------------
for (;;)
  send_pkt();
------------------

while the fixed rate use setitimer to wake up send_pkt() when needed.

Any idea of why this happen?

bye,
Marco

Re: Sending packet at fixed rate really bad performance

[Konrad Rzeszutek Wilk]

On Sun, Sep 27, 2009 at 07:26:49PM +0200, Marco Tizzoni wrote:
> Hi all,
> I'm doing some tests about i/o performance and cpu overhead in dom0.
> I've coded a small tool that can sends UDP packets at fixed rate or
> flood. While sending packet at fixed rate I'm experiencing performance
> problem, related to how Xen  handle signal, I guess.
> 
> On my laptop (Core2 duo P8700), with no Xen running, I can send
> packets at a rate of 20k, 40k, 60k, 80k/s achieving an actual packet
> rate per second pretty near to the requested rate.
> 
> On Xen (dom0 with no guest running and no load - Athlon 64 dual core
> 4000+ ), the performance are really bad.

Lets narrow this down a bit.  You are comparing two different systems
with potentially different interfaces. Lets concentrate on one machine
first. Is the performance on your Athlon 64 without Xen (meaning you
boot without Xen) in the same ball-park figure?

I want to make sure we can eliminate the hardware being at fault here.

Re: Sending packet at fixed rate really bad performance

[Marco Tizzoni]

[-- Attachment #1: Type: text/plain, Size: 1285 bytes --]

On Mon, Sep 28, 2009 at 4:17 PM, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:
> Lets narrow this down a bit.  You are comparing two different systems
> with potentially different interfaces. Lets concentrate on one machine
> first. Is the performance on your Athlon 64 without Xen (meaning you
> boot without Xen) in the same ball-park figure?
> I want to make sure we can eliminate the hardware being at fault here.

I've already done many tests to isolate the problem before writing here.
I've tried to use a different nic, different LAN, different switch and
loopback as well, same results.
Anyway forget aboout Athlon and let compare the same software, on the
same hardware with linux and xenolinux.

I've connected two pc running linux via a cross-cable and made some
tests. The test achieves good results in this case for rate over 10k
packets/second.
On xenolinux (credit and sedf behaviour the same) the performance are
really bad, 250 packets/second (as before).

Also I've just modified my software to not send packets, but simply
count, so no sendto() is involved. Same low rate.

May be there's something wrong in my code (not yet finished). Attached
you can find it if you want have a look or try to reproduce this
issue.

thx,
Marco

[-- Attachment #2: main.c.gz --]
[-- Type: application/x-gzip, Size: 2454 bytes --]

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Sending packet at fixed rate really bad performance

[Konrad Rzeszutek Wilk]

On Mon, Sep 28, 2009 at 06:28:04PM +0200, Marco Tizzoni wrote:
> On Mon, Sep 28, 2009 at 4:17 PM, Konrad Rzeszutek Wilk
> <konrad.wilk@oracle.com> wrote:
> > Lets narrow this down a bit.  You are comparing two different systems
> > with potentially different interfaces. Lets concentrate on one machine
> > first. Is the performance on your Athlon 64 without Xen (meaning you
> > boot without Xen) in the same ball-park figure?
> > I want to make sure we can eliminate the hardware being at fault here.
> 
> I've already done many tests to isolate the problem before writing here.
> I've tried to use a different nic, different LAN, different switch and
> loopback as well, same results.

The problem you described was with two guests. Were those two guests
on two different machines or on the same box?

> Anyway forget aboout Athlon and let compare the same software, on the
> same hardware with linux and xenolinux.
> 
> I've connected two pc running linux via a cross-cable and made some
> tests. The test achieves good results in this case for rate over 10k
> packets/second.
> On xenolinux (credit and sedf behaviour the same) the performance are
> really bad, 250 packets/second (as before).
>

This is with two machines, each connected via cross-cable, running the same
version of Linux, and each running under Xen, correct? The tests executing
under dom0 on both machines?
 
> Also I've just modified my software to not send packets, but simply
> count, so no sendto() is involved. Same low rate.
> 
> May be there's something wrong in my code (not yet finished). Attached
> you can find it if you want have a look or try to reproduce this
> issue.
> 
> thx,
> Marco


> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: Sending packet at fixed rate really bad performance

[Marco Tizzoni]

On Mon, Sep 28, 2009 at 6:44 PM, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:

> The problem you described was with two guests. Were those two guests
> on two different machines or on the same box?

The 1st box runs Xen and sends packet to the 2nd box.
The test runs on dom0 with no load at all (the same behaviour occurs
with 1 domU sending packets and dom0 doing nothing).
The 2nd box doesn't run xen and receive and counts udp packets
received without responding.


>> Anyway forget aboout Athlon and let compare the same software, on the
>> same hardware with linux and xenolinux.
>>
>> I've connected two pc running linux via a cross-cable and made some
>> tests. The test achieves good results in this case for rate over 10k
>> packets/second.
>> On xenolinux (credit and sedf behaviour the same) the performance are
>> really bad, 250 packets/second (as before).
>>
>
> This is with two machines, each connected via cross-cable, running the same
> version of Linux, and each running under Xen, correct?

No, the 1st runs Xen and 2nd not.

> The tests executing
> under dom0 on both machines?

dom0 on the first box. domU, as said before, act in the same way.

Marco

Re: Sending packet at fixed rate really bad performance

[Richard Brady]

[-- Attachment #1.1: Type: text/plain, Size: 929 bytes --]

Hi Marco

On Sun, Sep 27, 2009 at 07:26:49PM +0200, Marco Tizzoni wrote:
> Hi all,
> I'm doing some tests about i/o performance and cpu overhead in dom0.
> I've coded a small tool that can sends UDP packets at fixed rate or
> flood. While sending packet at fixed rate I'm experiencing performance
> problem, related to how Xen  handle signal, I guess.
>
> On my laptop (Core2 duo P8700), with no Xen running, I can send
> packets at a rate of 20k, 40k, 60k, 80k/s achieving an actual packet
> rate per second pretty near to the requested rate.
>
> On Xen (dom0 with no guest running and no load - Athlon 64 dual core
> 4000+ ), the performance are really bad.

Did you find a solution to this?

We are experiencing the same problem on hosts which are processing
VoIP packets. Like your test application, the packets are periodic but
high in volume.

Regards,
Richard

--
Richard Brady
T: +44 (0)7771 623 348
E: rnbrady@gmail.com

[-- Attachment #1.2: Type: text/html, Size: 2093 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel