what's maximum speed setting for xen network?

what's maximum speed setting for xen network?

[Xin Zhao]

For a physical network card, it has a maximum transmission speed such as
100Mb. How about network in a virtual machine? It is a virtual network
card. Is there any speed limit? Or it can transmit as fast as possible? If
so, I guess network transmission happened within a physical host should be
very fast (great than 100Mb). But this is not true in my experiment.
What's the bottleneck then?

Thanks in advance for your advice.

Xin

Re: what's maximum speed setting for xen network?

[Kip Macy]

Machine local networking is something that is currently not handled
well by xen. One ends up context switching for every packet sent. I
believe this is on the TODO list.

                                    -Kip 

On 4/20/05, Xin Zhao <zhaoxin@eecs.umich.edu> wrote:
> For a physical network card, it has a maximum transmission speed such as
> 100Mb. How about network in a virtual machine? It is a virtual network
> card. Is there any speed limit? Or it can transmit as fast as possible? If
> so, I guess network transmission happened within a physical host should be
> very fast (great than 100Mb). But this is not true in my experiment.
> What's the bottleneck then?
> 
> Thanks in advance for your advice.
> 
> Xin
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

Re: what's maximum speed setting for xen network?

[mukesh agrawal]

> For a physical network card, it has a maximum transmission speed such as
> 100Mb. How about network in a virtual machine? It is a virtual network
> card. Is there any speed limit? Or it can transmit as fast as possible? 
> If so, I guess network transmission happened within a physical host 
> should be very fast (great than 100Mb). But this is not true in my 
> experiment.

Out of curiousity, what kind of testing have you run, and what kind 
of speeds are you getting?

I've got a configuration with DomU talking to Dom0 over a vif, and seem to 
be seeing UDP losses, even at low data rates. (I haven't dug into this 
deeply to very the problem yet, though.) So I'd be interested if you're 
seeing something similar.

Thanks,
mukesh

Re: what's maximum speed setting for xen network?

[Xin Zhao]

I just setup network and tried to copy files between dom0 and domU. The
speed is around 100Mb/s. So I once thought it is simulating 100Mb network,
but after looking into the network driver, I failed to find anything about
the flow control.

I didn't try UDP, so have no idea about UDP losses.

Xin


On Fri, 22 Apr 2005, mukesh agrawal wrote:

>
> > For a physical network card, it has a maximum transmission speed such as
> > 100Mb. How about network in a virtual machine? It is a virtual network
> > card. Is there any speed limit? Or it can transmit as fast as possible?
> > If so, I guess network transmission happened within a physical host
> > should be very fast (great than 100Mb). But this is not true in my
> > experiment.
>
> Out of curiousity, what kind of testing have you run, and what kind
> of speeds are you getting?
>
> I've got a configuration with DomU talking to Dom0 over a vif, and seem to
> be seeing UDP losses, even at low data rates. (I haven't dug into this
> deeply to very the problem yet, though.) So I'd be interested if you're
> seeing something similar.
>
> Thanks,
> mukesh
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

Re: what's maximum speed setting for xen network?

[Mark Williamson]

It is possible to rate limit vifs but it won't be active by default.

What are you using to copy the files across?  If you're using scp, bear in 
bind that both virtual machines will have to do crypto, so you could actually 
be limited by CPU performance rather than virtual network.

Also, uniprocessor systems are a worst case for this sort of test because 
you'll incur a lot of context switches.  Hyperthreading should be able to 
mitigate this somewhat.

There are plans to optimise host-internal networking somewhat.

Cheers,
Mark

On Friday 22 April 2005 21:14, Xin Zhao wrote:
> I just setup network and tried to copy files between dom0 and domU. The
> speed is around 100Mb/s. So I once thought it is simulating 100Mb network,
> but after looking into the network driver, I failed to find anything about
> the flow control.
>
> I didn't try UDP, so have no idea about UDP losses.
>
> Xin
>
> On Fri, 22 Apr 2005, mukesh agrawal wrote:
> > > For a physical network card, it has a maximum transmission speed such
> > > as 100Mb. How about network in a virtual machine? It is a virtual
> > > network card. Is there any speed limit? Or it can transmit as fast as
> > > possible? If so, I guess network transmission happened within a
> > > physical host should be very fast (great than 100Mb). But this is not
> > > true in my experiment.
> >
> > Out of curiousity, what kind of testing have you run, and what kind
> > of speeds are you getting?
> >
> > I've got a configuration with DomU talking to Dom0 over a vif, and seem
> > to be seeing UDP losses, even at low data rates. (I haven't dug into this
> > deeply to very the problem yet, though.) So I'd be interested if you're
> > seeing something similar.
> >
> > Thanks,
> > mukesh
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: what's maximum speed setting for xen network?

[Xin Zhao]

Yes. I tried scp. I know CPU could be bottleneck. But I also tried ftp and
NFS, they do not have much better performance.  :) So it might be virtual
network issue.

xin


On Fri, 22 Apr 2005, Mark Williamson wrote:

> It is possible to rate limit vifs but it won't be active by default.
>
> What are you using to copy the files across?  If you're using scp, bear in
> bind that both virtual machines will have to do crypto, so you could actually
> be limited by CPU performance rather than virtual network.
>
> Also, uniprocessor systems are a worst case for this sort of test because
> you'll incur a lot of context switches.  Hyperthreading should be able to
> mitigate this somewhat.
>
> There are plans to optimise host-internal networking somewhat.
>
> Cheers,
> Mark
>
> On Friday 22 April 2005 21:14, Xin Zhao wrote:
> > I just setup network and tried to copy files between dom0 and domU. The
> > speed is around 100Mb/s. So I once thought it is simulating 100Mb network,
> > but after looking into the network driver, I failed to find anything about
> > the flow control.
> >
> > I didn't try UDP, so have no idea about UDP losses.
> >
> > Xin
> >
> > On Fri, 22 Apr 2005, mukesh agrawal wrote:
> > > > For a physical network card, it has a maximum transmission speed such
> > > > as 100Mb. How about network in a virtual machine? It is a virtual
> > > > network card. Is there any speed limit? Or it can transmit as fast as
> > > > possible? If so, I guess network transmission happened within a
> > > > physical host should be very fast (great than 100Mb). But this is not
> > > > true in my experiment.
> > >
> > > Out of curiousity, what kind of testing have you run, and what kind
> > > of speeds are you getting?
> > >
> > > I've got a configuration with DomU talking to Dom0 over a vif, and seem
> > > to be seeing UDP losses, even at low data rates. (I haven't dug into this
> > > deeply to very the problem yet, though.) So I'd be interested if you're
> > > seeing something similar.
> > >
> > > Thanks,
> > > mukesh
> > >
> > > _______________________________________________
> > > Xen-devel mailing list
> > > Xen-devel@lists.xensource.com
> > > http://lists.xensource.com/xen-devel
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

Re: what's maximum speed setting for xen network?

[Keir Fraser]

On 22 Apr 2005, at 20:53, mukesh agrawal wrote:

> I've got a configuration with DomU talking to Dom0 over a vif, and 
> seem to be seeing UDP losses, even at low data rates. (I haven't dug 
> into this deeply to very the problem yet, though.) So I'd be 
> interested if you're seeing something similar.

We have no back-pressure / flow control on inter-domain networking. 
This leads to embarrassing performance for UDP when the sender and 
receiver share a CPU. The sender fills the receiver's buffers but 
continues to send even though he drops further packets (no buffer space 
at receiver) -- then the receiver gets a time slice, receives a ring's 
worth of packets -- then the sender gets to run for another time slice, 
most of which again is wasted -- and so on.

We need a sensible mechanism for ensuring batched packet delivery, but 
also to ensure that comms rings do not so easily get overflowed. This 
would vastly improve the inter-domain performance, far more than 
striving for zero copy or other such tweaks.

It would be great if someone else is interested in investigating this. 
:-)

  -- Keir

Re: what's maximum speed setting for xen network?

[mukesh agrawal]

On Apr 23, 2005 7:31 AM,  Keir Fraser wrote:

> We have no back-pressure / flow control on inter-domain networking.
> This leads to embarrassing performance for UDP when the sender and
> receiver share a CPU. The sender fills the receiver's buffers but
> continues to send even though he drops further packets (no buffer space
> at receiver) -- then the receiver gets a time slice, receives a ring's
> worth of packets -- then the sender gets to run for another time slice,
> most of which again is wasted -- and so on.

FWIW, I don't think this is the problem I'm running into. The reason being 
that I don't think the workload I'm using is heavy enough to trigger 
overflow. I'm sending 102 packets over the course of 2 seconds. Of 
these, 5 are dropped.

Assuming the ring size is given by NETIF_RX_RING_SIZE (which is defined to 
256 in io/netif.h), I'm thinking that the drops can't be attributed to 
overflow.

Re: what's maximum speed setting for xen network?

[Keir Fraser]

On 23 Apr 2005, at 17:14, mukesh agrawal wrote:

> FWIW, I don't think this is the problem I'm running into. The reason 
> being that I don't think the workload I'm using is heavy enough to 
> trigger overflow. I'm sending 102 packets over the course of 2 
> seconds. Of these, 5 are dropped.
>
> Assuming the ring size is given by NETIF_RX_RING_SIZE (which is 
> defined to 256 in io/netif.h), I'm thinking that the drops can't be 
> attributed to overflow.

We also dynamically size the receive ring. It starts off small (8 
buffers) and is doubled in size every time packets may have been lost. 
It then slowly (linearly) shrinks when loss is not detected. The 
parameters for this are RX_MIN_TARGET and RX_MAX_TARGET in netfront.c 
-- currently they are not configurable at run-time, nor easily 
configurable at compile-time.

  -- Keir

Re: what's maximum speed setting for xen network?

[mukesh agrawal]

On Sat, 23 Apr 2005, Keir Fraser wrote:

> We also dynamically size the receive ring. It starts off small (8 buffers) 
> and is doubled in size every time packets may have been lost. It then slowly 
> (linearly) shrinks when loss is not detected. The parameters for this are 
> RX_MIN_TARGET and RX_MAX_TARGET in netfront.c -- currently they are not 
> configurable at run-time, nor easily configurable at compile-time.

If I just set RX_MIN_TARGET to NETIF_RX_RING_SIZE, and recompile, would 
that be enough to eliminate the adaptation (i.e. the ring size would 
always be 256)?

Thanks,
mukesh

Re: what's maximum speed setting for xen network?

[Keir Fraser]

On 23 Apr 2005, at 17:47, mukesh agrawal wrote:

> If I just set RX_MIN_TARGET to NETIF_RX_RING_SIZE, and recompile, 
> would that be enough to eliminate the adaptation (i.e. the ring size 
> would always be 256)?

Yes, that would do it.

  -- Keir

Re: what's maximum speed setting for xen network?

[mukesh agrawal]

> On 23 Apr 2005, at 17:47, mukesh agrawal wrote:
>
>> If I just set RX_MIN_TARGET to NETIF_RX_RING_SIZE, and recompile, would 
>> that be enough to eliminate the adaptation (i.e. the ring size would always 
>> be 256)?

FWIW, between this hack, and giving the domains more memory, the drops 
have gone away. (I haven't tried the changes separately, so I can't say 
for certain that either one alone is responsible.)

Thanks,
mukesh

Re: what's maximum speed setting for xen network?

[Nivedita Singhvi]

Keir Fraser wrote:
> 
> On 22 Apr 2005, at 20:53, mukesh agrawal wrote:
> 
>> I've got a configuration with DomU talking to Dom0 over a vif, and 
>> seem to be seeing UDP losses, even at low data rates. (I haven't dug 
>> into this deeply to very the problem yet, though.) So I'd be 
>> interested if you're seeing something similar.
> 
> 
> We have no back-pressure / flow control on inter-domain networking. This 
> leads to embarrassing performance for UDP when the sender and receiver 
> share a CPU. The sender fills the receiver's buffers but continues to 
> send even though he drops further packets (no buffer space at receiver) 
> -- then the receiver gets a time slice, receives a ring's worth of 
> packets -- then the sender gets to run for another time slice, most of 
> which again is wasted -- and so on.
> 
> We need a sensible mechanism for ensuring batched packet delivery, but 
> also to ensure that comms rings do not so easily get overflowed. This 
> would vastly improve the inter-domain performance, far more than 
> striving for zero copy or other such tweaks.
> 
> It would be great if someone else is interested in investigating this. :-)

Hi Keir,

Yep, I'm one of the people at least looking into this, don't know
if others are too :). Thrown some instrumentation in and doing some
testing as we speak. Really needed some profiling so was trying to
get the oprofile stuff to work in unstable. From what I saw from
his last post, Jose is working on it.

A few people have kindly agreed to do some testing for me too.

We'll throw some stuff out on the list after the weekend.  I am
tracking all the reports of poor networking performance so if
people do see some issues please continue to post on the list..

thanks,
Nivedita

RE: what's maximum speed setting for xen network?

[Santos, Jose Renato G (Jose Renato Santos)]

  Nivedita,

  Yes, I am working to get oprofile in xen-unstable.
  I was only able to spend a few  hours on this
  My own port to xen-unstable shows the same behavior as 
  the one done by Andrew Theurer.
  I will start debugging xenoprof this week to nail down 
  the problem. 
  Renato

> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com 
> [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of 
> Nivedita Singhvi
> Sent: Saturday, April 23, 2005 10:37 AM
> To: Keir Fraser
> Cc: xen-devel@lists.xensource.com; zhaoxin@eecs.umich.edu; 
> mukesh agrawal
> Subject: Re: [Xen-devel] what's maximum speed setting for xen network?
> 
> 
> Keir Fraser wrote:
> > 
> > On 22 Apr 2005, at 20:53, mukesh agrawal wrote:
> > 
> >> I've got a configuration with DomU talking to Dom0 over a vif, and
> >> seem to be seeing UDP losses, even at low data rates. (I 
> haven't dug 
> >> into this deeply to very the problem yet, though.) So I'd be 
> >> interested if you're seeing something similar.
> > 
> > 
> > We have no back-pressure / flow control on inter-domain networking. 
> > This
> > leads to embarrassing performance for UDP when the sender 
> and receiver 
> > share a CPU. The sender fills the receiver's buffers but 
> continues to 
> > send even though he drops further packets (no buffer space 
> at receiver) 
> > -- then the receiver gets a time slice, receives a ring's worth of 
> > packets -- then the sender gets to run for another time 
> slice, most of 
> > which again is wasted -- and so on.
> > 
> > We need a sensible mechanism for ensuring batched packet 
> delivery, but
> > also to ensure that comms rings do not so easily get 
> overflowed. This 
> > would vastly improve the inter-domain performance, far more than 
> > striving for zero copy or other such tweaks.
> > 
> > It would be great if someone else is interested in 
> investigating this. 
> > :-)
> 
> Hi Keir,
> 
> Yep, I'm one of the people at least looking into this, don't 
> know if others are too :). Thrown some instrumentation in and 
> doing some testing as we speak. Really needed some profiling 
> so was trying to get the oprofile stuff to work in unstable. 
> From what I saw from his last post, Jose is working on it.
> 
> A few people have kindly agreed to do some testing for me too.
> 
> We'll throw some stuff out on the list after the weekend.  I 
> am tracking all the reports of poor networking performance so 
> if people do see some issues please continue to post on the list..
> 
> thanks,
> Nivedita
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
>