[uml-devel] tun/tap network throughput problem

[uml-devel] tun/tap network throughput problem

[Young Koh]

Hi,

i was running an expriment to measure network throughput of UML.
physical machine A and B are connected with gigabit network. and UML
runs on machine A. i'm sending bulk UDP packets from UML to machine B
using ttcp network throughput benchmark program.
i'm using 2.4.26 for both UML and host. (skas + sysemu patched)

i modified UML kernel code to get high performance a little bit, and i
got some gain with TCP and UDP throughput. but the problem arises when
i send large UDP packets with high sending rate. when i send 16K-sized
UDP packets with rate of 500Mbits/s, receiving rate at machine B was
about the same (~500Mbits/s), i mean, no problem. but when i send
16K-sized UDP packets with higher rate of more than 700Mbits/s, the
receiving rate drops to 150Mbits/s. (MTU = 1500)

i'm using tun/tap + bridge. so, i checked tap0 and eth0 device status,
but no packet drop reported (using ifconfig). i also checked the
number of packets received and sent. tap0 at host side received around
60007 packets, and eth0 at machine A sends 505536 packets during that
experiment. eth0 at machine B receives 505533 packets. so, it looks
like between tap0 and eth0, packets are being dropped somehow. (the
reason of dropping rapidly from 700 to 150 is the entire UDP packet
will be lost even if one of the fragmented packets is lost, i think)

i used tcpdump to check any trouble with UDP checksum because i saw
some posting about bug of bad UDP checksum when a UDP packet is
fragmented, but no checksum failure reported. (and it works well with
the lower sending rate)

i found other postings discussing queues of tun/tap devices when
packets are coming to UML kernel and between two UML machines. but i
couldn't find posting about packets from UML to other machine, even
though they are all related, it looks.

anybody has any idea where packets are being dropped and how i can solve it?

thank you,


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

[uml-devel] Re: tun/tap network throughput problem

[Nuutti Kotivuori]

Young Koh wrote:
> i found other postings discussing queues of tun/tap devices when
> packets are coming to UML kernel and between two UML machines. but i
> couldn't find posting about packets from UML to other machine, even
> though they are all related, it looks.
>
> anybody has any idea where packets are being dropped and how i can
> solve it?

Remember to check the packet drop readings from inside the UML kernel
as well as on all interfaces on the host side.

Since you are sending large packets, I would suggest making sure that
all involved interfaces have large enough txqueues.

You can monitor queue buildup by:

  watch -n1 tc -d -s qdisc show

There should be no places for silent drops in the packet handling
things - go around and try to find where the packets are dropping (or
being queued forever).

-- Naked



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] Re: tun/tap network throughput problem

[Young Koh]

Hi,
thank you for your reply.

> Remember to check the packet drop readings from inside the UML kernel
> as well as on all interfaces on the host side.

i checked the packet drop reading from UML kernel shell, but it still
shows no packet dropping.

>   watch -n1 tc -d -s qdisc show

so, i ran tc in the host machine as you suggested, and it reported
some packets are being dropped in eth0. (even though the stat from
/sbin/ifconfig still didn't report any packet drops)

$watch -n1 tc -d -s qdisc show
qdisc pfifo_fast 0: dev eth0 [Unknown qdisc, optlen=20]
 Sent 18300080724 bytes 13121666 pkts (dropped 1328688, overlimits 0)
 
 qdisc pfifo_fast 0: dev tap0 [Unknown qdisc, optlen=20]
 Sent 7664312 bytes 17650 pkts (dropped 0, overlimits 0)
---

i saw around 50,000 packet droppings out of around 240,000 packets.
then, i tried larger txqueues in eth0. as i configured with the larger
txqueue, the less packet droppings i got.

so, when i finally i assigned 50,000 for txqueuelen in the eth0. there
was no packet dropping. and the final rates are, sending rate =
~800Mbits, and receiving rate(in another machine) = ~650Mbits

it seems to me that UML was sending too fast for eth0 to handle, so
there were packet droppings because of the rate difference between
tap0 and eth0. hrmm.. is there any flow control between tap0 and eth0?
(i guess not, from the result, though)

even in host Linux kernel, how is flow control managed between native
Linux and eth0? i mean, what happens if (host) Linux kernel sends too
many packets(or socket buffers) to eth0 device?

thank you very much,

-Young
> 
> Since you are sending large packets, I would suggest making sure that
> all involved interfaces have large enough txqueues.
> 
> You can monitor queue buildup by:
> 
>   watch -n1 tc -d -s qdisc show
> 
> There should be no places for silent drops in the packet handling
> things - go around and try to find where the packets are dropping (or
> being queued forever).
> 
> -- Naked
> 
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real users.
> Discover which products truly live up to the hype. Start reading now.
> http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
> _______________________________________________
> User-mode-linux-devel mailing list
> User-mode-linux-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel
>


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] Re: tun/tap network throughput problem

[Nuutti Kotivuori]

Young Koh wrote:
> it seems to me that UML was sending too fast for eth0 to handle, so
> there were packet droppings because of the rate difference between
> tap0 and eth0. hrmm.. is there any flow control between tap0 and
> eth0?  (i guess not, from the result, though)

Yes, that seems to be the correct assumption - eth0 is dropping the
packets. However, there is no 'rate difference' between tap0 and
eth0. Packets are queued from inside the UML kernel, to the internal
eth0 interface there (there's a packet queue there). Then they pop up
from tap0 on the host side and are synchronously queued to eth0
queue. From there on, they are queued onwards by eth0 hard_start_xmit,
which gets triggered from the interrupts. So there are two queues at
work here, the inside the UML kernel eth0 queue, and the host kernel
eth0 queue - of which the queue inside the UML kernel is faster and
apparently never drops packets.

And you are correct - there's nothing to limit the speed of packets
queued from inside the UML kernel to the speed that host side eth0
might accept them.

> even in host Linux kernel, how is flow control managed between
> native Linux and eth0? i mean, what happens if (host) Linux kernel
> sends too many packets(or socket buffers) to eth0 device?

All sent packets are added to eth0 queue. Eth0 queue is depleted by
hard_start_xmits triggered by interrupts (or other things, depending
on the driver). If too many packets are enqueued to eth0 queue, the
packets are being dropped.

If Linux tc is involved in limiting the queue speed at eth0, it merely
throttles the speed of packets received by the network card - all
packets are still queued at the same speed to eth0 queue.

In general, there's nothing to limit how many packets get queued to
the interface queue - if something sends a million packets, that is
how many packets are queued to the interface. However, for TCP, there
are two things at work. First of all is ofcourse the fact that TCP
will automatically throttle itself to the network speed. This one
works with UML as well. Second issue is that the kernel will throttle
TCP queues if they start taking too much memory, even before any
network device queues are filled. This will only work if the TCP
connection originates from the host machine - eg. not with UML.

With UDP, the userland program has no way to throttle itself or know
how fast the network can send UDP packets. In there, saying sendto for
the UDP packets, might return ENOBUFS, but from the manpage of
sendto():

  ENOBUFS
         The output queue for a network interface was full.  This  gener-
         ally  indicates  that the interface has stopped sending, but may
         be caused by transient congestion.   (Normally,  this  does  not
         occur  in Linux. Packets are just silently dropped when a device
         queue overflows.)

So, UDP send performance measures with packet drop are not really
meaningful, since it all depends on how much time the userland process
gets time and if that is more than what the network device can dish out.

-- Naked


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] Re: tun/tap network throughput problem

[Young Koh]

Hi,

> In general, there's nothing to limit how many packets get queued to
> the interface queue - if something sends a million packets, that is
> how many packets are queued to the interface. However, for TCP, there

interesting. then, the UDP sending rate should be solely determined by
system performance regardless of what kind of the network interface
used, cause the rate will be determined by how fast the application
enqueues packets to the device queue, which is an internal data
structure in the kernel, right? when i measured UDP sending rate in
the host machine, it was about 650Mbits/s. that means the throughput
of data transfer from an application to the kernel internal structure
is only 650Mbits/s? but, when i measured UDP sending rate with
loopback device (sending packets to 127.0.0.1), it was about
5,000Mbits/s, which is a lot faster. should they, i mean, the sending
rates for eth0 and lo, be the same? (maybe lo uses special queuing
mechanism?)

thank you very much!

-Young


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] Re: tun/tap network throughput problem

[Nuutti Kotivuori]

Young Koh wrote:
>> In general, there's nothing to limit how many packets get queued to
>> the interface queue - if something sends a million packets, that is
>> how many packets are queued to the interface. However, for TCP,
>> there
>
> interesting. then, the UDP sending rate should be solely determined
> by system performance regardless of what kind of the network
> interface used, cause the rate will be determined by how fast the
> application enqueues packets to the device queue, which is an
> internal data structure in the kernel, right? when i measured UDP
> sending rate in the host machine, it was about 650Mbits/s. that
> means the throughput of data transfer from an application to the
> kernel internal structure is only 650Mbits/s? but, when i measured
> UDP sending rate with loopback device (sending packets to
> 127.0.0.1), it was about 5,000Mbits/s, which is a lot faster. should
> they, i mean, the sending rates for eth0 and lo, be the same? (maybe
> lo uses special queuing mechanism?)

Almost, but not quite. When packets are queued to the lo interface,
they are handled immediately, synchronously even. When packets are
queued to the eth0 interface, they get queued and the hardware
interrupts trigger their processing. This means that the userland
process is constantly being interrupted by the hardware sending
packets, where as in the lo case it can just fire away as it wants.

However, the speed difference seems too large to be simply caused by
that - so I am wondering a bit about the test tool, if it might manage
to do something special. Perhaps there is a way to see if the queue is
full from an UDP socket somehow, perhaps select does not return write
availability or something. Or perhaps it simply asks how much data is
in buffers - but I thought that only worked for receiving and TCP. I
would need to look into this a bit further.

-- Naked


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] Re: tun/tap network throughput problem

[Nuutti Kotivuori]

Nuutti Kotivuori wrote:
> Young Koh wrote:
>> when i measured UDP sending rate in the host machine, it was about
>> 650Mbits/s. that means the throughput of data transfer from an
>> application to the kernel internal structure is only 650Mbits/s? 
>> but, when i measured UDP sending rate with loopback device (sending
>> packets to 127.0.0.1), it was about 5,000Mbits/s, which is a lot
>> faster. should they, i mean, the sending rates for eth0 and lo, be
>> the same? (maybe lo uses special queuing mechanism?)

[...]

> However, the speed difference seems too large to be simply caused by
> that - so I am wondering a bit about the test tool, if it might
> manage to do something special. Perhaps there is a way to see if the
> queue is full from an UDP socket somehow, perhaps select does not
> return write availability or something. Or perhaps it simply asks
> how much data is in buffers - but I thought that only worked for
> receiving and TCP. I would need to look into this a bit further.

I just made a trivial test program in ocaml:

,----[ ocamlopt -o udptest unix.cmxa udptest.ml ]
| let data = String.make 15000 ' ';;
| try
|   let dst = Unix.inet_addr_of_string Sys.argv.(1) in
|   let port = int_of_string Sys.argv.(2) in
|   let addr = Unix.ADDR_INET(dst, port) in
|   let sock = Unix.socket Unix.PF_INET Unix.SOCK_DGRAM 0 in
|   let _ = Unix.connect sock addr in
|   let total = ref 0 in
|   let starttime = Unix.gettimeofday () in
|     for i = 1 to 10000 do
|       let ret = Unix.send sock data 0 (String.length data) [] in
|         total := !total + ret
|     done;
|     let endtime = Unix.gettimeofday () in
|       print_endline ("Time: " ^ (string_of_float (endtime -. starttime))
|                      ^ " Total: " ^ (string_of_int !total))
| with Unix.Unix_error(e, f, a) ->
|   print_endline ("Error: " ^ (Unix.error_message e))
`----

Then ran it against localhost and an external host (with packets being
dropped at input, so we do not get connection refused back):

,----
| $ ./a.out 127.0.0.1 1024
| Time: 0.352205991745 Total: 150000000
| 
| $ ./a.out 172.20.0.1 1024
| Time: 0.478994131088 Total: 150000000
`----

So in the localhost case, the throughput is over 3 gigabits per
second, and in the case of the remote host, it is over 2 gigabits per
second. This seems more in line of what it should be. The network here
is a mere 100Mbit network - so the dropped counters in eth0 (seen with
tc) fly up during the test.

So, it is probable that your test software either manages to throttle
itself in case of network congestion, or simply only counts packets
that are not dropped, somehow.

-- Naked



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] tun/tap network throughput problem

[Blaisorblade]

On Monday 07 March 2005 21:56, Young Koh wrote:
> Hi,
>
> i was running an expriment to measure network throughput of UML.
> physical machine A and B are connected with gigabit network. and UML
> runs on machine A. i'm sending bulk UDP packets from UML to machine B
> using ttcp network throughput benchmark program.
> i'm using 2.4.26 for both UML and host. (skas + sysemu patched)
>
> i modified UML kernel code to get high performance a little bit, and i
> got some gain with TCP and UDP throughput.
Hmm, can you share your changes with us, please? We'd be very grateful...
-- 
Paolo Giarrusso, aka Blaisorblade
Linux registered user n. 292729
http://www.user-mode-linux.org/~blaisorblade




-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] tun/tap network throughput problem

[Young Koh]

sure, allow me to check if everything looks fine (at least) to me...


On Wed, 9 Mar 2005 20:29:29 +0100, Blaisorblade <blaisorblade@yahoo.it> wrote:
> On Monday 07 March 2005 21:56, Young Koh wrote:
> > Hi,
> >
> > i was running an expriment to measure network throughput of UML.
> > physical machine A and B are connected with gigabit network. and UML
> > runs on machine A. i'm sending bulk UDP packets from UML to machine B
> > using ttcp network throughput benchmark program.
> > i'm using 2.4.26 for both UML and host. (skas + sysemu patched)
> >
> > i modified UML kernel code to get high performance a little bit, and i
> > got some gain with TCP and UDP throughput.
> Hmm, can you share your changes with us, please? We'd be very grateful...
> --
> Paolo Giarrusso, aka Blaisorblade
> Linux registered user n. 292729
> http://www.user-mode-linux.org/~blaisorblade
> 
>


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] Re: tun/tap network throughput problem

[Rob Landley]

On Tuesday 08 March 2005 12:46 pm, Nuutti Kotivuori wrote:

> With UDP, the userland program has no way to throttle itself or know
> how fast the network can send UDP packets. In there, saying sendto for
> the UDP packets, might return ENOBUFS, but from the manpage of
> sendto():
>
>   ENOBUFS
>          The output queue for a network interface was full.  This  gener-
>          ally  indicates  that the interface has stopped sending, but may
>          be caused by transient congestion.   (Normally,  this  does  not
>          occur  in Linux. Packets are just silently dropped when a device
>          queue overflows.)
>
> So, UDP send performance measures with packet drop are not really
> meaningful, since it all depends on how much time the userland process
> gets time and if that is more than what the network device can dish out.
>
> -- Naked

The logical thing to return for a temporary nonblocking congestion condition 
is -EAGAIN to indicate the packet was dropped, but retransmitting it should 
work.  (Client programs are free to ignore this, and if broken clients didn't 
ignore it and it screwed up existing implementations, possibly an extra open 
flag would be necessary to trigger it.)  Putting this in UML itself wouldn't 
help, but if the host kernel did this UML (and TCP in general) could throttle 
more sanely.

However, nobody's ever bothered to fix this because UDP is not a guaranteed 
protocol over the internet or hub-based ethernet.  Now that we've got 
switched ethernet (synchronous in the case of gigabit), packets are virtually 
never dropped over a LAN and it's possible that some infrastructure to not 
drop packets without remarking on it in the kernel's own packet handling 
layers is now worth the bother...

But I'm unlikely to personally code it up any time soon. :)

Rob


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel