[uml-devel] UML tuntap driver problem with queueing

[uml-devel] UML tuntap driver problem with queueing

[Nuutti Kotivuori]

[-- Attachment #1: Type: text/plain, Size: 2924 bytes --]

I will first briefly outline here how tun/tap device queueing works.

Each tap device has two ends - one is the network device lookalike
'tap0' and the other is the fd received by opening '/dev/net/tun' and
running some ioctls. With UML, the host kernel has the 'tap0' device
and the UML process opens up '/dev/net/tun' side, showing it as 'eth0'
inside the UML kernel. Right now, we are only interested in how
packets travel from the host to the guest, eg. packets sent to the
'tap0' device travelling to the UML kernel.

There are two queues involved in this process. One is the normal
network device send queue attached to the 'tap0'. It can be
manipulated by the QoS tools, like 'tc'. I will call this
'txqueue'. The other is an internal queue of the tun/tap driver, an
skb queue, in the tun_struct as readq. I will call this simply
'readq'.

There are two alternatives on how packets are queued, controlled by
IFF_ONE_QUEUE flag when creating the tap device. The default is that
IFF_ONE_QUEUE is off. In that case, packets are first queued to the
'readq'. When 'readq' grows to 10 packets, netif_stop_queue is called
on the device, which causes the 'txqueue' to start accumulating the
following packets. When packets are read from the fd side, the queue
is started again, which starts filling 'readq' again. So, this
alternative should keep the queue mostly on the device side and allow
normal QoS routines to handle packet dropping and such.

The other alternative, when IFF_ONE_QUEUE is set, uses just a single
queue. Packets are queued to 'readq' always, the net device queue is
never stopped. If 'readq' grows to the interface 'txqueuelen', packets
are simply tail dropped. This means that the normal QoS tools cannot
affect how packets are dropped and the packet queue is 'hidden' inside
the kernel as there is no simple way to see it.

Now, here comes the actual bug report.

If the UML kernel is sent more packets than it can handle, and
IFF_ONE_QUEUE is not set (as it isn't by default), packets first fill
the 'readq' and then start amassing at 'txqueue', as they should. But
when no more packets are being sent, the queue does not start growing
smaller. If no packets are sent to the device, the queue stays there
indefinitely. When single packets are sent, the queue always decreases
by 10 packets (so 11 packets in total arrive on the guest side).

If I understood correctly from the code, UML uses SIGIO to trigger
packet reading. So, either SIGIO is not delivered properly in the case
of the two queues - or UML's SIGIO handling is broken.

If IFF_ONE_QUEUE is set, everything works fine, packets are delivered
in a timely manner and queue is never stalled.

As a workaround to the problem, I've added the IFF_ONE_QUEUE option to
all the places that open a tap device (namely tunctl, uml_net and
uml_router). The patch is attached. But the actual problem should most
likely be fixed once found.

-- Naked

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: tuntap_one_queue.diff --]
[-- Type: text/x-patch, Size: 1516 bytes --]

Index: uml-utilities-20040406/uml_net/tuntap.c
===================================================================
--- uml-utilities-20040406.orig/uml_net/tuntap.c
+++ uml-utilities-20040406/uml_net/tuntap.c
@@ -44,7 +44,7 @@
     return(-1);
   }
   memset(ifr, 0, sizeof(*ifr));
-  ifr->ifr_flags = IFF_TAP | IFF_NO_PI;
+  ifr->ifr_flags = IFF_TAP | IFF_NO_PI | IFF_ONE_QUEUE;
   ifr->ifr_name[0] = '\0';
   if(ioctl(tap_fd, TUNSETIFF, (void *) ifr) < 0){
     output_errno(output, "TUNSETIFF : ");
Index: uml-utilities-20040406/tunctl/tunctl.c
===================================================================
--- uml-utilities-20040406.orig/tunctl/tunctl.c
+++ uml-utilities-20040406/tunctl/tunctl.c
@@ -81,7 +81,7 @@
 
   memset(&ifr, 0, sizeof(ifr));
 
-  ifr.ifr_flags = IFF_TAP | IFF_NO_PI;
+  ifr.ifr_flags = IFF_TAP | IFF_NO_PI | IFF_ONE_QUEUE;
   strncpy(ifr.ifr_name, tun, sizeof(ifr.ifr_name) - 1);
   if(ioctl(tap_fd, TUNSETIFF, (void *) &ifr) < 0){
     perror("TUNSETIFF");
Index: uml-utilities-20040406/uml_router/tuntap.c
===================================================================
--- uml-utilities-20040406.orig/uml_router/tuntap.c
+++ uml-utilities-20040406/uml_router/tuntap.c
@@ -28,7 +28,7 @@
     return(-1);
   }
   memset(&ifr, 0, sizeof(ifr));
-  ifr.ifr_flags = IFF_TAP | IFF_NO_PI;
+  ifr.ifr_flags = IFF_TAP | IFF_NO_PI | IFF_ONE_QUEUE;
   strncpy(ifr.ifr_name, dev, sizeof(ifr.ifr_name) - 1);
   if(ioctl(fd, TUNSETIFF, (void *) &ifr) < 0){
     perror("TUNSETIFF failed");

[uml-devel] Re: UML tuntap driver problem with queueing

[Nuutti Kotivuori]

Nuutti Kotivuori wrote:
> As a workaround to the problem, I've added the IFF_ONE_QUEUE option
> to all the places that open a tap device (namely tunctl, uml_net and
> uml_router). The patch is attached. But the actual problem should
> most likely be fixed once found.

Just a heads up here to the UML folks in case it was missed - with the
current code, packet delivery to UML machines using tuntap driver
*will* mess up (stall totally or major bursts of lag) if more than 10
packets are queued to UML at a time.

This will happen if Linux reassembles packets (connection tracking is
enabled) and a packet of around 17000 bytes or more arrives, or if UML
is given packets faster than it can handle them (not very likely to
happen if the packets are received from a 100Mbit network, and not
generated locally, unless there is heavy load on the machine).

So this is a real problem, for which the patch given is a decent
workaround, as it works perfectly and only changes things if one
employs QoS on the tap device itself.

Ofcourse, the actual cause for this should be fixed (SIGIO mess up or
whatever it is).

-- Naked



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] Re: UML tuntap driver problem with queueing

[Jeff Dike]

naked@iki.fi said:
> ust a heads up here to the UML folks in case it was missed - with the
> current code, packet delivery to UML machines using tuntap driver
> *will* mess up (stall totally or major bursts of lag) if more than 10
> packets are queued to UML at a time. 

It wasn't missed.  Someone else posted 
(http://marc.theaimsgroup.com/?l=user-mode-linux-user&m=110752781409419&w=2)
that networking freezes until a character is typed on the console.

This is a clear sign of the SIGIO missing some traffic and letting it sit
on the host until some other IO happens.

> Ofcourse, the actual cause for this should be fixed (SIGIO mess up or
> whatever it is). 

Yup.  It does look like there's some sort of race in the SIGIO handler.

				Jeff



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel