Re: RFC: mac802154 Packet Queueing and Slave Devices

[Alan Ott <alan@signal11.us>]

On 09/09/2012 08:43 PM, Alan Ott wrote:
> Tony and I were recently talking about packet queueing on 802.15.4. What
> currently happens (in net/mac802154/tx.c) is that each tx packet (skb)
> is stuck on a work queue, and the worker function then sends each packet
> to the hardware driver in order.
> 
> The problem with this is that it defeats the netif flow control. The
> networking layer thinks the packet is sent as soon as it's put on the
> workqueue (because the function that queues it returns NETDEV_TX_OK to
> the networking layer), and the workqueue can then get arbitrarily large
> if an application tries to send a lot of data. (Tony has shown this with
> iperf)
> 

So I tried fixing this using netif_stop_queue() and netif_wake_queue(),
the standard way. The flow control works, but I'm now losing packets.

It happens like this:

ipv6           -> 6lowpan   -> net core -> mac802154         -> hardware
 single packet     fragment                 netif_stop_queue()
                   fragment
                   fragment
                   fragment

  Above: a single ipv6 packet is split into fragments by 6lowpan. Each
  fragment is sent through the networking core where it ends up in
  mac802154, which will call netif_stop_queue() and netif_wake_queue()
  for flow control as packets are sent.


The problem is that since many ieee802154 hardware devices can only hold
one packet at a time in their tx buffer, netif_stop_queue() gets called
after the first fragment. Since the 6lowpan code is trying to, in the
above case, send 4 packets, the remaining 3 will get dropped when
they're handed to the networking core (dev_queue_xmit()) when the queue
is stopped.

So as a solution, one could envision 6lowpan putting the fragments into
a queue, and submitting one at a time, as the queue gets woken. The
problem with this is that there's no way to get notification for when a
queue is woken. I checked both ppp and ax25 (which would seem to have
this same issue), and they both have a fragment queue, but they rely on
external events (mostly unrelated to the queue being woken) to trigger
sending packets from the queue (see calls to ax25_kick()). That seems
hacky at best.

A thread from pppoe[1] talks about what's a similar issue. The patch
from that email was never merged. Even so, their solution seems a bit
hacky too (because it would basically cause a kick to (in this case)
6lowpan, whenever an skb gets destroyed (ie: after it's sent). With the
desire for 6lowpan to be a generic protocol[2], one would want it to be
efficient on MAC layers which do support longer queues[3].

What am I missing here? What's the right way to do this?

Alan.

[1] http://thread.gmane.org/gmane.linux.network/233089
[2] There has been some discussion about using 6lowpan on Bluetooth
low-energy.
[3] There's also the case where 2 6lowpan instances are on attached to
the same hardware, or where 6lowpan and raw are being used concurrently.