Re: [PATCH v9] tilegx network driver: initial support

[Chris Metcalf <cmetcalf@tilera.com>]

On 6/6/2012 1:40 PM, Eric Dumazet wrote:
> On Mon, 2012-06-04 at 16:12 -0400, Chris Metcalf wrote:
>
>> +/* Allocate and push a buffer. */
>> +static bool tile_net_provide_buffer(bool small)
>> +{
>> +	int stack = small ? small_buffer_stack : large_buffer_stack;
>> +	const unsigned long buffer_alignment = 128;
>> +	struct sk_buff *skb;
>> +	int len;
>> +
>> +	len = sizeof(struct sk_buff **) + buffer_alignment;
>> +	len += (small ? 128 : 1664);
> 1664 is a magic number, it should be a nice define
>
> #define ..... ( ETH_DATA_LEN + .... )

Fair enough.  However, the magic-ness comes from the hardware header code
in arch/tile/gxio/mpipe.h, which provides a limited set of allowed buffer
sizes, including 1664.  But I can add these #defines at the top of this driver:

/* Buffer sizes and mpipe enum codes for buffer stacks.
 * See arch/tile/include/gxio/mpipe.h for the set of possible values.
 */
#define BUFFER_SIZE_SMALL_ENUM GXIO_MPIPE_BUFFER_SIZE_128
#define BUFFER_SIZE_SMALL 128
#define BUFFER_SIZE_LARGE_ENUM GXIO_MPIPE_BUFFER_SIZE_1664
#define BUFFER_SIZE_LARGE 1664


>> +	skb = dev_alloc_skb(len);
>> +	if (skb == NULL)
>> +		return false;
>> +
>> +	/* Make room for a back-pointer to 'skb' and guarantee alignment. */
>> +	skb_reserve(skb, sizeof(struct sk_buff **));
>> +	skb_reserve(skb, -(long)skb->data & (buffer_alignment - 1));
>> +
>> +	/* Save a back-pointer to 'skb'. */
>> +	*(struct sk_buff **)(skb->data - sizeof(struct sk_buff **)) = skb;
>> +
>> +	/* Make sure "skb" and the back-pointer have been flushed. */
>> +	wmb();
> Interesting, have you considered using build_skb() instead of this
> convoluted thing ?
>
> This could save some cache misses...

I hadn't looked at build_skb() before; we built up this driver mostly on a
base of 2.6.38, where it doesn't exist.  That said, it doesn't seem like it
matters; dev_alloc_skb() will just end up calling down to build_skb()
anyway, as far as I can tell.

The code where we do the two skb_reserves and then stuff in a backpointer
and do a barrier are because we track the skbuffs in hardware, and hardware
ignores the low 7 bits aof the address (thus the "buffer_alignment" part)
and we need to be able to pull the actual skb address out of the data when
the hardware returns a pointer to the data to us.

By the way, your question about tx_queue_len is a good one; I'm roping in
our other network developer folks to figure it out.  Originally it was a
performance optimization, I believe; I'm not sure it's still required. 
I'll follow up on that one when we've tracked it down.

-- 
Chris Metcalf, Tilera Corp.
http://www.tilera.com