sunvnet and ->xmit_more

sunvnet and ->xmit_more

[David Miller]

David and others working on sunvnet, I just wanted to point out that
in the net-next tree there is a new facility that can improve
performance quite a bit in sunvnet.

Basically in the ->ndo_start_xmit() handler, if you see skb->xmit_more
set then the stack is telling you that it guarentees that another
packet will be given to you immediately when ->ndo_start_xmit()
returns.

This means that, unless you have filled up your TX queue, you can
defer the TX indication to the device.

For example, in the virtio_net driver the test is:

	if (__netif_subqueue_stopped(dev, qnum) || !skb->xmit_more)
		virtqueue_kick(sq->vq);

The pktgen module also has a new "burst" parameter you can use to test
out this facility directly, and the qdisc layer has heuristics for
dequeueing multiple packets at a time for normal traffic.

Just FYI...

Re: sunvnet and ->xmit_more

[Sowmini Varadhan]

On (10/07/14 15:18), David Miller wrote:
> 
> David and others working on sunvnet, I just wanted to point out that
> in the net-next tree there is a new facility that can improve
> performance quite a bit in sunvnet.
> 
> Basically in the ->ndo_start_xmit() handler, if you see skb->xmit_more
> set then the stack is telling you that it guarentees that another
> packet will be given to you immediately when ->ndo_start_xmit()
> returns.
> 
> This means that, unless you have filled up your TX queue, you can
> defer the TX indication to the device.

I'm not sure how this can be useful to sunvnet- in sunvnet's case
we send the TX indication at the *start* of a burst, so if xmit_more
was set, sure- we can send out another packet immediately, and
avoid another START message (which we already do today), but 
nothing else to be gained from xmit_more?

BTW, I have most of the NAPI done, getting it stress-tested etc
(the recent jumbo commit added a few more races between vnet_port_remove
and vnet_start_xmit, thanks to the extra clean_timer) but I figure
I might as well fully test this internally since net-next is closed
for the moment anyway?

> 
> For example, in the virtio_net driver the test is:
> 
> 	if (__netif_subqueue_stopped(dev, qnum) || !skb->xmit_more)
> 		virtqueue_kick(sq->vq);
> 
> The pktgen module also has a new "burst" parameter you can use to test
> out this facility directly, and the qdisc layer has heuristics for
> dequeueing multiple packets at a time for normal traffic.
> 
> Just FYI...
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: sunvnet and ->xmit_more

[David Miller]

From: Sowmini Varadhan <sowmini.varadhan@oracle.com>
Date: Tue, 7 Oct 2014 15:29:22 -0400

> I'm not sure how this can be useful to sunvnet- in sunvnet's case
> we send the TX indication at the *start* of a burst, so if xmit_more
> was set, sure- we can send out another packet immediately, and
> avoid another START message (which we already do today), but 
> nothing else to be gained from xmit_more?

If you defer that __vnet_tx_trigger() call through all the ->xmit_more
SKBs, then you are less likely to see the DRING_STOPPED event from the
peer which will make you have to send a START again.

So, for an xmit_more burst of 3, instead of:

	->ndo_start_xmit()
		__vnet_tx_trigger()
	->ndo_start_xmit()
IRQ -> vnet_ack() -> STOPPED
	->ndo_start_xmit()
		__vnet_tx_trigger()

You would do something like:

	->ndo_start_xmit()
	->ndo_start_xmit()
	->ndo_start_xmit()
		__vnet_tx_trigger()

> BTW, I have most of the NAPI done, getting it stress-tested etc
> (the recent jumbo commit added a few more races between vnet_port_remove
> and vnet_start_xmit, thanks to the extra clean_timer) but I figure
> I might as well fully test this internally since net-next is closed
> for the moment anyway?

Yeah no rush.

Re: sunvnet and ->xmit_more

[Raghuram Kothakota]

On Oct 7, 2014, at 12:38 PM, David Miller <davem@davemloft.net> wrote:

> From: Sowmini Varadhan <sowmini.varadhan@oracle.com>
> Date: Tue, 7 Oct 2014 15:29:22 -0400
> 
>> I'm not sure how this can be useful to sunvnet- in sunvnet's case
>> we send the TX indication at the *start* of a burst, so if xmit_more
>> was set, sure- we can send out another packet immediately, and
>> avoid another START message (which we already do today), but 
>> nothing else to be gained from xmit_more?
> 
> If you defer that __vnet_tx_trigger() call through all the ->xmit_more
> SKBs, then you are less likely to see the DRING_STOPPED event from the
> peer which will make you have to send a START again.
> 
> So, for an xmit_more burst of 3, instead of:
> 
> 	->ndo_start_xmit()
> 		__vnet_tx_trigger()
> 	->ndo_start_xmit()
> IRQ -> vnet_ack() -> STOPPED
> 	->ndo_start_xmit()
> 		__vnet_tx_trigger()
> 
> You would do something like:
> 
> 	->ndo_start_xmit()
> 	->ndo_start_xmit()
> 	->ndo_start_xmit()
> 		__vnet_tx_trigger()

In case of sunvnet,  consumer starting as soon as possible would help
as the consumer is slower as it copies the data to local buffers. The recent
change by DLS avoids the copy in the Tx side, so the producer could be
faster than the receiver.  If the trigger is sent after n packets, then consumer
will really start after n packets in the ring and it may have lost that much time
to pickup the packets.

-Raghuram

> 
>> BTW, I have most of the NAPI done, getting it stress-tested etc
>> (the recent jumbo commit added a few more races between vnet_port_remove
>> and vnet_start_xmit, thanks to the extra clean_timer) but I figure
>> I might as well fully test this internally since net-next is closed
>> for the moment anyway?
> 
> Yeah no rush.

Re: sunvnet and ->xmit_more

[David Miller]

From: Raghuram Kothakota <Raghuram.Kothakota@oracle.com>
Date: Tue, 7 Oct 2014 13:38:07 -0700

> In case of sunvnet,  consumer starting as soon as possible would help
> as the consumer is slower as it copies the data to local buffers. The recent
> change by DLS avoids the copy in the Tx side, so the producer could be
> faster than the receiver.  If the trigger is sent after n packets, then consumer
> will really start after n packets in the ring and it may have lost that much time
> to pickup the packets.

I agree for the first few TX queue entires, or even just the first, but after
that it always pays to defer and decrease the number of hcalls, locks, etc.

Play with it, you'll see.

Re: sunvnet and ->xmit_more

[Sowmini Varadhan]

On (10/07/14 16:46), David Miller wrote:
> 
> Play with it, you'll see.

at the moment, there are enough fun and games just doing a 
"modload -r sunvnet; modload sunvnet" in a loop while running iperf.

I think running stably is as important as tput, no? :-) Lets fix
all the races first.

--Sowmini

Re: sunvnet and ->xmit_more

[Sowmini Varadhan]

One thought occurred to me about a way to use this..

It's true that in the case of sunvnet, we send the tx-trigger
at the *start* of a burst, thus we would do the same amount of work
whether we had 1 or N (>1) packets for a burst, until a STOPPED was
received, 

and if we tried playing games with this, for every sequence where
the game-playing around "when should I send START" helps, there would
always be a counter-example where it would hurt..

but there *is* one way in which it could be used- if the producer
could somehow signal the "xmit_more" info to the consumer, then
the consumer could factor that information in, when trying to
decide whether or not to declare STOPPED (recall the discussion 
around the fudge-factor that we had earlier in
http://www.spinics.net/lists/netdev/msg294549.html -  the skb_more
provides a better hint than merely guessing a udelay)

Raghuram just pointed out to me

" But this can only be used as a hint, as we cannot rely on the peer to
  send the next pack in specific time frame, the peer may be stuck in something
  else or switched out due to scheduling or some bug etc.

  To pass this hint, ..  we could use a bit in a descriptor in TxDring too."

That might be worth investigating (but I might not get to it till
I get the NAPI on its way).

--Sowmini

Re: sunvnet and ->xmit_more

[David L Stevens]

On 10/07/2014 03:18 PM, David Miller wrote:
> 
> David and others working on sunvnet, I just wanted to point out that
> in the net-next tree there is a new facility that can improve
> performance quite a bit in sunvnet.

Yes, that looks interesting. Thanks!

				+-DLS