inter-packet gap in pktgen

inter-packet gap in pktgen

[Lennert Buytenhek]

Hi Robert,

For TX/RX tests, I've been trying to convince pktgen to spit out exactly
N packets per second (for various values of N) by tweaking the inter-packet
gap parameter, and I've noticed the following.

The 'ipg' parameter to pktgen seems to be neither (i) the actual
on-the-wire inter-packet gap (which is the time between the FCS of one
packet and the preamble of the next), nor (ii) the time between the first
data bit of one packet and that of the next.

>From observing that the pps rate for a given 'ipg' does not depend on the
packet size, it would seem that 'ipg' attempts to be (ii).  But it doesn't
quite succeed in that -- it appears to be always ~496ns off on my box.
For example, if I send 500B packets with param 'ipg' equal to 10000ns, I
would expect to end up either with 71kpps(i) or with 100kpps(ii).  But what
I get is 95kpps, 1e9/(10000+496).  At 5000ns ipg, I get neither 109kpps(i)
nor 200kpps(ii) but 182kpps, 1e9/(5000+496).

Presumably this 496ns is the CPU cost of shoving one packet towards the
NIC, and pktgen only after sending a packet starts waiting for 'ipg' ns
before transmitting the next packet.

Can we not compensate for this cost so that we either always get (i) or
(ii)?  Possibly by first getting the current time X, then transmitting
the packet, and then waiting until X+ipg, which would then give us (ii).
(We'd have to rename it to 'inter-packet-start gap' though, or something
like that.)

By tweaking the 'ipg' parameter I can generate pretty much any packet rate
I want, as long as I set ipg=(1e9/rate)-496 instead of something possibly
more straightforward.


thanks,
Lennert

Re: inter-packet gap in pktgen

[Ben Greear]

Lennert Buytenhek wrote:

> By tweaking the 'ipg' parameter I can generate pretty much any packet rate
> I want, as long as I set ipg=(1e9/rate)-496 instead of something possibly
> more straightforward.

That 496 will also change with load on the system, at least on average.  I
dealt with this by having a user-space app sample the rate and adjust the ipg
to keep the average rate where I want it.

So, I'd suggest leaving the ipg as it is, and use external tools to get
the exact pps that you are looking for.

Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: inter-packet gap in pktgen

[Lennert Buytenhek]

On Tue, Dec 07, 2004 at 02:47:15PM -0800, Ben Greear wrote:

> >By tweaking the 'ipg' parameter I can generate pretty much any packet rate
> >I want, as long as I set ipg=(1e9/rate)-496 instead of something possibly
> >more straightforward.
> 
> That 496 will also change with load on the system, at least on average.  I
> dealt with this by having a user-space app sample the rate and adjust the 
> ipg to keep the average rate where I want it.
> 
> So, I'd suggest leaving the ipg as it is, and use external tools to get
> the exact pps that you are looking for.

Just because it's possible that way doesn't mean that it's the only
way or even _the_ way of doing it.

Another option is:

	next_tx = get_time_in_ns();
	while (--count) {
		tx_packet();
		next_tx += 1e9/intended_pps;
		nanospin(next_tx - get_time_in_ns());
	}

This should be relatively independent of system load.  OK, I know,
time to show some code.


--L

Re: inter-packet gap in pktgen

[Robert Olsson]

Lennert Buytenhek writes:
 > 
 > Another option is:
 > 
 > 	next_tx = get_time_in_ns();
 > 	while (--count) {
 > 		tx_packet();
 > 		next_tx += 1e9/intended_pps;
 > 		nanospin(next_tx - get_time_in_ns());
 > 	}

 Hello!

 I think this what Ben is doing with his userland app. Ev. adjusting
 the ipg delay in runtime? A kind of control system. Even the device 
 stats could possible be read.

 
 > This should be relatively independent of system load.  OK, I know,
 > time to show some code.
 
 Also an idea might be to have some kind of option to use pktgen w. 
 existent qdisc/tc infrastructure for this type of tests. Ben did
 you try this?
 
						--ro

Re: inter-packet gap in pktgen

[Lennert Buytenhek]

On Wed, Dec 08, 2004 at 11:21:54AM +0100, Robert Olsson wrote:

>  > Another option is:
>  > 
>  > 	next_tx = get_time_in_ns();
>  > 	while (--count) {
>  > 		tx_packet();
>  > 		next_tx += 1e9/intended_pps;
>  > 		nanospin(next_tx - get_time_in_ns());
>  > 	}
> 
>  Hello!
> 
>  I think this what Ben is doing with his userland app. Ev. adjusting
>  the ipg delay in runtime? A kind of control system.

I think what Ben is doing is just measuring the # of pps and then
using a PI-controller or something like that to adjust the ipg.

What I mean is something like this (warning: whitespace damaged.)  But
this gives me an IPG that is consistently too small.  Are the nanospin()
and pg_udelay() functions accurate?


--L


--- pktgen.c.04111.orig 2004-12-08 11:57:03.627392497 +0100
+++ pktgen.c    2004-12-08 12:11:35.777150214 +0100
@@ -2794,7 +2794,13 @@
                                pkt_dev->last_ok = 0;
                                pkt_dev->next_tx_ns = getRelativeCurNs(); /* TODO */
                }
-               pkt_dev->next_tx_ns = getRelativeCurNs() + pkt_dev->ipg;
+               if (now == 0)
+                       now = getRelativeCurNs();
+               pkt_dev->next_tx_ns += pkt_dev->ipg;
+               if (pkt_dev->next_tx_ns < now) {
+                       pkt_dev->next_tx_ns = now;
+                       pkt_dev->errors++;         /* missed ipg deadline */
+               }
        }

        else {  /* Retry it next time */

Re: inter-packet gap in pktgen

[Ben Greear]

Robert Olsson wrote:
> Lennert Buytenhek writes:
>  > 
>  > Another option is:
>  > 
>  > 	next_tx = get_time_in_ns();
>  > 	while (--count) {
>  > 		tx_packet();
>  > 		next_tx += 1e9/intended_pps;
>  > 		nanospin(next_tx - get_time_in_ns());
>  > 	}
> 
>  Hello!
> 
>  I think this what Ben is doing with his userland app. Ev. adjusting
>  the ipg delay in runtime? A kind of control system. Even the device 
>  stats could possible be read.

Actually, I do return virtually the entire pktgen_interface_info
structure through an IOCTL call.  I do adjust the ipg often based on
what my control logic tells me I should do, 10 times a second I believe.

I found I needed external control because the control is fairly complex,
and I didn't want it in the kernel.  I also aim to allow X number of
interfaces to be used at once, and allow a mixture of different speeds.
The traffic on the interfaces affect each other..so I found the external
control useful again...

All that said, when we do calculate the 'next-tx' timer in the kernel,
there is no reason I see not to use the method above to get as accurate as
possible.

>  > This should be relatively independent of system load.  OK, I know,
>  > time to show some code.
>  
>  Also an idea might be to have some kind of option to use pktgen w. 
>  existent qdisc/tc infrastructure for this type of tests. Ben did
>  you try this?

Actually, yes.  My pktgen no longer busy-spins.  I probably add a bit
of jitter at a very low level, but over all system performance is much
better.  I use a callback from net_device.h to wake up the pktgen thread,
and I put it to sleep as soon as I have no more work to do, or I get
a hard-start-xmit error.  The netdev.h code looks like this:

@@ -474,9 +482,18 @@
  	void                    (*poll_controller)(struct net_device *dev);
  #endif

+        /* Callback for when the queue is woken, used by pktgen currently */
+        int			(*notify_queue_woken)(struct net_device *dev);
+        void* nqw_data; /* To be used by the method above as needed */
+
  	/* bridge stuff */
  	struct net_bridge_port	*br_port;





static inline void netif_wake_queue(struct net_device *dev)
{
#ifdef CONFIG_NETPOLL_TRAP
	if (netpoll_trap())
		return;
#endif
	if (test_and_clear_bit(__LINK_STATE_XOFF, &dev->state)) {
		__netif_schedule(dev);

                 if (dev->notify_queue_woken) {
                    dev->notify_queue_woken(dev);
                 }
         }
}


pktgen registers with devices' notify_queue_woken hook.  For VLANs, I have some fairly complex
logic in pktgen so that it still registers with the 'real' device
since the VLAN won't have the netif_wake_queue called when the
real device drains it's fifo.

I'd be happy to have this included in the kernel, or be a basis for something
similar, so let me know if you'd like to see the full patch.  It is unlikely
that davem will allow my pktgen-rx code in as well, but the tx stuff might
still prove useful.

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: inter-packet gap in pktgen

[Lennert Buytenhek]

On Wed, Dec 08, 2004 at 11:32:14AM -0800, Ben Greear wrote:

> Actually, yes.  My pktgen no longer busy-spins.  [...]

If you post a patch we can all perhaps have a look at it?


--L

Re: inter-packet gap in pktgen

[Ben Greear]

Lennert Buytenhek wrote:
> On Wed, Dec 08, 2004 at 11:32:14AM -0800, Ben Greear wrote:
> 
> 
>>Actually, yes.  My pktgen no longer busy-spins.  [...]
> 
> 
> If you post a patch we can all perhaps have a look at it?

My consolidated patch, which includes pktgen (with rx-pktgen logic),
macvlans, my send-to-self patch, some e100 and e1000 diagnostics work, etc,
can be found here:

http://www.candelatech.com/oss/candela_2.6.9.patch

The pktgen diff is about the same size as the pktgen.c file, so I also
uploaded the pktgen files here:

http://www.candelatech.com/oss/pktgen.h
http://www.candelatech.com/oss/pktgen.c

Enjoy,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com