[patch] pktgen: bug when calling ndelay in x86 architectures

[patch] pktgen: bug when calling ndelay in x86 architectures

[Daniel Turull]

The value selected to delay the transmission in pktgen with the ndelay function should be lower.
In Linux/arch/x86/include/asm/delay.h and Linux/arch/sh/include/asm/delay.h
the maximal expected value for a constant is 20000 ns.

Signed-off-by: Daniel Turull <daniel.turull@gmail.com>
---
diff --git a/net/core/pktgen.c b/net/core/pktgen.c
index 796044a..e17bd41 100644
--- a/net/core/pktgen.c
+++ b/net/core/pktgen.c
@@ -2145,7 +2145,7 @@ static void spin(struct pktgen_dev *pkt_dev, ktime_t spin_until)
 	}
 
 	start_time = ktime_now();
-	if (remaining < 100000)
+	if (remaining < 20000)
 		ndelay(remaining);	/* really small just spin */
 	else {
 		/* see do_nanosleep */

Re: [patch] pktgen: bug when calling ndelay in x86 architectures

[Eric Dumazet]

Le mardi 18 octobre 2011 à 13:08 +0200, Daniel Turull a écrit :
> The value selected to delay the transmission in pktgen with the ndelay function should be lower.
> In Linux/arch/x86/include/asm/delay.h and Linux/arch/sh/include/asm/delay.h
> the maximal expected value for a constant is 20000 ns.
> 
> Signed-off-by: Daniel Turull <daniel.turull@gmail.com>
> ---
> diff --git a/net/core/pktgen.c b/net/core/pktgen.c
> index 796044a..e17bd41 100644
> --- a/net/core/pktgen.c
> +++ b/net/core/pktgen.c
> @@ -2145,7 +2145,7 @@ static void spin(struct pktgen_dev *pkt_dev, ktime_t spin_until)
>  	}
>  
>  	start_time = ktime_now();
> -	if (remaining < 100000)
> +	if (remaining < 20000)
>  		ndelay(remaining);	/* really small just spin */
>  	else {
>  		/* see do_nanosleep */

But 'remaining' is not a constant.

If we want exactly 40.000 packets per second rate (25 us between
packets), your patch makes this not quite possible without
CONFIG_HIGH_RES_TIMERS and probable high jitter because of scheduler
effects.

pktgen is kind of special, we _want_ a cpu for our exclusive use.

Re: [patch] pktgen: bug when calling ndelay in x86 architectures

[Ben Hutchings]

On Tue, 2011-10-18 at 13:56 +0200, Eric Dumazet wrote:
> Le mardi 18 octobre 2011 à 13:08 +0200, Daniel Turull a écrit :
> > The value selected to delay the transmission in pktgen with the ndelay function should be lower.
> > In Linux/arch/x86/include/asm/delay.h and Linux/arch/sh/include/asm/delay.h
> > the maximal expected value for a constant is 20000 ns.
> > 
> > Signed-off-by: Daniel Turull <daniel.turull@gmail.com>
> > ---
> > diff --git a/net/core/pktgen.c b/net/core/pktgen.c
> > index 796044a..e17bd41 100644
> > --- a/net/core/pktgen.c
> > +++ b/net/core/pktgen.c
> > @@ -2145,7 +2145,7 @@ static void spin(struct pktgen_dev *pkt_dev, ktime_t spin_until)
> >  	}
> >  
> >  	start_time = ktime_now();
> > -	if (remaining < 100000)
> > +	if (remaining < 20000)
> >  		ndelay(remaining);	/* really small just spin */
> >  	else {
> >  		/* see do_nanosleep */
> 
> But 'remaining' is not a constant.
> 
> If we want exactly 40.000 packets per second rate (25 us between
> packets), your patch makes this not quite possible without
> CONFIG_HIGH_RES_TIMERS and probable high jitter because of scheduler
> effects.
> 
> pktgen is kind of special, we _want_ a cpu for our exclusive use.

AIUI, the reason for limits on delays is not that it's bad practice to
spin for so long, but that the delay calculations may overflow or
otherwise become inaccurate.

Ben.

-- 
Ben Hutchings, Staff Engineer, Solarflare
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.

Re: [patch] pktgen: bug when calling ndelay in x86 architectures

[Eric Dumazet]

Le mardi 18 octobre 2011 à 15:00 +0100, Ben Hutchings a écrit :

> AIUI, the reason for limits on delays is not that it's bad practice to
> spin for so long, but that the delay calculations may overflow or
> otherwise become inaccurate.

OK, I can understand that, then a more appropriate patch would be :


diff --git a/net/core/pktgen.c b/net/core/pktgen.c
index 796044a..28bbf5b 100644
--- a/net/core/pktgen.c
+++ b/net/core/pktgen.c
@@ -2145,9 +2145,12 @@ static void spin(struct pktgen_dev *pkt_dev, ktime_t spin_until)
 	}
 
 	start_time = ktime_now();
-	if (remaining < 100000)
-		ndelay(remaining);	/* really small just spin */
-	else {
+	if (remaining < 100000) {
+		if (remaining >= 10000)
+			udelay(remaining/1000);
+		else
+			ndelay(remaining);
+	} else {
 		/* see do_nanosleep */
 		hrtimer_init_sleeper(&t, current);
 		do {

Re: [patch] pktgen: bug when calling ndelay in x86 architectures

[Eric Dumazet]

Le mercredi 19 octobre 2011 à 11:33 +0200, Daniel Turull a écrit :
> Hi,
> then if we want to use the spin more often.
> maybe we can increase the constant from 100000 (0.1ms) to 1000000 (1ms)?
> How was the current value chosen?
> 

Based on user needs ;)

> I did some measurements of the inter-arrival time between packets
> and with bigger values the maximal is reduced in the rates between
> 2kpps and 20kpps.
> 

ndelay()/udelay() have some inaccuracies, for 'long' values, because of
rounding errors.

If we spin, just call ktime_now() in a loop until spin_until is
reached...

That way you get max possible resolution, given kernel time service
constraints.

Untested patch :

diff --git a/net/core/pktgen.c b/net/core/pktgen.c
index 38d6577..5c7e900 100644
--- a/net/core/pktgen.c
+++ b/net/core/pktgen.c
@@ -2145,9 +2145,11 @@ static void spin(struct pktgen_dev *pkt_dev, ktime_t spin_until)
 	}
 
 	start_time = ktime_now();
-	if (remaining < 100000)
-		ndelay(remaining);	/* really small just spin */
-	else {
+	if (remaining < 100000) {
+		do {
+			end_time = ktime_now();
+		} while (ktime_lt(end_time, spin_until));
+	} else {
 		/* see do_nanosleep */
 		hrtimer_init_sleeper(&t, current);
 		do {
@@ -2162,8 +2164,8 @@ static void spin(struct pktgen_dev *pkt_dev, ktime_t spin_until)
 			hrtimer_cancel(&t.timer);
 		} while (t.task && pkt_dev->running && !signal_pending(current));
 		__set_current_state(TASK_RUNNING);
+		end_time = ktime_now();
 	}
-	end_time = ktime_now();
 
 	pkt_dev->idle_acc += ktime_to_ns(ktime_sub(end_time, start_time));
 	pkt_dev->next_tx = ktime_add_ns(spin_until, pkt_dev->delay);

Re: [patch] pktgen: bug when calling ndelay in x86 architectures

[Daniel Turull]

Hi,

I tested the patch and it works well.


On 10/19/2011 12:13 PM, Eric Dumazet wrote:
> Le mercredi 19 octobre 2011 à 11:33 +0200, Daniel Turull a écrit :
>> Hi,
>> then if we want to use the spin more often.
>> maybe we can increase the constant from 100000 (0.1ms) to 1000000 (1ms)?
>> How was the current value chosen?
>>
> 
> Based on user needs ;)

I think if we increase the constant to 1ms, we will reduce the jitter if we have
a rate between 1kpps and 10 kpps, but I guess is not a big deal.

I've plot this new graph with this patch:
http://tslab.ssvl.kth.se/pktgen/img/inter_eric1.eps

> 
>> I did some measurements of the inter-arrival time between packets
>> and with bigger values the maximal is reduced in the rates between
>> 2kpps and 20kpps.
>>
> 
> ndelay()/udelay() have some inaccuracies, for 'long' values, because of
> rounding errors.
> 
> If we spin, just call ktime_now() in a loop until spin_until is
> reached...
> 
> That way you get max possible resolution, given kernel time service
> constraints.
> 
> Untested patch :
> 
> diff --git a/net/core/pktgen.c b/net/core/pktgen.c
> index 38d6577..5c7e900 100644
> --- a/net/core/pktgen.c
> +++ b/net/core/pktgen.c
> @@ -2145,9 +2145,11 @@ static void spin(struct pktgen_dev *pkt_dev, ktime_t spin_until)
>  	}
>  
>  	start_time = ktime_now();
> -	if (remaining < 100000)
> -		ndelay(remaining);	/* really small just spin */
> -	else {
> +	if (remaining < 100000) {
> +		do {
> +			end_time = ktime_now();
> +		} while (ktime_lt(end_time, spin_until));
> +	} else {
>  		/* see do_nanosleep */
>  		hrtimer_init_sleeper(&t, current);
>  		do {
> @@ -2162,8 +2164,8 @@ static void spin(struct pktgen_dev *pkt_dev, ktime_t spin_until)
>  			hrtimer_cancel(&t.timer);
>  		} while (t.task && pkt_dev->running && !signal_pending(current));
>  		__set_current_state(TASK_RUNNING);
> +		end_time = ktime_now();
>  	}
> -	end_time = ktime_now();
>  
>  	pkt_dev->idle_acc += ktime_to_ns(ktime_sub(end_time, start_time));
>  	pkt_dev->next_tx = ktime_add_ns(spin_until, pkt_dev->delay);
> 
> 

Daniel

Re: [patch] pktgen: bug when calling ndelay in x86 architectures

[Eric Dumazet]

Le jeudi 20 octobre 2011 à 15:22 +0200, Daniel Turull a écrit :
> Hi,
> 
> I tested the patch and it works well.
> 

Thanks !


> 
> I think if we increase the constant to 1ms, we will reduce the jitter if we have
> a rate between 1kpps and 10 kpps, but I guess is not a big deal.
> 


> I've plot this new graph with this patch:
> http://tslab.ssvl.kth.se/pktgen/img/inter_eric1.eps

Unfortunately, the sender cpu might be preempted by timer irq or other
expensive irq, so the Min/Max values are not very different I guess.

I dont understand your Min values.

At 100 pps, how is it possible to have a Min value of ~5000 ns ?

Re: [patch] pktgen: bug when calling ndelay in x86 architectures

[Daniel Turull]

On 10/20/2011 03:44 PM, Eric Dumazet wrote:
> Le jeudi 20 octobre 2011 à 15:22 +0200, Daniel Turull a écrit :
>> Hi,
>>
>> I tested the patch and it works well.
>>
> 
> Thanks !
> 
> 
>>
>> I think if we increase the constant to 1ms, we will reduce the jitter if we have
>> a rate between 1kpps and 10 kpps, but I guess is not a big deal.
>>
> 
> 
>> I've plot this new graph with this patch:
>> http://tslab.ssvl.kth.se/pktgen/img/inter_eric1.eps
> 
> Unfortunately, the sender cpu might be preempted by timer irq or other
> expensive irq, so the Min/Max values are not very different I guess.
> 
> I dont understand your Min values.
> 
> At 100 pps, how is it possible to have a Min value of ~5000 ns ?
> 
> 
My assumption is that for low rate, the min value is caused in the 
beginning of the test. When we start the transmission in pktgen_run(),
we set the pkt_dev->next_tx to the current time but the are
more operation to do, so the first transmission is a bit delayed. 
Even more if the cpu is preempted.
For the second packet, we are taking the pkt_dev->next_tx as a reference
and add the delay, in order to decide when to send.
So, my guess is that the first packet is delayed
and then we send the second packet only after a short time, in order
to keep the average rate in the transmission.

Daniel

Re: [patch] pktgen: bug when calling ndelay in x86 architectures

[David Miller]

From: Eric Dumazet <eric.dumazet@gmail.com>
Date: Tue, 18 Oct 2011 16:47:44 +0200

> Le mardi 18 octobre 2011 à 15:00 +0100, Ben Hutchings a écrit :
> 
>> AIUI, the reason for limits on delays is not that it's bad practice to
>> spin for so long, but that the delay calculations may overflow or
>> otherwise become inaccurate.
> 
> OK, I can understand that, then a more appropriate patch would be :

I think doing the udelay/ndelay thing is the way to go for 'net' and
-stable.  We can do something sophisticated with ktime et al. in
'net-next'.

Eric, could you please formally submit this patch with proper
changelog etc.?

Thanks.

Re: [patch] pktgen: bug when calling ndelay in x86 architectures

[Eric Dumazet]

Le jeudi 20 octobre 2011 à 16:24 -0400, David Miller a écrit :
> From: Eric Dumazet <eric.dumazet@gmail.com>
> Date: Tue, 18 Oct 2011 16:47:44 +0200
> 
> > Le mardi 18 octobre 2011 à 15:00 +0100, Ben Hutchings a écrit :
> > 
> >> AIUI, the reason for limits on delays is not that it's bad practice to
> >> spin for so long, but that the delay calculations may overflow or
> >> otherwise become inaccurate.
> > 
> > OK, I can understand that, then a more appropriate patch would be :
> 
> I think doing the udelay/ndelay thing is the way to go for 'net' and
> -stable.  We can do something sophisticated with ktime et al. in
> 'net-next'.
> 

Well, I am not sure a patch is needed for net, since there is no bug,
but maybe small inaccuracies ? Correct me if I misunderstood Daniel !

> Eric, could you please formally submit this patch with proper
> changelog etc.?

Sure !

[PATCH net-next] pktgen: remove ndelay() call

Daniel Turull reported inaccuracies in pktgen when using low packet
rates, because we call ndelay(val) with values bigger than 20000.

Instead of calling ndelay() for delays < 100us, we can instead loop
calling ktime_now() only.

Reported-by: Daniel Turull <daniel.turull@gmail.com>
Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
---
 net/core/pktgen.c |   11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/net/core/pktgen.c b/net/core/pktgen.c
index 6bbf008..0001c24 100644
--- a/net/core/pktgen.c
+++ b/net/core/pktgen.c
@@ -2145,9 +2145,12 @@ static void spin(struct pktgen_dev *pkt_dev, ktime_t spin_until)
 	}
 
 	start_time = ktime_now();
-	if (remaining < 100000)
-		ndelay(remaining);	/* really small just spin */
-	else {
+	if (remaining < 100000) {
+		/* for small delays (<100us), just loop until limit is reached */
+		do {
+			end_time = ktime_now();
+		} while (ktime_lt(end_time, spin_until));
+	} else {
 		/* see do_nanosleep */
 		hrtimer_init_sleeper(&t, current);
 		do {
@@ -2162,8 +2165,8 @@ static void spin(struct pktgen_dev *pkt_dev, ktime_t spin_until)
 			hrtimer_cancel(&t.timer);
 		} while (t.task && pkt_dev->running && !signal_pending(current));
 		__set_current_state(TASK_RUNNING);
+		end_time = ktime_now();
 	}
-	end_time = ktime_now();
 
 	pkt_dev->idle_acc += ktime_to_ns(ktime_sub(end_time, start_time));
 	pkt_dev->next_tx = ktime_add_ns(spin_until, pkt_dev->delay);

Re: [patch] pktgen: bug when calling ndelay in x86 architectures

[David Miller]

From: Eric Dumazet <eric.dumazet@gmail.com>
Date: Thu, 20 Oct 2011 22:55:38 +0200

> Well, I am not sure a patch is needed for net, since there is no bug,
> but maybe small inaccuracies ? Correct me if I misunderstood Daniel !

Ok that appears to be the case, so this is not something we should
deal with in the 'net' tree.

The constant ndelay() case would purposely cause a build failure for
values larger than 40000, but the specific call site we are discussing
in pktgen is never constant and therefore should would never trigger
that bug check.

>> Eric, could you please formally submit this patch with proper
>> changelog etc.?
> 
> Sure !
> 
> [PATCH net-next] pktgen: remove ndelay() call

Applied, thanks!