udp weirdness

udp weirdness

[Eric Lemoine]

I'm observing some UDP weirdness, or I'd better say some UDP behaviour
that I can't explain.

Two machines: one sending a UDP flow (using sendto) and another receiving 
this UDP flow (using bind + recv). 

When the dgram length is lower that 357 Bytes I observe strange results
at the send side. My home-made udp_tx program gives the following:

$./udp_tx -h 192.168.4.1 -m 357
357 1312621 357.518

357 is the dgram length (in B), 1312621 the number of dgrams sent and 
357.518 the perceived thruput (in Mbits/s). The weirdness is that I
get 357.518 Mbits/s whereas the underlying network is 10Mbits/s!

At the receive side the results are consistent (obviously):

$./udp_rx -m 357
357 29519 8.00884

<netstat -s --udp> on the send machine before and after the run also
gives me such a large amount of sent packets (~1312700), whereas
</sbin/ifconfig> confirms that about 29519 packets have been sent
out.

Below 357 Bytes, the same kind of results are observed. Above 357 Bytes,
the results make more sense to me:

$./udp_tx -h 192.168.4.1 -m 358
358 29505 8.04393

$./udp_rx -m 358
358 29468 8.0179

Does anybody know where I lose packets? And why do I lose them only when
the dgram length is below 357 Bytes?

BTW, I'm running 2.4.18-vanilla w/ the 3c59x driver.

Thx.
-- 
Eric

Re: udp weirdness

[Eric Lemoine]

On Tue, Sep 24, 2002 at 08:50:46AM +0200, Eric Lemoine wrote:
> I'm observing some UDP weirdness, or I'd better say some UDP behaviour
> that I can't explain.
> 
> Two machines: one sending a UDP flow (using sendto) and another receiving 
> this UDP flow (using bind + recv). 
> 
> When the dgram length is lower that 357 Bytes I observe strange results
> at the send side. My home-made udp_tx program gives the following:
> 
> $./udp_tx -h 192.168.4.1 -m 357
> 357 1312621 357.518
> 
> 357 is the dgram length (in B), 1312621 the number of dgrams sent and 
> 357.518 the perceived thruput (in Mbits/s). The weirdness is that I
> get 357.518 Mbits/s whereas the underlying network is 10Mbits/s!
> 
> At the receive side the results are consistent (obviously):
> 
> $./udp_rx -m 357
> 357 29519 8.00884
> 
> <netstat -s --udp> on the send machine before and after the run also
> gives me such a large amount of sent packets (~1312700), whereas
> </sbin/ifconfig> confirms that about 29519 packets have been sent
> out.
> 
> Below 357 Bytes, the same kind of results are observed. Above 357 Bytes,
> the results make more sense to me:
> 
> $./udp_tx -h 192.168.4.1 -m 358
> 358 29505 8.04393
> 
> $./udp_rx -m 358
> 358 29468 8.0179
> 
> Does anybody know where I lose packets? And why do I lose them only when
> the dgram length is below 357 Bytes?
> 
> BTW, I'm running 2.4.18-vanilla w/ the 3c59x driver.

I figured out that packets can be dropped in pfifo_fast_enqueue()
[the default qdisc's enqueue func], even though the driver/kernel 
flow control has triggered. 

And sendto does not notify the user when packet gets dropped because 
the output queue overflows (as indicated in sendto manpage).

Why doesn't the kernel just put the process into sleep instead of 
dropping packets?

Thanks.
-- 
Eric

Re: udp weirdness

[jamal]

On Fri, 27 Sep 2002, Eric Lemoine wrote:

> I figured out that packets can be dropped in pfifo_fast_enqueue()
> [the default qdisc's enqueue func], even though the driver/kernel
> flow control has triggered.
>
> And sendto does not notify the user when packet gets dropped because
> the output queue overflows (as indicated in sendto manpage).
>
> Why doesn't the kernel just put the process into sleep instead of
> dropping packets?
>

What trigger do you suggest to wake up the process again?
A better idea maybe to return something to the socket so it can
manage things instead -- not sure what to return though that wouldnt
break some standard;

cheers,
jamal

Re: udp weirdness

[Matti Aarnio]

On Fri, Sep 27, 2002 at 10:53:00AM -0400, jamal wrote:
> On Fri, 27 Sep 2002, Eric Lemoine wrote:
> 
> > I figured out that packets can be dropped in pfifo_fast_enqueue()
> > [the default qdisc's enqueue func], even though the driver/kernel
> > flow control has triggered.
...
> What trigger do you suggest to wake up the process again?
> A better idea maybe to return something to the socket so it can
> manage things instead -- not sure what to return though that wouldnt
> break some standard;


"man sendto" error return codes:

       ENOBUFS
              The  output queue for a network interface was full.
              This generally indicates  that  the  interface  has
              stopped  sending,  but  may  be caused by transient
              congestion.  (This cannot occur in  Linux,  packets
              are just silently dropped when a device queue over­
              flows.)

> cheers,
> jamal

/Matti Aarnio

Re: udp weirdness

[Eric Lemoine]

On Fri, Sep 27, 2002 at 10:53:00AM -0400, jamal wrote:
>
> On Fri, 27 Sep 2002, Eric Lemoine wrote:
> 
> > I figured out that packets can be dropped in pfifo_fast_enqueue()
> > [the default qdisc's enqueue func], even though the driver/kernel
> > flow control has triggered.
> >
> > And sendto does not notify the user when packet gets dropped because
> > the output queue overflows (as indicated in sendto manpage).
> >
> > Why doesn't the kernel just put the process into sleep instead of
> > dropping packets?
> >
> 
> What trigger do you suggest to wake up the process again?
> A better idea maybe to return something to the socket so it can
> manage things instead -- not sure what to return though that wouldnt
> break some standard;

Linux seems to be the only one to not return ENOBUFS in this overflow
case (from sendto manpage), so i suspect it should not break any
standard.

-- 
Eric

Re: udp weirdness

[Eric Lemoine]

> > I figured out that packets can be dropped in pfifo_fast_enqueue()
> > [the default qdisc's enqueue func], even though the driver/kernel
> > flow control has triggered.
> >
> > And sendto does not notify the user when packet gets dropped because
> > the output queue overflows (as indicated in sendto manpage).
> >
> > Why doesn't the kernel just put the process into sleep instead of
> > dropping packets?
> >
> 
> What trigger do you suggest to wake up the process again?

I was thinking of putting processes into sleep on per-device lists.
Once the net driver sees the NIC is ready to send again it wakes up all
processes sleeping on its list. [If the socket from which the packet
comes from is marked non-blocking the process is not put into sleep 
(obviously) and EAGAIN is returned.]

Throwing away packets when not absolutely necessary does not make sense
to me. Please correct me if i'm wrong.

> A better idea maybe to return something to the socket so it can
> manage things instead -- not sure what to return though that wouldnt
> break some standard;
> 
> cheers,
> jamal

-- 
Eric

Re: udp weirdness

[jamal]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: TEXT/PLAIN; charset=X-UNKNOWN, Size: 1262 bytes --]


Ok, understood. Actually we already seem to have enobufs being returned.

Eric,
Does the attached patch fix it? Not tested or even compiled.
Someone going to change the manpages?

cheers,
jamal

On Fri, 27 Sep 2002, Matti Aarnio wrote:

> On Fri, Sep 27, 2002 at 10:53:00AM -0400, jamal wrote:
> > On Fri, 27 Sep 2002, Eric Lemoine wrote:
> >
> > > I figured out that packets can be dropped in pfifo_fast_enqueue()
> > > [the default qdisc's enqueue func], even though the driver/kernel
> > > flow control has triggered.
> ...
> > What trigger do you suggest to wake up the process again?
> > A better idea maybe to return something to the socket so it can
> > manage things instead -- not sure what to return though that wouldnt
> > break some standard;
>
>
> "man sendto" error return codes:
>
>        ENOBUFS
>               The  output queue for a network interface was full.
>               This generally indicates  that  the  interface  has
>               stopped  sending,  but  may  be caused by transient
>               congestion.  (This cannot occur in  Linux,  packets
>               are just silently dropped when a device queue over­
>               flows.)
>
> > cheers,
> > jamal
>
> /Matti Aarnio
>

[-- Attachment #2: Type: TEXT/PLAIN, Size: 922 bytes --]

--- linux/net/ipv4/ip_output.c	2002/09/29 10:43:10	1.1
+++ linux/net/ipv4/ip_output.c	2002/09/29 10:43:10
@@ -603,8 +603,11 @@
 		err = NF_HOOK(PF_INET, NF_IP_LOCAL_OUT, skb, NULL, 
 			      skb->dst->dev, output_maybe_reroute);
 		if (err) {
-			if (err > 0)
-				err = sk->protinfo.af_inet.recverr ? net_xmit_errno(err) : 0;
+			if (err > 0) {
+				err = net_xmit_errno(err);
+				if (err && sk->protinfo.af_inet.recverr)
+				       sk->protinfo.af_inet.recverr = err;
+			}
 			if (err)
 				goto error;
 		}
@@ -713,8 +716,11 @@
 
 	err = NF_HOOK(PF_INET, NF_IP_LOCAL_OUT, skb, NULL, rt->u.dst.dev,
 		      output_maybe_reroute);
-	if (err > 0)
-		err = sk->protinfo.af_inet.recverr ? net_xmit_errno(err) : 0;
+	if (err > 0) {
+		err = net_xmit_errno(err);
+		if (err && sk->protinfo.af_inet.recverr)
+		       	 sk->protinfo.af_inet.recverr = err;
+	}
 	if (err)
 		goto error;
 out:

Re: udp weirdness

[Eric Lemoine]

> Ok, understood. Actually we already seem to have enobufs being returned.
> 
> Eric,
> Does the attached patch fix it? Not tested or even compiled.
> Someone going to change the manpages?

protinfo.af_inet.recverr is a flag (1-bit variable) to enable the
"extended reliable error message passing"; I dont see any reason 
for modifying it here. Setting this flag is the user's responsability, 
isn't it?

My patch would look like this:

--- ip_output.c.old     Mon Sep 30 10:34:07 2002
+++ ip_output.c Mon Sep 30 10:40:08 2002
@@ -604,7 +604,7 @@
                              skb->dst->dev, output_maybe_reroute);
                if (err) {
                        if (err > 0)
-                               err = sk->protinfo.af_inet.recverr ? net_xmit_errno(err) : 0;
+                               err = (err == NET_XMIT_DROP || sk->protinfo.af_inet.recverr ) ? net_xmit_errno(err) : 0;
                        if (err)
                                goto error;
                }
@@ -714,7 +714,7 @@
        err = NF_HOOK(PF_INET, NF_IP_LOCAL_OUT, skb, NULL,
rt->u.dst.dev,
                      output_maybe_reroute);
        if (err > 0)
-               err = sk->protinfo.af_inet.recverr ? net_xmit_errno(err) : 0;
+               err = (err == NET_XMIT_DROP || sk->protinfo.af_inet.recverr ) ? net_xmit_errno(err) : 0;
        if (err)
                goto error;
 out:

> --- linux/net/ipv4/ip_output.c	2002/09/29 10:43:10	1.1
> +++ linux/net/ipv4/ip_output.c	2002/09/29 10:43:10
> @@ -603,8 +603,11 @@
>  		err = NF_HOOK(PF_INET, NF_IP_LOCAL_OUT, skb, NULL, 
>  			      skb->dst->dev, output_maybe_reroute);
>  		if (err) {
> -			if (err > 0)
> -				err = sk->protinfo.af_inet.recverr ? net_xmit_errno(err) : 0;
> +			if (err > 0) {
> +				err = net_xmit_errno(err);
> +				if (err && sk->protinfo.af_inet.recverr)
> +				       sk->protinfo.af_inet.recverr = err;
> +			}
>  			if (err)
>  				goto error;
>  		}
> @@ -713,8 +716,11 @@
>  
>  	err = NF_HOOK(PF_INET, NF_IP_LOCAL_OUT, skb, NULL, rt->u.dst.dev,
>  		      output_maybe_reroute);
> -	if (err > 0)
> -		err = sk->protinfo.af_inet.recverr ? net_xmit_errno(err) : 0;
> +	if (err > 0) {
> +		err = net_xmit_errno(err);
> +		if (err && sk->protinfo.af_inet.recverr)
> +		       	 sk->protinfo.af_inet.recverr = err;
> +	}
>  	if (err)
>  		goto error;
>  out:

Re: udp weirdness

[jamal]

On Mon, 30 Sep 2002, Eric Lemoine wrote:

> > Ok, understood. Actually we already seem to have enobufs being returned.
> >
> > Eric,
> > Does the attached patch fix it? Not tested or even compiled.
> > Someone going to change the manpages?
>
> protinfo.af_inet.recverr is a flag (1-bit variable) to enable the
> "extended reliable error message passing"; I dont see any reason
> for modifying it here. Setting this flag is the user's responsability,
> isn't it?

I dont see any reason for modifying it there either - it is used for
mostly for incoming ICMP errors; but the original piece of code was
modifying it so i left it as is; however, thats not what you were chasing
and i dont see how what you just posted is solving your problem.
Did you even bother to test what i sent you? Or for that matter
bother testing what you posted?

cheers,
jamal

Re: udp weirdness

[jamal]

On Mon, 30 Sep 2002, Eric Lemoine wrote:

> > Ok, understood. Actually we already seem to have enobufs being returned.
> >
> > Eric,
> > Does the attached patch fix it? Not tested or even compiled.
> > Someone going to change the manpages?
>
> protinfo.af_inet.recverr is a flag (1-bit variable) to enable the
> "extended reliable error message passing"; I dont see any reason
> for modifying it here. Setting this flag is the user's responsability,
> isn't it?
>

Ok, sorry, I didnt realize i was the one setting it; What i meant
was to use sk->protinfo.af_inet.recverr only in the case of all is
clean from the device layer.
i.e do something along the lines of:

--- ip_output.c	2002/09/29 08:50:36	1.1
+++ ip_output.c	2002/09/30 06:24:32
@@ -603,8 +603,11 @@
 		err = NF_HOOK(PF_INET, NF_IP_LOCAL_OUT, skb, NULL,
 			      skb->dst->dev, output_maybe_reroute);
 		if (err) {
-			if (err > 0)
-				err = sk->protinfo.af_inet.recverr ? net_xmit_errno(err) : 0;
+			if (err > 0) {
+				err = net_xmit_errno(err);
+				if (!err && sk->protinfo.af_inet.recverr)
+				       err = sk->protinfo.af_inet.recverr;
+			}
 			if (err)
 				goto error;
 		}
@@ -713,8 +716,11 @@

 	err = NF_HOOK(PF_INET, NF_IP_LOCAL_OUT, skb, NULL, rt->u.dst.dev,
 		      output_maybe_reroute);
-	if (err > 0)
-		err = sk->protinfo.af_inet.recverr ? net_xmit_errno(err) : 0;
+	if (err > 0) {
+		err = net_xmit_errno(err);
+		if (!err && sk->protinfo.af_inet.recverr)
+		       	 err = sk->protinfo.af_inet.recverr;
+	}
 	if (err)
 		goto error;
 out:

Re: udp weirdness

[jamal]

On Mon, 30 Sep 2002, jamal wrote:

> was to use sk->protinfo.af_inet.recverr only in the case of all is
> clean from the device layer.

Of course if you want to say sk->protinfo.af_inet.recverr is higher
priority then the patch becomes;->

--- ip_output.c	2002/09/29 08:50:36	1.1
+++ ip_output.c	2002/09/30 06:38:32
@@ -603,8 +603,9 @@
 		err = NF_HOOK(PF_INET, NF_IP_LOCAL_OUT, skb, NULL,
 			      skb->dst->dev, output_maybe_reroute);
 		if (err) {
-			if (err > 0)
-				err = sk->protinfo.af_inet.recverr ? net_xmit_errno(err) : 0;
+			if (err > 0) {
+				 err = sk->protinfo.af_inet.recverr ? sk->protinfo.af_inet.recverr:net_xmit_errno(err);
+			}
 			if (err)
 				goto error;
 		}
@@ -713,8 +714,9 @@

 	err = NF_HOOK(PF_INET, NF_IP_LOCAL_OUT, skb, NULL, rt->u.dst.dev,
 		      output_maybe_reroute);
-	if (err > 0)
-		err = sk->protinfo.af_inet.recverr ? net_xmit_errno(err) : 0;
+	if (err > 0) {
+			err = sk->protinfo.af_inet.recverr ? sk->protinfo.af_inet.recverr:net_xmit_errno(err);
+	}
 	if (err)
 		goto error;
 out:

PATCH Re: udp weirdness

[jamal]

Eric,

you are right about sk->protinfo.af_inet.recverr ..
I checked what other OSes do and i am convinced that the patch below
at least makes us behave like the BSDs.
Alexey/Dave, sk->protinfo.af_inet.recverr is not needed for enobufs to
be propagated back to the socket level; please review and probably
apply:

cheers,
jamal

--- ip_output.c	2002/09/29 08:50:36	1.1
+++ ip_output.c	2002/09/30 06:24:32
@@ -603,8 +603,11 @@
 		err = NF_HOOK(PF_INET, NF_IP_LOCAL_OUT, skb, NULL,
 			      skb->dst->dev, output_maybe_reroute);
 		if (err) {
-			if (err > 0)
-				err = sk->protinfo.af_inet.recverr ? net_xmit_errno(err) : 0;
+			if (err > 0) {
+				err = net_xmit_errno(err);
+				if (!err && sk->protinfo.af_inet.recverr)
+				       err = sk->protinfo.af_inet.recverr;
+			}
 			if (err)
 				goto error;
 		}
@@ -713,8 +716,11 @@

 	err = NF_HOOK(PF_INET, NF_IP_LOCAL_OUT, skb, NULL, rt->u.dst.dev,
 		      output_maybe_reroute);
-	if (err > 0)
-		err = sk->protinfo.af_inet.recverr ? net_xmit_errno(err) : 0;
+	if (err > 0) {
+		err = net_xmit_errno(err);
+		if (!err && sk->protinfo.af_inet.recverr)
+		       	 err = sk->protinfo.af_inet.recverr;
+	}
 	if (err)
 		goto error;
 out:

Re: PATCH Re: udp weirdness

[Eric Lemoine]

> Eric,
> 
> you are right about sk->protinfo.af_inet.recverr ..
> I checked what other OSes do and i am convinced that the patch below
> at least makes us behave like the BSDs.
> Alexey/Dave, sk->protinfo.af_inet.recverr is not needed for enobufs to
> be propagated back to the socket level; please review and probably
> apply:

Jamal, 

Have you posted the right patch? I see that sk->protinfo.af_inet.recverr is
still around. Here follows the patch that I think is the correct one.
Please confirm. Thx.

--- ip_output.c.old     Mon Sep 30 10:34:07 2002
+++ ip_output.c Tue Oct  1 00:00:05 2002
@@ -604,7 +604,7 @@
                              skb->dst->dev, output_maybe_reroute);
                if (err) {
                        if (err > 0)
-                               err = sk->protinfo.af_inet.recverr ? net_xmit_errno(err) : 0;
+                               err = net_xmit_errno(err);
                        if (err)
                                goto error;
                }
@@ -714,7 +714,7 @@
        err = NF_HOOK(PF_INET, NF_IP_LOCAL_OUT, skb, NULL,
rt->u.dst.dev,
                      output_maybe_reroute);
        if (err > 0)
-               err = sk->protinfo.af_inet.recverr ? net_xmit_errno(err) : 0;
+               err = net_xmit_errno(err);
        if (err)
                goto error;
 out:

> --- ip_output.c	2002/09/29 08:50:36	1.1
> +++ ip_output.c	2002/09/30 06:24:32
> @@ -603,8 +603,11 @@
>  		err = NF_HOOK(PF_INET, NF_IP_LOCAL_OUT, skb, NULL,
>  			      skb->dst->dev, output_maybe_reroute);
>  		if (err) {
> -			if (err > 0)
> -				err = sk->protinfo.af_inet.recverr ? net_xmit_errno(err) : 0;
> +			if (err > 0) {
> +				err = net_xmit_errno(err);
> +				if (!err && sk->protinfo.af_inet.recverr)
> +				       err = sk->protinfo.af_inet.recverr;
> +			}
>  			if (err)
>  				goto error;
>  		}
> @@ -713,8 +716,11 @@
> 
>  	err = NF_HOOK(PF_INET, NF_IP_LOCAL_OUT, skb, NULL, rt->u.dst.dev,
>  		      output_maybe_reroute);
> -	if (err > 0)
> -		err = sk->protinfo.af_inet.recverr ? net_xmit_errno(err) : 0;
> +	if (err > 0) {
> +		err = net_xmit_errno(err);
> +		if (!err && sk->protinfo.af_inet.recverr)
> +		       	 err = sk->protinfo.af_inet.recverr;
> +	}
>  	if (err)
>  		goto error;
>  out:

-- 
Eric

Re: PATCH Re: udp weirdness

[jamal]

On Tue, 1 Oct 2002, Eric Lemoine wrote:

> Jamal,
>
> Have you posted the right patch?

;-> I dont know what happened; patch below is what i was trying to post
I deleted all the old patches this time to make sure ;->

> I see that sk->protinfo.af_inet.recverr is
> still around. Here follows the patch that I think is the correct one.
> Please confirm. Thx.
>

Yes it is correct but was missing one more bit from v6 added below.
Alexey/Dave this is the correct version ;->

cheers,
jamal

--- linux/net/ipv4/ip_output.c	2002/09/29 08:50:36     1.1
+++ linux/net/ipv4/ip_output.c	2002/09/30 09:11:15
@@ -603,8 +603,8 @@
 		err = NF_HOOK(PF_INET, NF_IP_LOCAL_OUT, skb, NULL,
 			      skb->dst->dev, output_maybe_reroute);
 		if (err) {
-			if (err > 0)
-				err = sk->protinfo.af_inet.recverr ? net_xmit_errno(err) : 0;
+			if (err > 0)
+				 err = net_xmit_errno(err);
 			if (err)
 				goto error;
 		}
@@ -713,8 +713,8 @@

 	err = NF_HOOK(PF_INET, NF_IP_LOCAL_OUT, skb, NULL, rt->u.dst.dev,
 		      output_maybe_reroute);
-	if (err > 0)
-		err = sk->protinfo.af_inet.recverr ? net_xmit_errno(err) : 0;
+	if (err > 0)
+			err = net_xmit_errno(err);
 	if (err)
 		goto error;
 out:
--- linux/net/ipv6/ip6_output.c	2002/09/30 18:30:52     1.1
+++ linux/net/ipv6/ip6_output.c	2002/09/30 18:31:30
@@ -684,7 +684,7 @@
 out:
 	ip6_dst_store(sk, dst, fl->nl_u.ip6_u.daddr == &np->daddr ? &np->daddr : NULL);
 	if (err > 0)
-		err = np->recverr ? net_xmit_errno(err) : 0;
+		err = net_xmit_errno(err);
 	return err;
 }

Re: PATCH Re: udp weirdness

[kuznet]

Hello!

> be propagated back to the socket level; please review and probably
> apply:

That was tried and failed miserably ages ago. Applications become
insane when getting this error from logging tons of useless messages
to abort()ing and cpu hogging. From the other hand silent reaction
to loss is almost exactly desired behaviour in presence of congestion
in the most of cases.

I have no idea, why that OSes work. Probably, they are so great that
never lose packets. We do. 

Probably, the world has changed since that time, but I have no desire
to repeat the experiment to be honest.

Alexey

Re: PATCH Re: udp weirdness

[jamal]

Alexey,

I agree; so just ignore the patch. You know you could have saved me
about an hour if you spoke earlier ;->
Eric, note what Alexey says. So to get enobufs please
use the sockoption to set IP_RECVERR
Someone should document this in the manpages.

cheers,
jamal

On Tue, 1 Oct 2002 kuznet@ms2.inr.ac.ru wrote:

> Hello!
>
> > be propagated back to the socket level; please review and probably
> > apply:
>
> That was tried and failed miserably ages ago. Applications become
> insane when getting this error from logging tons of useless messages
> to abort()ing and cpu hogging. From the other hand silent reaction
> to loss is almost exactly desired behaviour in presence of congestion
> in the most of cases.
>
> I have no idea, why that OSes work. Probably, they are so great that
> never lose packets. We do.
>
> Probably, the world has changed since that time, but I have no desire
> to repeat the experiment to be honest.
>
> Alexey
>
>
>

Re: PATCH Re: udp weirdness

[Chris Friesen]

kuznet@ms2.inr.ac.ru wrote:

> That was tried and failed miserably ages ago. Applications become
> insane when getting this error from logging tons of useless messages
> to abort()ing and cpu hogging. From the other hand silent reaction
> to loss is almost exactly desired behaviour in presence of congestion
> in the most of cases.
> 
> I have no idea, why that OSes work. Probably, they are so great that
> never lose packets. We do. 

One of the principles of software design that I was taught was the 
principle of least surprise.

If I'm looping on a sendto() with a blocking socket, I would expect the 
syscall to block until the packet has been handed off to the device 
driver.  This may mean blocking until local congestion backs off.

If I'm using non-blocking IO and there is local congestion, I would 
expect to get ENOBUFS or maybe EAGAIN/EWOULDBLOCK.

The way we do it now means that we can chew up massive amounts of cpu 
creating packets in userspace and throwing them away in the kernel, with 
no way of knowing from userspace that it is happening.  This just 
doesn't seem like the right thing to do.

Chris

Re: PATCH Re: udp weirdness

[kuznet]

Hello!

> If I'm looping on a sendto() with a blocking socket, I would expect the 
> syscall to block until the packet has been handed off to the device 
> driver.  This may mean blocking until local congestion backs off.

Feel free to implement. :-)


> The way we do it now means that we can chew up massive amounts of cpu 
> creating packets in userspace and throwing them away in the kernel, with 
> no way of knowing from userspace that it is happening.

What? If your applications is enough clever to handle ENOBUFS right,
set IP_RECVERR and live in peace.

Alexey

Re: PATCH Re: udp weirdness

[Chris Friesen]

kuznet@ms2.inr.ac.ru wrote:

> Feel free to implement. :-)

I may have to poke around...if nothing else I'll learn more about the 
networking code...

> What? If your applications is enough clever to handle ENOBUFS right,
> set IP_RECVERR and live in peace.

The original poster was complaining about messages being silently 
dropped when there is congestion.  If IP_RECVERR is turned on, would 
sendto() then return -1 so I know to try and read the error messages? 
I'm assuming I get ENOBUFS back in the ee_code field?  Or can I get away 
with reading errno and ignoring the error queue?

I've never used IP_RECVERR and there doesn't seem to be a lot of 
documentation about it.

Chris

Re: PATCH Re: udp weirdness

[kuznet]

Hello!

> > Feel free to implement. :-)
> 
> I may have to poke around...if nothing else I'll learn more about the 
> networking code...

It is difficult task, if possible at all.

The main obstacle is that we must not block after select() succeeded,
otherwise applications will lockup. Taking into account nature of datagram
services (and generally of networking services, where routes change et al.)
you do not know at time of select(), where the datagram will go.
So, blocking can be made only based on a criterium not depending on this.

We use sndbuf (like all the OSes). Actually, the problem with silent
losses is solved by tuning SO_SNDBUF. Though it is not a complete
solution (failing whith lots of senders), it solves all those bullshit
problems with silent losses. People just do not care about this, so
they get the thing which they deserve.


> dropped when there is congestion.  If IP_RECVERR is turned on, would 
> sendto() then return -1 so I know to try and read the error messages? 

Yes.


> I'm assuming I get ENOBUFS back in the ee_code field?  Or can I get away 
> with reading errno and ignoring the error queue?

No, error queue should be read. But not all the errors are queued,
only those which are supported asynchronously. ENOBUFS is still not.
F.e. when packet is dropped lately (some qdiscs do this, dropping already
queued packets to give place for another ones) ENOBUFS is not sent back.


>			     there doesn't seem to be a lot of 
> documentation about it.

Damn! (I'm sorry) It is documented _very_ well, thanks to Andi.
I really hate this mode when people do not worrying even to look to manpages
before pronouncing such statements.

man recvmsg
man ip

Alexey

Re: PATCH Re: udp weirdness

[Chris Friesen]

kuznet@ms2.inr.ac.ru wrote:

 > But not all the errors are queued,
> only those which are supported asynchronously. ENOBUFS is still not.
> F.e. when packet is dropped lately (some qdiscs do this, dropping already
> queued packets to give place for another ones) ENOBUFS is not sent back.

Hmm...so even with IP_RECVERR I may not be notified if the packet is 
dropped?


>>			     there doesn't seem to be a lot of 
>>documentation about it.
>>
> 
> Damn! (I'm sorry) It is documented _very_ well, thanks to Andi.
> I really hate this mode when people do not worrying even to look to manpages
> before pronouncing such statements.
> 
> man recvmsg
> man ip

Yes, I looked at both of those, and man udp as well.  My point about a 
lack of documentation was more related to corner cases and exactly what 
gets returned when.  The example above (where there are cases where no 
errors are returned even with IP_RECVERR turned on) is not mentioned 
anywhere in the documentation.

I write code for a telephony softswitch.  We are running a legacy app on 
top of an emulator on top of linux.  I want to ensure that my packets 
either a) got out onto the wire, or b) my app got an error message back 
explaining why the message didn't get onto the wire so that it can 
decide how to proceed.

Now my overall bandwidth requirements aren't too bad, but the traffic is 
bursty, with a batch of messages being sent in a tight sendto() loop.  I 
do NOT want messages to be silently dropped in the kernel with no error 
returned.

Chris

Re: PATCH Re: udp weirdness

[kuznet]

Hello!

> Hmm...so even with IP_RECVERR I may not be notified if the packet is 
> dropped?

Do you not jest occasionally? :-) "Even" sounds interesting, because
this kind of errors cannot be reported _principially_ without IP_RECVERR,
providing asynchronous error reporting.

Packet can be dropped, damaged _after_ they are successfully queued
to the device. And this need not to be emphasized in documentation
on IP_RECVERR, except for a section "FUTURE DEVELOPMENT".


> either a) got out onto the wire, or b) my app got an error message back 
> explaining why the message didn't get onto the wire so that it can 
> decide how to proceed.

Then use IP_RECVERR. It reports all the errors which are detectable.

Alexey

Re: PATCH Re: udp weirdness

[Ben Greear]

kuznet@ms2.inr.ac.ru wrote:
> Hello!
> 
> 
>>>Feel free to implement. :-)
>>
>>I may have to poke around...if nothing else I'll learn more about the 
>>networking code...
> 
> 
> It is difficult task, if possible at all.
> 
> The main obstacle is that we must not block after select() succeeded,
> otherwise applications will lockup. Taking into account nature of datagram
> services (and generally of networking services, where routes change et al.)
> you do not know at time of select(), where the datagram will go.
> So, blocking can be made only based on a criterium not depending on this.
> 
> We use sndbuf (like all the OSes). Actually, the problem with silent
> losses is solved by tuning SO_SNDBUF. Though it is not a complete
> solution (failing whith lots of senders), it solves all those bullshit
> problems with silent losses. People just do not care about this, so
> they get the thing which they deserve.

Changing the size of the sending buffer will make you able to drop fewer
packets in bursty behaviour, but it does not guarantee you anything, does
it?  For those of us wanting to write programs with very little slop in
their behaviour, 'most of the time' is not good enough.

I have not yet looked at the UDP code in detail, but it does appear we
can know if a NIC accepts the packet for transmit or not.  If it does
not, then just don't dequeue off of the send list, retry it next time.

I cannot see any logical reason that we cannot ensure that a packet is
not at least given to the driver when accepted by sendto.  As soon as I
get my pktgen and send-to-self code cleaned up, I am planning to start
working on making UDP reliably send packets, or return an error to
the calling code.  I will, of course, keep you informed if I actually
get something working...

Enjoy,
Ben


> 
> 
> 
>>dropped when there is congestion.  If IP_RECVERR is turned on, would 
>>sendto() then return -1 so I know to try and read the error messages? 
> 
> 
> Yes.
> 
> 
> 
>>I'm assuming I get ENOBUFS back in the ee_code field?  Or can I get away 
>>with reading errno and ignoring the error queue?
> 
> 
> No, error queue should be read. But not all the errors are queued,
> only those which are supported asynchronously. ENOBUFS is still not.
> F.e. when packet is dropped lately (some qdiscs do this, dropping already
> queued packets to give place for another ones) ENOBUFS is not sent back.
> 
> 
> 
>>			     there doesn't seem to be a lot of 
>>documentation about it.
> 
> 
> Damn! (I'm sorry) It is documented _very_ well, thanks to Andi.
> I really hate this mode when people do not worrying even to look to manpages
> before pronouncing such statements.
> 
> man recvmsg
> man ip
> 
> Alexey
> 


-- 
Ben Greear <greearb@candelatech.com>       <Ben_Greear AT excite.com>
President of Candela Technologies Inc      http://www.candelatech.com
ScryMUD:  http://scry.wanfear.com     http://scry.wanfear.com/~greear

Re: PATCH Re: udp weirdness

[Chris Friesen]

Ben Greear wrote:

> I cannot see any logical reason that we cannot ensure that a packet is
> not at least given to the driver when accepted by sendto.  As soon as I
> get my pktgen and send-to-self code cleaned up, I am planning to start
> working on making UDP reliably send packets, or return an error to
> the calling code.  I will, of course, keep you informed if I actually
> get something working...


That would be fantastic.

Chris

Re: PATCH Re: udp weirdness

[Chris Friesen]

kuznet@ms2.inr.ac.ru wrote:

>>Hmm...so even with IP_RECVERR I may not be notified if the packet is 
>>dropped?
>>
> 
> Do you not jest occasionally? :-) "Even" sounds interesting, because
> this kind of errors cannot be reported _principially_ without IP_RECVERR,
> providing asynchronous error reporting.

The point that I fundamentally need IP_RECVERR for async errors is 
interesting, I'll have to look into that.  Thanks for the pointer.

> Packet can be dropped, damaged _after_ they are successfully queued
> to the device. And this need not to be emphasized in documentation
> on IP_RECVERR, except for a section "FUTURE DEVELOPMENT".

This is certainly true, but I would hope that packets being damaged or 
dropped after being queued to the device would be a rare event.  It 
would still be good to have them reported though.

I understand the problems with regards to blocking sockets, since you 
can't block after the select() passes and select() doesn't know where 
the packet is going.  It seems that it should be pretty straightforward 
to enable ENOBUFS for nonblocking sockets though--am I missing something?

For now I'll play with SO_SNDBUF and IP_RECVERR.

Thanks,

Chris

Re: PATCH Re: udp weirdness

[jamal]

On Tue, 1 Oct 2002, Ben Greear wrote:

> get my pktgen and send-to-self code cleaned up, I am planning to start
> working on making UDP reliably send packets, or return an error to
> the calling code.  I will, of course, keep you informed if I actually
> get something working...

If you want realibility then thats what TCP is for.
I am curious why you would even want to retransmit a voice packet or
why a local drop should be treated any different from a remote/network
drop in a voice application ...

When you fail in sendto/sendmsg, errno is set to ENOBUFS as long as you
set IP_RECVERR in the socket options; you can also receive ICMP errors
as described in the manpages (use a msg_control buffer and call recvmsg
with MSG_ERRQUEUE).

BTW, a good sample of an app that makes good use of ENOBUFS to do
congestion control, IP_RECVERR and MSG_ERRQUEUE is the ping app in Alexeys
iputils package. Why did i not remember all this before chasing the
phantom with Eric is an indication i need to increase my cafeine
consumption.

cheers,
jamal

Re: PATCH Re: udp weirdness

[jamal]

On Tue, 1 Oct 2002, Chris Friesen wrote:

> to be silently dropped by the kernel because userspace is sending faster
> than they can get onto the wire during that tight loop.
>

So what happens when you find packets being dropped?
AFAIK, a dropped voice packet is as good as dead whether local or remote.

> Okay, so with IP_RECVERR set the case that Eric saw will not happen?  I
> mean that sendto() will return with -1 and errno set to ENOBUFS?

yes

cheers,
jamal

Re: PATCH Re: udp weirdness

[Chris Friesen]

jamal wrote:
> 
> On Tue, 1 Oct 2002, Ben Greear wrote:
> 
> 
>>get my pktgen and send-to-self code cleaned up, I am planning to start
>>working on making UDP reliably send packets, or return an error to
>>the calling code.  I will, of course, keep you informed if I actually
>>get something working...
>>
> 
> If you want realibility then thats what TCP is for.

There is a requirement to interwork with other equipment using UDP.

> I am curious why you would even want to retransmit a voice packet or
> why a local drop should be treated any different from a remote/network
> drop in a voice application ...

The legacy app generates batches of messages, and the emulator layer 
then sends them out in a tight loop on sendto().  I don't want packets 
to be silently dropped by the kernel because userspace is sending faster 
than they can get onto the wire during that tight loop.

Eric's original testcase was a tight loop on sendto() resulting in 
userspace sending at a way higher rate than could be put onto the wire, 
so the kernel was silently dropping them.  This is exactly what I want 
to avoid.

> When you fail in sendto/sendmsg, errno is set to ENOBUFS as long as you
> set IP_RECVERR in the socket options; you can also receive ICMP errors
> as described in the manpages (use a msg_control buffer and call recvmsg
> with MSG_ERRQUEUE).

Okay, so with IP_RECVERR set the case that Eric saw will not happen?  I 
mean that sendto() will return with -1 and errno set to ENOBUFS?

> BTW, a good sample of an app that makes good use of ENOBUFS to do
> congestion control, IP_RECVERR and MSG_ERRQUEUE is the ping app in Alexeys
> iputils package. Why did i not remember all this before chasing the
> phantom with Eric is an indication i need to increase my cafeine
> consumption

I'll take a look and try things out.

Chris

Re: PATCH Re: udp weirdness

[Ben Greear]

jamal wrote:
> 
> On Tue, 1 Oct 2002, Ben Greear wrote:
> 
> 
>>get my pktgen and send-to-self code cleaned up, I am planning to start
>>working on making UDP reliably send packets, or return an error to
>>the calling code.  I will, of course, keep you informed if I actually
>>get something working...
> 
> 
> If you want realibility then thats what TCP is for.
> I am curious why you would even want to retransmit a voice packet or
> why a local drop should be treated any different from a remote/network
> drop in a voice application ...

Don't worry so much about why I want to do this, just assume that I do! :)
I have explained in the past, but since our needs are different, it does
not seem to impress upon anyone (I want to send at high speeds, and detect
every possible packet dropped by the network.  If my local machine drops
the packet, my detection of network-dropped packets is bogus...)

The reason a local packet drop is different, is because it can be different.
Any feedback of this nature that we can give to user-space can be used to
make any recovery or throttling decisions better.  I accept that I cannot
guarantee a UDP packet is received by it's intended target, but I do not
accept that the local machine cannot even guarantee that it has sent the
packet onto the network.

> 
> When you fail in sendto/sendmsg, errno is set to ENOBUFS as long as you
> set IP_RECVERR in the socket options; you can also receive ICMP errors
> as described in the manpages (use a msg_control buffer and call recvmsg
> with MSG_ERRQUEUE).

I will investigate the IP_RECVERR more closely, it may do just what I need.

Thanks,
Ben

> 
> BTW, a good sample of an app that makes good use of ENOBUFS to do
> congestion control, IP_RECVERR and MSG_ERRQUEUE is the ping app in Alexeys
> iputils package. Why did i not remember all this before chasing the
> phantom with Eric is an indication i need to increase my cafeine
> consumption.
> 
> cheers,
> jamal
> 


-- 
Ben Greear <greearb@candelatech.com>       <Ben_Greear AT excite.com>
President of Candela Technologies Inc      http://www.candelatech.com
ScryMUD:  http://scry.wanfear.com     http://scry.wanfear.com/~greear

Re: PATCH Re: udp weirdness

[Ben Greear]

jamal wrote:
> 
> On Tue, 1 Oct 2002, Chris Friesen wrote:
> 
> 
>>to be silently dropped by the kernel because userspace is sending faster
>>than they can get onto the wire during that tight loop.
>>
> 
> 
> So what happens when you find packets being dropped?
> AFAIK, a dropped voice packet is as good as dead whether local or remote.

If it is dropped locally, you can re-send in way less than 1 milisecond,
which is well within the realm of expected jitter.  You can also back off
for 5 miliseconds to alleviate congestion, which is still within acceptable
jitter.

If you think you sent it, but didn't actually send it, then it looks like
your network is shitting, when in fact your local machine is acting shitty.

Ben

> 
> 
>>Okay, so with IP_RECVERR set the case that Eric saw will not happen?  I
>>mean that sendto() will return with -1 and errno set to ENOBUFS?
> 
> 
> yes
> 
> cheers,
> jamal
> 


-- 
Ben Greear <greearb@candelatech.com>       <Ben_Greear AT excite.com>
President of Candela Technologies Inc      http://www.candelatech.com
ScryMUD:  http://scry.wanfear.com     http://scry.wanfear.com/~greear

Re: PATCH Re: udp weirdness

[Chris Friesen]

jamal wrote:
 >
 > On Tue, 1 Oct 2002, Chris Friesen wrote:
 >
 >> to be silently dropped by the kernel because userspace is sending
 >> faster than they can get onto the wire during that tight loop.

 > So what happens when you find packets being dropped? AFAIK, a
 > dropped voice packet is as good as dead whether local or remote.

We do have some leeway in terms of latency, and delayed leaving the box
is not the same as dropped.

We know that call processing messaging can only ever go out one ethernet
interface at a time, and we know that the call agent is guaranteed 90% 
of the userspace cpu time (scheduler changes).  We certify the box for a 
certain engineered throughput, so we know the average packets/sec value. 
We also have total knowledge/control over the other apps running on the 
box in question.

So really all I'm protecting against is from one single userspace app 
generating packets faster than the network can keep up.  As long as I 
get EAGAIN/EWOULDBLOCK/ENOBUFS on my non-blocking socket then I'll just 
try again until it succeeds or I get a more serious error.  So far this 
seems to be working nicely in 2.2, but we're just in the middle of a 
switch to 2.4 for the new release and I want to make sure I've got a 
handle on things.

Thanks for your help,

Chris

PS. I realize that this design is simplistic, but we are severely 
constrained by the legacy app running on the emulator.

Re: PATCH Re: udp weirdness

[Eric Lemoine]

> > I may have to poke around...if nothing else I'll learn more about the 
> > networking code...
> 
> It is difficult task, if possible at all.
> 
> The main obstacle is that we must not block after select() succeeded,
> otherwise applications will lockup. Taking into account nature of datagram
> services (and generally of networking services, where routes change et al.)
> you do not know at time of select(), where the datagram will go.
> So, blocking can be made only based on a criterium not depending on this.
> problems with silent losses. People just do not care about this, so
> they get the thing which they deserve.

Alexey,

Would you mind explaining a bit more why apps will lockup if we block
after select() succeeded. Or anyone?

Thx.

	Eric.

Re: PATCH Re: udp weirdness

[Chris Friesen]

Eric Lemoine wrote:

> Would you mind explaining a bit more why apps will lockup if we block
> after select() succeeded. Or anyone?

I suspect he means a temporary lockup.

There are many apps that use blocking sockets and rely on select() to 
tell them when they may write out to the socket without blocking.

If the write then blocks, the whole app is blocked.  Some of them 
probably wouldn't like this.

Was that it Alexey? Or is there something more that I missed?

Chris

Re: PATCH Re: udp weirdness

[Ben Greear]

Eric Lemoine wrote:
>>>I may have to poke around...if nothing else I'll learn more about the 
>>>networking code...
>>
>>It is difficult task, if possible at all.
>>
>>The main obstacle is that we must not block after select() succeeded,
>>otherwise applications will lockup. Taking into account nature of datagram
>>services (and generally of networking services, where routes change et al.)
>>you do not know at time of select(), where the datagram will go.
>>So, blocking can be made only based on a criterium not depending on this.
>>problems with silent losses. People just do not care about this, so
>>they get the thing which they deserve.
> 
> 
> Alexey,
> 
> Would you mind explaining a bit more why apps will lockup if we block
> after select() succeeded. Or anyone?

Actually, I'm more interested to know why we would **need** to block after
select has succeeded.  It would seem to me that select is busted in this
case.  For the case of a very large UDP packet and a small send buffer,
select gets confused, but at least when the send buffer is > 128k, it should
be right...


> 
> Thx.
> 
> 	Eric.
> 


-- 
Ben Greear <greearb@candelatech.com>       <Ben_Greear AT excite.com>
President of Candela Technologies Inc      http://www.candelatech.com
ScryMUD:  http://scry.wanfear.com     http://scry.wanfear.com/~greear

Re: PATCH Re: udp weirdness

[Eric Lemoine]

> > >It is difficult task, if possible at all.
> > >
> > >The main obstacle is that we must not block after select() succeeded,
> > >otherwise applications will lockup. Taking into account nature of datagram
> > >services (and generally of networking services, where routes change et 
> > >al.)
> > >you do not know at time of select(), where the datagram will go.
> > >So, blocking can be made only based on a criterium not depending on this.
> > >problems with silent losses. People just do not care about this, so
> > >they get the thing which they deserve.
> >
> >Alexey,
> >
> >Would you mind explaining a bit more why apps will lockup if we block
> >after select() succeeded. Or anyone?
> 
> Actually, I'm more interested to know why we would **need** to block after
> select has succeeded.  It would seem to me that select is busted in this
> case.  For the case of a very large UDP packet and a small send buffer,
> select gets confused, but at least when the send buffer is > 128k, it should
> be right...

With the current impl. process calling sendto() doesn't go to sleep 
after select() has succeeded. My question applied in the context where 
process goes to sleep when the qdisc queue is overflowed.

-- 
Eric

Re: PATCH Re: udp weirdness

[kuznet]

Hello!

> > >Would you mind explaining a bit more why apps will lockup if we block
> > >after select() succeeded. Or anyone?

Because queue overflow is persistant condition. Do you want your routing
daemon went to coma when sending an update to dead ppp link?

But, actually, it is simply required by standards. No blocking after
select success is definition of select, in fact.

Alexey