Problems Receiving UDP Broadcasts on Some Versions of Linux

Problems Receiving UDP Broadcasts on Some Versions of Linux

[Curt Sampson]

I'm posting this here because it smells as if it may be something
a where a kernel network code developer might just say, "oh yeah,
obviously your problem is this."

I'm not subscribed to this list, so please cc me on any replies. (The
Mail-followup-to header may do this automatically for you.)

The short summary is that on two very similar servers on the same
network, an application that listens for broadcast UDP packets on a
given port works fine on the Ubuntu 8.04/Linux 2.6.24-26-server box but
does not receive the packets on the Ubuntu 9.04/Linux 2.6.28-17-server
box. I have confirmed that tcpdump sees identical packets being received
on both hosts.

Did something change between these two kernel revisions that I need to
listen differently?

Full details, including sample code, are available at:

  http://stackoverflow.com/questions/2130563/udp-broadcast-reception-problems-with-ubuntu-9-04-but-not-8-04

Thanks for any help you can give me. Even if you're not sure, but have
clues I should consider following up, feel free to e-mail me. I'm
also happy to take pointers to kernel code I should read, if there's
enlightenment there.

cjs
-- 
Curt Sampson         <cjs@cynic.net>         +81 90 7737 2974
             http://www.starling-software.com
The power of accurate observation is commonly called cynicism
by those who have not got it.    --George Bernard Shaw

Re: Problems Receiving UDP Broadcasts on Some Versions of Linux

[Eric Dumazet]

Le 25/01/2010 08:27, Curt Sampson a écrit :
> I'm posting this here because it smells as if it may be something
> a where a kernel network code developer might just say, "oh yeah,
> obviously your problem is this."
> 
> I'm not subscribed to this list, so please cc me on any replies. (The
> Mail-followup-to header may do this automatically for you.)
> 
> The short summary is that on two very similar servers on the same
> network, an application that listens for broadcast UDP packets on a
> given port works fine on the Ubuntu 8.04/Linux 2.6.24-26-server box but
> does not receive the packets on the Ubuntu 9.04/Linux 2.6.28-17-server
> box. I have confirmed that tcpdump sees identical packets being received
> on both hosts.
> 
> Did something change between these two kernel revisions that I need to
> listen differently?
> 
> Full details, including sample code, are available at:
> 
>   http://stackoverflow.com/questions/2130563/udp-broadcast-reception-problems-with-ubuntu-9-04-but-not-8-04
> 
> Thanks for any help you can give me. Even if you're not sure, but have
> clues I should consider following up, feel free to e-mail me. I'm
> also happy to take pointers to kernel code I should read, if there's
> enlightenment there.
> 
> cjs

Hi Curt

Could you post a sample of tcpdump trafic you receive on the 9.04 host, and your network config ?
"ifconfig -a"

Re: Problems Receiving UDP Broadcasts on Some Versions of Linux

[Curt Sampson]

On 2010-01-25 09:10 +0100 (Mon), Eric Dumazet wrote:

> > The short summary is that on two very similar servers on the same
> > network, an application that listens for broadcast UDP packets on a
> > given port works fine on the Ubuntu 8.04/Linux 2.6.24-26-server box but
> > does not receive the packets on the Ubuntu 9.04/Linux 2.6.28-17-server
> > box.
>
> Could you post a sample of tcpdump trafic you receive on the 9.04
> host, and your network config ? "ifconfig -a"

I've added both to the end of the StackOverflow entry:

  http://stackoverflow.com/questions/2130563/udp-broadcast-reception-problems-with-ubuntu-9-04-but-not-8-04

Again, please cc me on any responses; I'm on neither the linux-kernel
nor netdev lists.

cjs
-- 
Curt Sampson         <cjs@cynic.net>         +81 90 7737 2974
             http://www.starling-software.com
The power of accurate observation is commonly called cynicism
by those who have not got it.    --George Bernard Shaw

Re: Problems Receiving UDP Broadcasts on Some Versions of Linux

[Eric Dumazet]

Le mercredi 27 janvier 2010 à 09:37 +0900, Curt Sampson a écrit :
> On 2010-01-25 09:10 +0100 (Mon), Eric Dumazet wrote:
> 
> > > The short summary is that on two very similar servers on the same
> > > network, an application that listens for broadcast UDP packets on a
> > > given port works fine on the Ubuntu 8.04/Linux 2.6.24-26-server box but
> > > does not receive the packets on the Ubuntu 9.04/Linux 2.6.28-17-server
> > > box.
> >
> > Could you post a sample of tcpdump trafic you receive on the 9.04
> > host, and your network config ? "ifconfig -a"
> 
> I've added both to the end of the StackOverflow entry:
> 
>   http://stackoverflow.com/questions/2130563/udp-broadcast-reception-problems-with-ubuntu-9-04-but-not-8-04
> 
> Again, please cc me on any responses; I'm on neither the linux-kernel
> nor netdev lists.
> 
> cjs

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1362 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:30:48:d9:38:da brd ff:ff:ff:ff:ff:ff
3: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 00:1b:21:36:19:fd brd ff:ff:ff:ff:ff:ff
4: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 100
    link/ether 00:30:48:d9:38:db brd ff:ff:ff:ff:ff:ff
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1362 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:30:48:d9:38:da brd ff:ff:ff:ff:ff:ff
    inet 192.168.228.132/28 brd 192.168.228.143 scope global eth0
    inet6 fe80::230:48ff:fed9:38da/64 scope link 
       valid_lft forever preferred_lft forever
3: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 00:1b:21:36:19:fd brd ff:ff:ff:ff:ff:ff
4: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 100
    link/ether 00:30:48:d9:38:db brd ff:ff:ff:ff:ff:ff
    inet 172.40.4.132/24 brd 172.40.4.255 scope global eth1
    inet6 fe80::230:48ff:fed9:38db/64 scope link 
       valid_lft forever preferred_lft forever

And you receive a frame on eth1 with src ip address = 192.166.1.120

I suspect your kernel drops this frame because of asymetric routing

Check "ip route get 192.166.1.120" and if answer is directed to another interface,
this may explain the problem.

Solution : tweak rp_filter settings.

Documentation/networking/ip-sysctl.txt

rp_filter - INTEGER
        0 - No source validation.
        1 - Strict mode as defined in RFC3704 Strict Reverse Path
            Each incoming packet is tested against the FIB and if the
interface
            is not the best reverse path the packet check will fail.
            By default failed packets are discarded.
        2 - Loose mode as defined in RFC3704 Loose Reverse Path
            Each incoming packet's source address is also tested against
the FIB
            and if the source address is not reachable via any interface
            the packet check will fail.

        Current recommended practice in RFC3704 is to enable strict mode
        to prevent IP spoofing from DDos attacks. If using asymmetric
routing
        or other complicated routing, then loose mode is recommended.

        The max value from conf/{all,interface}/rp_filter is used
        when doing source validation on the {interface}.

        Default value is 0. Note that some distributions enable it
        in startup scripts.


I suspect Ubuntu 9.04 enables rp_filter (set to one)

(note 2 value (loose mode) might be not available yet on 2.6.28 kernels)

Check/change /etc/sysctl.conf settings

Re: Problems Receiving UDP Broadcasts on Some Versions of Linux

[Curt Sampson]

On 2010-01-27 04:56 +0100 (Wed), Eric Dumazet wrote:

> And you receive a frame on eth1 with src ip address = 192.166.1.120
> 
> I suspect your kernel drops this frame because of asymetric routing
> 
> Check "ip route get 192.166.1.120" and if answer is directed to another interface,
> this may explain the problem.

Indeed, you are absolutely correct. I've further verified that this is
the issue by having the 8.04 host generate broadcast packets on eth0 and
eth1, and the "malfunctioning" 9.04 host receives them correctly in both
cases.

I ought to have known, looking at that patently bogus source address
(there's no way I should be seeing anything to 255.255.255.255 that has
a source address from a different network), to start looking at these
sorts

> Solution : tweak rp_filter settings.

And indeed, that worked. (I've added the old settings to the
StackOverflow page.) Thank you very much!

Let me know (by private e-mail) if you're going to add an answer to my
question on StackOverflow, so I can give you the reputation points.
Otherwise I'll add the answer myself. The URL, again, is:

  http://stackoverflow.com/questions/2130563/udp-broadcast-reception-problems-with-ubuntu-9-04-but-not-8-04

cjs
-- 
Curt Sampson         <cjs@cynic.net>         +81 90 7737 2974
             http://www.starling-software.com
The power of accurate observation is commonly called cynicism
by those who have not got it.    --George Bernard Shaw