missing icmp redirects

missing icmp redirects

[Per Jessen]

See complete description here:

https://bugzilla.kernel.org/show_bug.cgi?id=72561

Summary - some ICMP redirects seem to be missing.

Alan suggests I gather some traces, but I'm am unsure of what to trace. 
Suggestions please?


Thanks
Per Jessen

RE: missing icmp redirects

[Lukas Tribus]

> See complete description here:
>
> https://bugzilla.kernel.org/show_bug.cgi?id=72561
>
> Summary - some ICMP redirects seem to be missing.
>
> Alan suggests I gather some traces, but I'm am unsure of what to trace.
> Suggestions please?

Reproduce the actual issue and post a simplified testcase, without
transparent proxying, etc.

 		 	   		  

Re: missing icmp redirects

[Per Jessen]

Lukas Tribus wrote:
>> See complete description here:
>>
>> https://bugzilla.kernel.org/show_bug.cgi?id=72561
>>
>> Summary - some ICMP redirects seem to be missing.
>>
>> Alan suggests I gather some traces, but I'm am unsure of what to trace.
>> Suggestions please?
> 
> Reproduce the actual issue and post a simplified testcase, without
> transparent proxying, etc.

Okay, I have devised a way to reproduce the problem:

I have a test setup of three machines:

“client”, “firewall” and “server”. All on the same network.

Client:
Set up default route via “firewall”.

Server:
Assign 10.232.1.1-2-3-4-...-15/24 to an interface.
Run a tcp echo service (port 7).

Firewall:
Create routing table “test99”.  /etc/iproute2/rt_tables.
ip route add default via <server> dev eth0 table test99
ip rule add fwmark 5 table test99
iptables -A PREROUTING -t mangle -i eth0 -p tcp --dport 7 -j MARK --set-mark 5

(this setup is what will produce the ICMP redirects).

On “firewall”, run tcpdump to document (missing) redirects:
tcpdump -n -i eth0 proto \\icmp

On “client”, create some test input:

cat <<XXX >test.input
klop
alpha
nothing
tagi
line1
line2
line3
line4
XXX

Create a script
cat <<XXX >doit
for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
do
     telnet 10.232.1.$i 7 <test.input &
done

When you run “sh doit”, all of the telnet requests to 10.232.1.x should be 
redirected, but the tcpdump running on "firewall" will only show some of them.

Re: missing icmp redirects

[Dmitry Popov]

On Tue, 10 Jun 2014 20:10:23 +0200
Per Jessen <per@computer.org> wrote:

> Lukas Tribus wrote:
> >> See complete description here:
> >>
> >> https://bugzilla.kernel.org/show_bug.cgi?id=72561
> >>
> >> Summary - some ICMP redirects seem to be missing.
> >>
> Okay, I have devised a way to reproduce the problem:
> 
> ...

It's probably not a bug, but intended behavior. See ip_rt_send_redirect in 
net/ipv4/route.c. The kernel has a built-in per-source-address limiter for icmp
redirects. 

You can disable it for example with 
echo -1 > /proc/sys/net/ipv4/route/redirect_silence
Though I am not sure if it is ok in wild networks.

Re: missing icmp redirects

[Per Jessen]

Dmitry Popov wrote:
> On Tue, 10 Jun 2014 20:10:23 +0200
> Per Jessen <per@computer.org> wrote:
> 
>> Lukas Tribus wrote:
>>>> See complete description here:
>>>>
>>>> https://bugzilla.kernel.org/show_bug.cgi?id=72561
>>>>
>>>> Summary - some ICMP redirects seem to be missing.
>>>>
>> Okay, I have devised a way to reproduce the problem:
>>
>> ...
> 
> It's probably not a bug, but intended behavior. See ip_rt_send_redirect in 
> net/ipv4/route.c. The kernel has a built-in per-source-address limiter for icmp
> redirects. 

Hi Dmitry

Any possibility that this limiter may have been inactive/faulty in e.g. 2.6.34? 
  We only hit this issue when we upgraded the firewall to a newer openSUSE (for 
ebtables) which came with kernel 3.11.6, but prior to that we had no issue with 
missing redirects.

> You can disable it for example with 
> echo -1 > /proc/sys/net/ipv4/route/redirect_silence
> Though I am not sure if it is ok in wild networks.

Okay, that worked on the first attempt.
Should I also be looking at the other redirect settings?

/proc/sys/net/ipv4/route/redirect_load
/proc/sys/net/ipv4/route/redirect_number



thanks
Per

Re: missing icmp redirects

[Dmitry Popov]

On Wed, 11 Jun 2014 09:19:22 +0200
Per Jessen <per@computer.org> wrote:

> Dmitry Popov wrote:
> > On Tue, 10 Jun 2014 20:10:23 +0200
> > Per Jessen <per@computer.org> wrote:
> > 
> >> Lukas Tribus wrote:
> >>>> See complete description here:
> >>>>
> >>>> https://bugzilla.kernel.org/show_bug.cgi?id=72561
> >>>>
> >>>> Summary - some ICMP redirects seem to be missing.
> >>>>
> > It's probably not a bug, but intended behavior. See ip_rt_send_redirect in 
> > net/ipv4/route.c. The kernel has a built-in per-source-address limiter for icmp
> > redirects. 
> Any possibility that this limiter may have been inactive/faulty in e.g. 2.6.34? 

Hi!

This limiter worked per-destination-address before kernel v3.6. More precisely,
commit 1d861aa4b3(inet: Minimize use of cached route inetpeer) changed that
behavior. I don't know the reasons behind it (both approaches have pros and 
cons), I think you should ask David if you're curious.

* upd: it's not a limiter for icmp redirects only, it is used for other icmp
errors too. 

> > You can disable it for example with 
> > echo -1 > /proc/sys/net/ipv4/route/redirect_silence
> > Though I am not sure if it is ok in wild networks.
> 
> Okay, that worked on the first attempt.
> Should I also be looking at the other redirect settings?
> 
> /proc/sys/net/ipv4/route/redirect_load
> /proc/sys/net/ipv4/route/redirect_number

redirect_silence is enough, you can look at the algorithm (it's simple enough)
in net/ipv4/route.c:ip_rt_send_redirect if you're curious how it works and what 
those settings mean.