Re: route cache flush??

[Stephen Clark <sclark46@earthlink.net>]

On 02/03/2012 10:05 AM, Eric Dumazet wrote:
> Le vendredi 03 février 2012 à 09:59 -0500, Stephen Clark a écrit :
>    
>> Hello,
>>
>> I have been beating my head against the wall for 2 days trying to
>> figure why when I change a route and do a "ip route flush cache"
>> it still takes up to a minute for packets to start using the new
>> route.
>>
>> Is there a step I am missing?
>>
>> kernel is 2.6.32
>>
>>      
> Nothing comes to mind, please share more information ?
>
>    
Sure

I have a box "A" with 2 interfaces ips 2.2.2.1 and 3.3.3.1 going to 
another box "B" that has 3 nics 2.2.2.254
3.3.3.254 and 1.1.1.254 there is a third box "C" with ip 1.1.1.1. Box B 
is to simulate the net.

I have 2 vpns on box A going to box C. The default rt on A is 
2.2.2.254.  If I don't do anything else routing wise
esp packet originated on 1.1.1.1 come 3.3.3.1 on box A but the response 
packet from 3.3.3.1 goes out the
default route. This works OK in the lab but in the field the isps 
generally drop packet that have a source address
that doesn't match their subnet.

So I have found on the net how to set up a simple rule to route packets 
with src address 3.3.3.1 back out that interface.
$ ip r s
2.2.2.0/24 dev eth1  proto kernel  scope link  src 2.2.2.1
3.3.3.0/24 dev eth2  proto kernel  scope link  src 3.3.3.1
10.0.128.0/17 dev eth0  proto kernel  scope link  src 10.0.133.22
default via 2.2.2.254 dev eth1
L703103:~
$ ip r s table second
default via 3.3.3.254 dev eth2  src 3.3.3.1
L703103:~
$ ip rule list
0:      from all lookup local
200:    from 3.3.3.1 lookup second
32766:  from all lookup main
32767:  from all lookup default

So I run a script to disable the src route an flush the cache.$ 
date;sudo /usr/local/pgsql/storeproc/programs/src_rt_off.sh
Fri Feb  3 10:20:16 EST 2012  --- notice the time.
++ id -un
+ '[' root '!=' root ']'
+ /sbin/ip route delete default table second
+ /sbin/ip rule delete table second
+ /sbin/ip route flush cache
+ /sbin/ip rule list
0:      from all lookup local
32766:  from all lookup main
32767:  from all lookup default


This is a tcpdump of the interface with 3.3.3.1 ip address - I ran the 
script at 10:20:16
Notice how long before response quit going out this interface and switch 
to the default
route. I see similar behavior when I reinstall the src route and flush 
the cache.

10:20:19.102448 IP 3.3.3.1 > 1.1.1.1: ESP(spi=0x05690af9,seq=0x24e), 
length 116
10:20:19.103221 IP 1.1.1.1 > 3.3.3.1: ESP(spi=0x021bd1a9,seq=0x24e), 
length 116
10:20:19.498523 IP 1.1.1.1 > 3.3.3.1: ESP(spi=0x021bd1a9,seq=0x24f), 
length 116
10:20:19.498701 IP 3.3.3.1 > 1.1.1.1: ESP(spi=0x05690af9,seq=0x24f), 
length 116
10:20:30.704175 IP 1.1.1.1 > 3.3.3.1: ESP(spi=0x021bd1a9,seq=0x250), 
length 116
10:20:30.704357 IP 3.3.3.1 > 1.1.1.1: ESP(spi=0x05690af9,seq=0x250), 
length 116
10:20:34.217349 IP 3.3.3.1 > 1.1.1.1: ESP(spi=0x05690af9,seq=0x251), 
length 116
10:20:34.218150 IP 1.1.1.1 > 3.3.3.1: ESP(spi=0x021bd1a9,seq=0x251), 
length 116
10:20:34.333011 IP 3.3.3.1 > 1.1.1.1: ESP(spi=0x05690af9,seq=0x252), 
length 116
10:20:34.333795 IP 1.1.1.1 > 3.3.3.1: ESP(spi=0x021bd1a9,seq=0x252), 
length 116
10:20:40.826996 IP 3.3.3.1 > 1.1.1.1: ESP(spi=0x05690af9,seq=0x253), 
length 116
10:20:40.827775 IP 1.1.1.1 > 3.3.3.1: ESP(spi=0x021bd1a9,seq=0x253), 
length 116
10:20:41.412308 IP 3.3.3.1 > 1.1.1.1: ESP(spi=0x05690af9,seq=0x254), 
length 116
10:20:41.413081 IP 1.1.1.1 > 3.3.3.1: ESP(spi=0x021bd1a9,seq=0x254), 
length 116
10:20:41.910528 IP 1.1.1.1 > 3.3.3.1: ESP(spi=0x021bd1a9,seq=0x255), 
length 116
10:20:41.910711 IP 3.3.3.1 > 1.1.1.1: ESP(spi=0x05690af9,seq=0x255), 
length 116
10:20:42.413504 IP 3.3.3.1 > 1.1.1.1: ESP(spi=0x05690af9,seq=0x256), 
length 116
10:20:42.414260 IP 1.1.1.1 > 3.3.3.1: ESP(spi=0x021bd1a9,seq=0x256), 
length 116
10:20:43.413755 IP 3.3.3.1 > 1.1.1.1: ESP(spi=0x05690af9,seq=0x257), 
length 116
10:20:43.414510 IP 1.1.1.1 > 3.3.3.1: ESP(spi=0x021bd1a9,seq=0x257), 
length 116
10:20:44.413807 IP 3.3.3.1 > 1.1.1.1: ESP(spi=0x05690af9,seq=0x258), 
length 116
10:20:44.414560 IP 1.1.1.1 > 3.3.3.1: ESP(spi=0x021bd1a9,seq=0x258), 
length 116
10:20:49.468466 IP 3.3.3.1 > 1.1.1.1: ESP(spi=0x05690af9,seq=0x259), 
length 116
10:20:49.469267 IP 1.1.1.1 > 3.3.3.1: ESP(spi=0x021bd1a9,seq=0x259), 
length 116
10:20:49.576539 IP 3.3.3.1 > 1.1.1.1: ESP(spi=0x05690af9,seq=0x25a), 
length 116
10:20:49.577318 IP 1.1.1.1 > 3.3.3.1: ESP(spi=0x021bd1a9,seq=0x25a), 
length 116
10:20:53.116021 IP 1.1.1.1 > 3.3.3.1: ESP(spi=0x021bd1a9,seq=0x25b), 
length 116
10:20:53.116196 IP 3.3.3.1 > 1.1.1.1: ESP(spi=0x05690af9,seq=0x25b), 
length 116
10:21:04.320972 IP 1.1.1.1 > 3.3.3.1: ESP(spi=0x021bd1a9,seq=0x25c), 
length 116<<<<<<<
10:21:04.720342 IP 1.1.1.1 > 3.3.3.1: ESP(spi=0x021bd1a9,seq=0x25d), 
length 116
10:21:04.808653 IP 1.1.1.1 > 3.3.3.1: ESP(spi=0x021bd1a9,seq=0x25e), 
length 116
10:21:08.422372 IP 1.1.1.1 > 3.3.3.1: ESP(spi=0x021bd1a9,seq=0x25f), 
length 116
10:21:09.423730 IP 1.1.1.1 > 3.3.3.1: ESP(spi=0x021bd1a9,seq=0x260), 
length 116
10:21:10.424971 IP 1.1.1.1 > 3.3.3.1: ESP(spi=0x021bd1a9,seq=0x261), 
length 116
10:21:11.425469 IP 1.1.1.1 > 3.3.3.1: ESP(spi=0x021bd1a9,seq=0x262), 
length 116
10:21:15.525572 IP 1.1.1.1 > 3.3.3.1: ESP(spi=0x021bd1a9,seq=0x263), 
length 116










-- 

"They that give up essential liberty to obtain temporary safety,
deserve neither liberty nor safety."  (Ben Franklin)

"The course of history shows that as a government grows, liberty
decreases."  (Thomas Jefferson)