[LARTC] Re: "dst cache overflow" messages and crash

[LARTC] Re: "dst cache overflow" messages and crash

[Frédéric Massot]

Ivan Vladimirov wrote:
> Frédéric Massot wrote:
>> Hi,
[...]
>> gc_thresh:4096
>> gc_timeout:300
>> max_delay:10
>> max_size:65536
>> min_adv_mss:256
>> min_delay:2
>> min_pmtu:552
>> mtu_expires:600
>> redirect_load:5
>> redirect_number:9
>> redirect_silence:5120
>> secret_interval:600
>>
>> I can increase the maximum size of the cache, but that will do nothing
>> but delay the crash.
>>
>> Can you help me?
>>
>> Regards.
> max_sizee536
> is to low increase size to 256k

Hi,

Thank you for your answer.

This change prevents the server crach or does nothing but delay it?

Regards.
-- 
=======================
|              FRÉDÉRIC MASSOT               |
|     http://www.juliana-multimedia.com      |
|   mailto:frederic@juliana-multimedia.com   |
=============Þbian=GNU/Linux=_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Re: "dst cache overflow" messages and crash

[Ivan Vladimirov]

Frédéric Massot wrote:
> Ivan Vladimirov wrote:
>> Frédéric Massot wrote:
>>> Hi,
> [...]
>>> gc_thresh:4096
>>> gc_timeout:300
>>> max_delay:10
>>> max_size:65536
>>> min_adv_mss:256
>>> min_delay:2
>>> min_pmtu:552
>>> mtu_expires:600
>>> redirect_load:5
>>> redirect_number:9
>>> redirect_silence:5120
>>> secret_interval:600
>>>
>>> I can increase the maximum size of the cache, but that will do nothing
>>> but delay the crash.
>>>
>>> Can you help me?
>>>
>>> Regards.
>> max_sizee536
>> is to low increase size to 256k
>
> Hi,
>
> Thank you for your answer.
>
> This change prevents the server crach or does nothing but delay it?
>
> Regards.
This change prevents server crash but you need careful tweaking of this
parameter
use slabtop to see how much new objects in dstcache you have over 5min
Also value of this parameter depends on type of traffic you have and
number of destination routes which server have to cache.

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

[LARTC] Re: "dst cache overflow" messages and crash

[Frédéric Massot]

Ivan Vladimirov wrote:
> Frédéric Massot wrote:
>> Hi,
>>
>> I regularly have errors (kernel: dst cache overflow) and crash of a
>> firewall under Linux 2.6.17 and the route patch from Julian Anastasov.
>>
>> With rtstat I see that the route cache size increases regularly without
>> never decreasing.
>>
>> I have this parameters:
>> fw:/proc/sys/net/ipv4/route# grep . *
>> error_burst:1250
>> error_cost:250
>> gc_elasticity:15
>> gc_interval:60
>> gc_min_interval:0
>> gc_min_interval_ms:500
>> gc_thresh:4096
>> gc_timeout:300
>> max_delay:10
>> max_size:65536
>> min_adv_mss:256
>> min_delay:2
>> min_pmtu:552
>> mtu_expires:600
>> redirect_load:5
>> redirect_number:9
>> redirect_silence:5120
>> secret_interval:600
>>
>> I can increase the maximum size of the cache, but that will do nothing
>> but delay the crash.
>>
>> Can you help me?
>>
>> Regards.
> max_sizee536
> is to low increase size to 256k

Hi,

I supervised the system during a few weeks with slabtop and rtstat.

What I could see, it is that the ip_dst_cache cache grow without never 
being cleaned by the garbage collector.

At the end of a few days, the traffic is slowed down and the customer 
restart the firewall. When the cache reaches its maximum value there is 
the error message (kernel: dst cache overflow) and the traffic is really 
disturbed.

I use the Linux kernel 2.6.17 and the route patch from Julian Anastasov.

- The bug comes from the kernel or the patch?

- Do you know if this bug were corrected in the new versions of the kernel?

Regards.
-- 
=======================
|              FRÉDÉRIC MASSOT               |
|     http://www.juliana-multimedia.com      |
|   mailto:frederic@juliana-multimedia.com   |
=============Þbian=GNU/Linux=
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Re: "dst cache overflow" messages and crash

[Ivan Vladimirov]

Patches from Julian Anastasov works only as he mentioned for 2.4 series
of kernels
His patches are untested for 2.6 kernels
2.6.17 kernel das not suppose to have this bug cos it was fixed earlier
in 2.6.16
My advise is to switch from 2.6.17 to 2.6.16.29 or lather and avoid
patches from Julian
there are other ways to perform same task without patches.
 
Frédéric Massot wrote:
> Ivan Vladimirov wrote:
>> Frédéric Massot wrote:
>>> Hi,
>>>
>>> I regularly have errors (kernel: dst cache overflow) and crash of a
>>> firewall under Linux 2.6.17 and the route patch from Julian Anastasov.
>>>
>>> With rtstat I see that the route cache size increases regularly without
>>> never decreasing.
>>>
>>> I have this parameters:
>>> fw:/proc/sys/net/ipv4/route# grep . *
>>> error_burst:1250
>>> error_cost:250
>>> gc_elasticity:15
>>> gc_interval:60
>>> gc_min_interval:0
>>> gc_min_interval_ms:500
>>> gc_thresh:4096
>>> gc_timeout:300
>>> max_delay:10
>>> max_size:65536
>>> min_adv_mss:256
>>> min_delay:2
>>> min_pmtu:552
>>> mtu_expires:600
>>> redirect_load:5
>>> redirect_number:9
>>> redirect_silence:5120
>>> secret_interval:600
>>>
>>> I can increase the maximum size of the cache, but that will do nothing
>>> but delay the crash.
>>>
>>> Can you help me?
>>>
>>> Regards.
>> max_sizee536
>> is to low increase size to 256k
>
> Hi,
>
> I supervised the system during a few weeks with slabtop and rtstat.
>
> What I could see, it is that the ip_dst_cache cache grow without never
> being cleaned by the garbage collector.
>
> At the end of a few days, the traffic is slowed down and the customer
> restart the firewall. When the cache reaches its maximum value there
> is the error message (kernel: dst cache overflow) and the traffic is
> really disturbed.
>
> I use the Linux kernel 2.6.17 and the route patch from Julian Anastasov.
>
> - The bug comes from the kernel or the patch?
>
> - Do you know if this bug were corrected in the new versions of the
> kernel?
>
> Regards.

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Re: "dst cache overflow" messages and crash

[Julian Anastasov]

	Hello,

On Thu, 22 Mar 2007, [ISO-8859-15] Frédéric Massot wrote:

> >> I regularly have errors (kernel: dst cache overflow) and crash of a
> >> firewall under Linux 2.6.17 and the route patch from Julian Anastasov.

	I assume IP_ROUTE_MULTIPATH_CACHED is disabled. Do you have
BRIDGE_NETFILTER enabled/used?

> I use the Linux kernel 2.6.17 and the route patch from Julian Anastasov.
> 
> - The bug comes from the kernel or the patch?
> 
> - Do you know if this bug were corrected in the new versions of the kernel?

	It should be the patch that triggers the problem.

Regards

--
Julian Anastasov <ja@ssi.bg>
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Re: "dst cache overflow" messages and crash

[Frédéric Massot]

Julian Anastasov wrote:
> 	Hello,
> 
> On Thu, 22 Mar 2007, [ISO-8859-15] Frédéric Massot wrote:
> 
>>>> I regularly have errors (kernel: dst cache overflow) and crash of a
>>>> firewall under Linux 2.6.17 and the route patch from Julian Anastasov.
> 
> 	I assume IP_ROUTE_MULTIPATH_CACHED is disabled. Do you have
> BRIDGE_NETFILTER enabled/used?

Hi,

Thank you for your answer.

- IP_ROUTE_MULTIPATH_CACHED is not set
- BRIDGE_NETFILTER is set, but I do not use it.


Regards.
-- 
=======================
|              FRÉDÉRIC MASSOT               |
|     http://www.juliana-multimedia.com      |
|   mailto:frederic@juliana-multimedia.com   |
=============Þbian=GNU/Linux=_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Re: "dst cache overflow" messages and crash

[Julian Anastasov]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 792 bytes --]


	Hello,

On Fri, 23 Mar 2007, Frédéric Massot wrote:

> >>>> I regularly have errors (kernel: dst cache overflow) and crash of a
> >>>> firewall under Linux 2.6.17 and the route patch from Julian Anastasov.
> > 
> > 	I assume IP_ROUTE_MULTIPATH_CACHED is disabled. Do you have
> > BRIDGE_NETFILTER enabled/used?
> 
> - IP_ROUTE_MULTIPATH_CACHED is not set
> - BRIDGE_NETFILTER is set, but I do not use it.

	ok, then can you try the attached patch, it solves dst cache
problem for another user, may be it will help you too. This patch can
be used with or without routes patches. It makes sure we don't leak
dst entry in bridge-netfilter. If the patch does not help let me know
and we can add some printks to catch the problem.

Regards

--
Julian Anastasov <ja@ssi.bg>

[-- Attachment #2: brnf can leak dst --]
[-- Type: TEXT/PLAIN, Size: 697 bytes --]

	Bridge-Netfilter should be gentle when calling ip_route_input,
	skb->dst can be already set.

diff -urp v2.6.20/linux/net/bridge/br_netfilter.c linux/net/bridge/br_netfilter.c
--- v2.6.20/linux/net/bridge/br_netfilter.c	2007-02-11 01:06:29.000000000 +0200
+++ linux/net/bridge/br_netfilter.c	2007-03-23 01:07:40.000000000 +0200
@@ -269,6 +269,10 @@ static int br_nf_pre_routing_finish(stru
 	struct nf_bridge_info *nf_bridge = skb->nf_bridge;
 	int err;
 
+	/* Old skb->dst is not expected, it is lost in all cases */
+	dst_release(skb->dst);
+	skb->dst = NULL;
+
 	if (nf_bridge->mask & BRNF_PKT_TYPE) {
 		skb->pkt_type = PACKET_OTHERHOST;
 		nf_bridge->mask ^= BRNF_PKT_TYPE;

[-- Attachment #3: Type: text/plain, Size: 143 bytes --]

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Re: "dst cache overflow" messages and crash

[Frédéric Massot]

Julian Anastasov wrote:
> 	Hello,
> 
> On Fri, 23 Mar 2007, Frédéric Massot wrote:
> 
>>>>>> I regularly have errors (kernel: dst cache overflow) and crash of a
>>>>>> firewall under Linux 2.6.17 and the route patch from Julian Anastasov.
>>> 	I assume IP_ROUTE_MULTIPATH_CACHED is disabled. Do you have
>>> BRIDGE_NETFILTER enabled/used?
>> - IP_ROUTE_MULTIPATH_CACHED is not set
>> - BRIDGE_NETFILTER is set, but I do not use it.
> 
> 	ok, then can you try the attached patch, it solves dst cache
> problem for another user, may be it will help you too. This patch can
> be used with or without routes patches. It makes sure we don't leak
> dst entry in bridge-netfilter. If the patch does not help let me know
> and we can add some printks to catch the problem.

Hi,

Thank you for your answer, as your patch comes from the kernel 2.6.20, I 
installed this version of the kernel with the patch (routes-2.6.20-14.diff).

That made a little more than one week that I supervise and it cache is 
well cleaned regularly. All seems to be good.  :o)

Regards.
-- 
=======================
|              FRÉDÉRIC MASSOT               |
|     http://www.juliana-multimedia.com      |
|   mailto:frederic@juliana-multimedia.com   |
=============Þbian=GNU/Linux=_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc