* Re: ip_conntrack_tcp Errors
2004-06-28 12:30 ` Dimitar Katerinski
@ 2004-06-28 12:45 ` Dimitar Katerinski
2004-06-28 12:55 ` Jozsef Kadlecsik
` (2 subsequent siblings)
3 siblings, 0 replies; 16+ messages in thread
From: Dimitar Katerinski @ 2004-06-28 12:45 UTC (permalink / raw)
To: netfilter
Hm, it seems that Jozsef Kadlecsik wrote tcp window tracking patch ;-)
So he should know better whats really going on, right Jozsef?
Regards,
Dimitar
--
"The only thing necessary for the triumph of evil is for good men to do nothing."
--Edmund Burke.
^ permalink raw reply [flat|nested] 16+ messages in thread* Re: ip_conntrack_tcp Errors
2004-06-28 12:30 ` Dimitar Katerinski
2004-06-28 12:45 ` Dimitar Katerinski
@ 2004-06-28 12:55 ` Jozsef Kadlecsik
2004-06-28 14:40 ` Evgeni Vachkov
2004-06-28 15:45 ` Evgeni Vachkov
2004-06-30 9:25 ` ip_conntrack_tcp Errors - backing a patch Evgeni Vachkov
3 siblings, 1 reply; 16+ messages in thread
From: Jozsef Kadlecsik @ 2004-06-28 12:55 UTC (permalink / raw)
To: Dimitar Katerinski; +Cc: netfilter
On Mon, 28 Jun 2004, Dimitar Katerinski wrote:
> > Is that a problem with conntrack and its tunning or I am missing some
> > patch? ...Or perhaps it is some other problem with other parts of the
> > kernel?
> It seems to me that you have applied the tcp window tracking patch from pom-ng.
> The problem is that the client and the server have done the first step of the
> three way handshake, and are in sync, but the firewall for some reason is not.
Sorry, but I have to contradict: there must be a connection in conntrack
which overlaps with the SYN/ACK packet detected. But why the connection
initiating SYN was not detected the same way? That is the question which
should be answered somehow.
> So it drops the SYN/ACK, and thus forcing the client to retransmit its SYN and
> initiate a new session (as descibed in the source code of the patch)
No, it does not drop the packet but ignores it as the log says. Look at
the lines in the source code a few lines below.
> My advice is if you have applied this patch, to remove it, and test the load on the
> firewall again.
Yep, that's a solution. And there won't be an answer explaining the case
then.
Best regards,
Jozsef
-
E-mail : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
H-1525 Budapest 114, POB. 49, Hungary
^ permalink raw reply [flat|nested] 16+ messages in thread* Re: ip_conntrack_tcp Errors
2004-06-28 12:55 ` Jozsef Kadlecsik
@ 2004-06-28 14:40 ` Evgeni Vachkov
2004-06-29 7:55 ` Jozsef Kadlecsik
0 siblings, 1 reply; 16+ messages in thread
From: Evgeni Vachkov @ 2004-06-28 14:40 UTC (permalink / raw)
To: netfilter
Jozsef,
I understand the need to identify the problem and I am more than willing
to help. It would be appreciated if you could adise me on the tcpdump
command switches to run, since I dont want to scan through thousands of
packets to identify the missing one. Also I am not an expert in tcpdump,
so I will need your help.
Note that this problem occurs only when ip_conntrack table reaches
arround 25000 connections. The software I am running is using ab (a
shell script incorporatring ab).
Regards,
Evgeni Vachkov
On Mon, 2004-06-28 at 13:55, Jozsef Kadlecsik wrote:
> On Mon, 28 Jun 2004, Dimitar Katerinski wrote:
>
> > > Is that a problem with conntrack and its tunning or I am missing some
> > > patch? ...Or perhaps it is some other problem with other parts of the
> > > kernel?
> > It seems to me that you have applied the tcp window tracking patch from pom-ng.
> > The problem is that the client and the server have done the first step of the
> > three way handshake, and are in sync, but the firewall for some reason is not.
>
> Sorry, but I have to contradict: there must be a connection in conntrack
> which overlaps with the SYN/ACK packet detected. But why the connection
> initiating SYN was not detected the same way? That is the question which
> should be answered somehow.
>
> > So it drops the SYN/ACK, and thus forcing the client to retransmit its SYN and
> > initiate a new session (as descibed in the source code of the patch)
>
> No, it does not drop the packet but ignores it as the log says. Look at
> the lines in the source code a few lines below.
>
> > My advice is if you have applied this patch, to remove it, and test the load on the
> > firewall again.
>
> Yep, that's a solution. And there won't be an answer explaining the case
> then.
>
> Best regards,
> Jozsef
> -
> E-mail : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu
> PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
> Address : KFKI Research Institute for Particle and Nuclear Physics
> H-1525 Budapest 114, POB. 49, Hungary
^ permalink raw reply [flat|nested] 16+ messages in thread* Re: ip_conntrack_tcp Errors
2004-06-28 14:40 ` Evgeni Vachkov
@ 2004-06-29 7:55 ` Jozsef Kadlecsik
2004-06-29 9:30 ` Evgeni Vachkov
0 siblings, 1 reply; 16+ messages in thread
From: Jozsef Kadlecsik @ 2004-06-29 7:55 UTC (permalink / raw)
To: Evgeni Vachkov; +Cc: netfilter
On 28 Jun 2004, Evgeni Vachkov wrote:
> I understand the need to identify the problem and I am more than willing
> to help. It would be appreciated if you could adise me on the tcpdump
> command switches to run, since I dont want to scan through thousands of
> packets to identify the missing one. Also I am not an expert in tcpdump,
> so I will need your help.
Assuming that eth0 faces with the client machine, issue the
following command before starting the test:
# tcpdump -i eth0 -w capture.log tcp
If the machine is short in disk space you can compress the log on the fly
instead:
# tcpdump -i eth0 -w - tcp | gzip -c9 > capture.log.gz
and then you can stop tcpdump by
# killall -INT tcpdump
> Note that this problem occurs only when ip_conntrack table reaches
> arround 25000 connections. The software I am running is using ab (a
> shell script incorporatring ab).
I assume you run Linux on the client machine running ab as well, don't
you? What it the output of
# cat /proc/sys/net/ipv4/ip_local_port_range
on that machine?
Best regards,
Jozsef
-
E-mail : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
H-1525 Budapest 114, POB. 49, Hungary
^ permalink raw reply [flat|nested] 16+ messages in thread* Re: ip_conntrack_tcp Errors
2004-06-29 7:55 ` Jozsef Kadlecsik
@ 2004-06-29 9:30 ` Evgeni Vachkov
2004-06-29 9:52 ` Jozsef Kadlecsik
0 siblings, 1 reply; 16+ messages in thread
From: Evgeni Vachkov @ 2004-06-29 9:30 UTC (permalink / raw)
To: netfilter
On Tue, 2004-06-29 at 08:55, Jozsef Kadlecsik wrote:
> Assuming that eth0 faces with the client machine, issue the
> following command before starting the test:
>
> # tcpdump -i eth0 -w capture.log tcp
>
I have completed the dump, file is quite big > 63MB. Is there a
possibility to limit the dump to the tested source and destination ip-s
only?
What shall I do with it next?
> I assume you run Linux on the client machine running ab as well, don't
> you?
>
Yes.
> What it the output of
>
> # cat /proc/sys/net/ipv4/ip_local_port_range
32768 61000
>
> on that machine?
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: ip_conntrack_tcp Errors
2004-06-29 9:30 ` Evgeni Vachkov
@ 2004-06-29 9:52 ` Jozsef Kadlecsik
0 siblings, 0 replies; 16+ messages in thread
From: Jozsef Kadlecsik @ 2004-06-29 9:52 UTC (permalink / raw)
To: Evgeni Vachkov; +Cc: netfilter
On 29 Jun 2004, Evgeni Vachkov wrote:
> On Tue, 2004-06-29 at 08:55, Jozsef Kadlecsik wrote:
> > Assuming that eth0 faces with the client machine, issue the
> > following command before starting the test:
> >
> > # tcpdump -i eth0 -w capture.log tcp
> >
> I have completed the dump, file is quite big > 63MB. Is there a
> possibility to limit the dump to the tested source and destination ip-s
> only?
I assumed there is no other traffic on the wire, just the test. You can
filter out the traffic between the two interesting parties by
tcpdump -r capture.log -w selected.log host a.b.c.d and host x.y.z.w
You can compress it as well. I'd need both the tcpdump log file and the
kernel log from the same time slice the tcpdump ran.
> > What it the output of
> >
> > # cat /proc/sys/net/ipv4/ip_local_port_range
>
> 32768 61000
That's 28232 ports available, over the 25000 connections you mentioned.
Best regards,
Jozsef
-
E-mail : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
H-1525 Budapest 114, POB. 49, Hungary
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: ip_conntrack_tcp Errors
2004-06-28 12:30 ` Dimitar Katerinski
2004-06-28 12:45 ` Dimitar Katerinski
2004-06-28 12:55 ` Jozsef Kadlecsik
@ 2004-06-28 15:45 ` Evgeni Vachkov
2004-06-29 7:46 ` Jozsef Kadlecsik
2004-06-30 9:25 ` ip_conntrack_tcp Errors - backing a patch Evgeni Vachkov
3 siblings, 1 reply; 16+ messages in thread
From: Evgeni Vachkov @ 2004-06-28 15:45 UTC (permalink / raw)
To: netfilter
> >
> > The ip_conntrack version is 2.1. kernel is v 2.4.26
> >
> > Is that a problem with conntrack and its tunning or I am missing some
> > patch? ...Or perhaps it is some other problem with other parts of the
> > kernel?
> It seems to me that you have applied the tcp window tracking patch from pom-ng.
> The problem is that the client and the server have done the first step of the
> three way handshake, and are in sync, but the firewall for some reason is not.
> So it drops the SYN/ACK, and thus forcing the client to retransmit its SYN and
> initiate a new session (as descibed in the source code of the patch)
>
> My advice is if you have applied this patch, to remove it, and test the load on the
> firewall again.
As far as I can figure out we are running with
patch-o-matic-ng-20040302. I can see that the latest is
patch-o-matic-ng-20040621. Is this problem present at the version we are
running and is the new version still having the window tracking issues?
Regards,
Evgeni Vachkov
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: ip_conntrack_tcp Errors
2004-06-28 15:45 ` Evgeni Vachkov
@ 2004-06-29 7:46 ` Jozsef Kadlecsik
0 siblings, 0 replies; 16+ messages in thread
From: Jozsef Kadlecsik @ 2004-06-29 7:46 UTC (permalink / raw)
To: Evgeni Vachkov; +Cc: netfilter
On 28 Jun 2004, Evgeni Vachkov wrote:
> > > patch? ...Or perhaps it is some other problem with other parts of the
> > > kernel?
> > It seems to me that you have applied the tcp window tracking patch from pom-ng.
> > The problem is that the client and the server have done the first step of the
> > three way handshake, and are in sync, but the firewall for some reason is not.
> > So it drops the SYN/ACK, and thus forcing the client to retransmit its SYN and
> > initiate a new session (as descibed in the source code of the patch)
> >
> > My advice is if you have applied this patch, to remove it, and test the load on the
> > firewall again.
>
> As far as I can figure out we are running with
> patch-o-matic-ng-20040302. I can see that the latest is
> patch-o-matic-ng-20040621. Is this problem present at the version we are
> running and is the new version still having the window tracking issues?
SACK support was added to the patch in patch-o-matic-ng-20040621 with some
minor fixes, compared to patch-o-matic-ng-20040302. It does not hurt to
upgrade but I don't expect that it'd behave differently in your test.
Best regards,
Jozsef
-
E-mail : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
H-1525 Budapest 114, POB. 49, Hungary
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: ip_conntrack_tcp Errors - backing a patch
2004-06-28 12:30 ` Dimitar Katerinski
` (2 preceding siblings ...)
2004-06-28 15:45 ` Evgeni Vachkov
@ 2004-06-30 9:25 ` Evgeni Vachkov
2004-06-30 12:58 ` Jozsef Kadlecsik
3 siblings, 1 reply; 16+ messages in thread
From: Evgeni Vachkov @ 2004-06-30 9:25 UTC (permalink / raw)
To: netfilter
On Mon, 2004-06-28 at 13:30, Dimitar Katerinski wrote:
> > Is that a problem with conntrack and its tunning or I am missing some
> > patch? ...Or perhaps it is some other problem with other parts of the
> > kernel?
> It seems to me that you have applied the tcp window tracking patch from pom-ng.
> The problem is that the client and the server have done the first step of the
> three way handshake, and are in sync, but the firewall for some reason is not.
> So it drops the SYN/ACK, and thus forcing the client to retransmit its SYN and
> initiate a new session (as descibed in the source code of the patch)
>
> My advice is if you have applied this patch, to remove it, and test the load on the
> firewall again.
> >
I would like to remove the patch, but unfortunately the person who
compiled last time didnt leave the code.
So....
Would it be possible to compile the relevent modules only (is it
conntrack only?) from scratch and to insmod them into the kernel?
Or there are parts of the kernel that I would need to recompile?
In any case, what code do I need to download and compile and where do I
find the relevant instructions?
In ideal situation I'd like to have a few versions of the conntrack (and
other related?) and be able to insmod them on the fly.
Also can I just rmmod the conntrack module or I need to unload iptables
and all the rest before I can insmod the new ones?
Many thanks,
Evgeni
^ permalink raw reply [flat|nested] 16+ messages in thread* Re: ip_conntrack_tcp Errors - backing a patch
2004-06-30 9:25 ` ip_conntrack_tcp Errors - backing a patch Evgeni Vachkov
@ 2004-06-30 12:58 ` Jozsef Kadlecsik
0 siblings, 0 replies; 16+ messages in thread
From: Jozsef Kadlecsik @ 2004-06-30 12:58 UTC (permalink / raw)
To: Evgeni Vachkov; +Cc: netfilter
On 30 Jun 2004, Evgeni Vachkov wrote:
> On Mon, 2004-06-28 at 13:30, Dimitar Katerinski wrote:
> > > Is that a problem with conntrack and its tunning or I am missing some
> > > patch? ...Or perhaps it is some other problem with other parts of the
> > > kernel?
> > My advice is if you have applied this patch, to remove it, and test the load on the
> > firewall again.
>
> I would like to remove the patch, but unfortunately the person who
> compiled last time didnt leave the code.
That's bad: with pom-ng, you can remove installed patches as well.
> Would it be possible to compile the relevent modules only (is it
> conntrack only?) from scratch and to insmod them into the kernel?
Download the kernel source code and compile it, that's all. If there were
other pom-ng patches applied, figure out which one are required in your
setup and apply them before compiling the kernel.
> Or there are parts of the kernel that I would need to recompile?
Because you want to leave out the TCP window tracking patch, therefore no.
In the case of other way around, because the patch requires a new feature
in the networking stack, you'd need to recompile (and install) the whole
kernel.
> In ideal situation I'd like to have a few versions of the conntrack (and
> other related?) and be able to insmod them on the fly.
Usually it's doable, sometimes it's not. You have to know on which
services offered by the networking core do the given patches depend.
You can find the relevant info in the pom-ng info files, but otherwise
it's not documented.
> Also can I just rmmod the conntrack module or I need to unload iptables
> and all the rest before I can insmod the new ones?
You can rmmod ip_conntrack anytime with the price of losing the conntrack
info.
Best regards,
Jozsef
-
E-mail : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
H-1525 Budapest 114, POB. 49, Hungary
^ permalink raw reply [flat|nested] 16+ messages in thread