* ip_conntrack_tcp Errors
@ 2004-06-28 11:47 Evgeni Vachkov
2004-06-28 12:04 ` Jozsef Kadlecsik
` (2 more replies)
0 siblings, 3 replies; 16+ messages in thread
From: Evgeni Vachkov @ 2004-06-28 11:47 UTC (permalink / raw)
To: netfilter
Hi all,
When I load test one of our firewalls, when the concurrent connections
reach arround 230, I am getting a lot of error messages as shown below.
Mostly indicating that the server has sent an invalid SYN. This is a
heavy load firewall. I thought that increasing
ip_conntrack_max and ip_conntrack_buckets would help, but this wasnt the
case.
The ip_conntrack version is 2.1. kernel is v 2.4.26
Is that a problem with conntrack and its tunning or I am missing some
patch? ...Or perhaps it is some other problem with other parts of the
kernel?
Your quick help is greatly appreciated.
Regards,
Evgeni Vachkov
Jun 25 16:38:51 myserver kernel: ip_conntrack_tcp: INVALID: invalid SYN
(ignored) SRC=172.30.4.200 DST=192.168.30.3 LEN=60 TOS=0x00 PREC=0x00
TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=43226 SEQ=461046254 ACK=654564425
WINDOW=5792 RES=0x00 ECE ACK SYN URGP=0 OPT
(020405B40402080A2D5584DD106BFE1501030300)
Jun 25 16:38:55 myserver kernel: NET: 171 messages suppressed.
Jun 25 16:38:55 myserver kernel: ip_conntrack_tcp: INVALID: invalid SYN
(ignored) SRC=172.300.40.20 DST=192.168.130.30 LEN=60 TOS=0x00 PREC=0x00
TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=39098 SEQ=449809028 ACK=643180415
WINDOW=5792 RES=0x00 ECE ACK SYN URGP=0 OPT
(020405B40402080A2D558695106BFD6F01030300)
Jun 25 16:39:02 myserver kernel: NET: 314 messages suppressed.
Jun 25 16:39:02 myserver kernel: ip_conntrack_tcp: IGNORED: Out of
window data; SEQ is under the lower bound (retransmitted already ACKed
data)
Jun 25 16:39:02 myserver kernel: SRC=172.300.40.20 DST=192.168.130.30
LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=32438 DF PROTO=TCP SPT=80 DPT=42539
SEQ=4211842796 ACK=108452331 WINDOW=5792 RES=0x00 ACK FIN URGP=0 OPT
(0101080A2D558930106B4C75)
Jun 25 16:39:05 myserver kernel: NET: 421 messages suppressed.
Jun 25 16:39:05 myserver kernel: ip_conntrack_tcp: INVALID: invalid SYN
(ignored) SRC=172.300.40.20 DST=192.168.130.30 LEN=60 TOS=0x00 PREC=0x00
TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44563 SEQ=471823482 ACK=665607863
WINDOW=5792 RES=0x00 ECE ACK SYN URGP=0 OPT
(020405B40402080A2D558A7D106C033A01030300)
Jun 25 16:39:10 myserver kernel: NET: 249 messages suppressed.
Jun 25 16:39:10 myserver kernel: ip_conntrack_tcp: INVALID: invalid SYN
(ignored) SRC=172.300.40.20 DST=192.168.130.30 LEN=60 TOS=0x00 PREC=0x00
TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=45696 SEQ=469531976 ACK=675466444
WINDOW=5792 RES=0x00 ECE ACK SYN URGP=0 OPT
(020405B40402080A2D558C71106C051B01030300)
Jun 25 16:39:17 myserver kernel: NET: 86 messages suppressed.
Jun 25 16:39:17 myserver kernel: ip_conntrack_tcp: INVALID: invalid SYN
(ignored) SRC=172.130.40.20 DST=192.168.130.30 LEN=60 TOS=0x00 PREC=0x00
TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44525 SEQ=459293358 ACK=669647335
WINDOW=5792 RES=0x00 ECE ACK SYN URGP=0 OPT
(020405B40402080A2D558EF1106C061A01030300)
Jun 25 16:39:21 myserver kernel: NET: 244 messages suppressed.
Jun 25 16:39:21 myserver kernel: ip_conntrack_tcp: IGNORED: Out of
window data; SEQ is under the lower bound (retransmitted already ACKed
data)
Jun 25 16:39:21 myserver kernel: SRC=172.130.40.20 DST=192.168.130.30
LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=13539 DF PROTO=TCP SPT=80 DPT=46472
SEQ=4233682626 ACK=136266752 WINDOW=5792 RES=0x00 ACK FIN URGP=0 OPT
(0101080A2D559068106B55F1)
Jun 25 16:39:28 myserver kernel: NET: 196 messages suppressed.
Jun 25 16:39:28 myserver kernel: ip_conntrack_tcp: IGNORED: Out of
window data; SEQ is under the lower bound (retransmitted already ACKed
data)
Jun 25 16:39:28 myserver kernel: SRC=172.130.40.20 DST=192.168.130.30
LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=6361 DF PROTO=TCP SPT=80 DPT=42381
SEQ=4200515268 ACK=98652606 WINDOW=5792 RES=0x00 ACK FIN URGP=0 OPT
(0101080A2D55934B106B4A1D)
Jun 25 16:39:34 myserver kernel: NET: 23 messages suppressed.
Jun 25 16:39:34 myserver kernel: ip_conntrack_tcp: IGNORED: Out of
window data; SEQ is under the lower bound (retransmitted already ACKed
data)
Jun 25 16:39:34 myserver kernel: SRC=172.130.40.20 DST=192.168.130.30
LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=63653 DF PROTO=TCP SPT=80 DPT=46497
SEQ=4227142309 ACK=128387976 WINDOW=5792 RES=0x00 ACK FIN URGP=0 OPT
(0101080A2D55957C106B55F1)
Jun 25 16:39:41 myserver kernel: NET: 9 messages suppressed.
Jun 25 16:39:41 myserver kernel: ip_conntrack_tcp: INVALID: invalid SYN
(ignored) SRC=172.130.40.20 DST=192.168.130.30 LEN=60 TOS=0x00 PREC=0x00
TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44443 SEQ=464964164 ACK=658995371
WINDOW=5792 RES=0x00 ECE ACK SYN URGP=0 OPT
(020405B40402080A2D559865106C0BDA01030300)
Jun 25 16:39:41 myserver kernel: ip_conntrack_tcp: INVALID: invalid SYN
(ignored) SRC=172.130.40.20 DST=192.168.130.30 LEN=60 TOS=0x00 PREC=0x00
TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=44491 SEQ=461162821 ACK=673035892
WINDOW=5792 RES=0x00 ECE ACK SYN URGP=0 OPT
(020405B40402080A2D559865106C0BDA01030300)
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: ip_conntrack_tcp Errors
2004-06-28 11:47 ip_conntrack_tcp Errors Evgeni Vachkov
@ 2004-06-28 12:04 ` Jozsef Kadlecsik
2004-06-28 12:29 ` Evgeni Vachkov
2004-06-28 12:05 ` Antony Stone
2004-06-28 12:30 ` Dimitar Katerinski
2 siblings, 1 reply; 16+ messages in thread
From: Jozsef Kadlecsik @ 2004-06-28 12:04 UTC (permalink / raw)
To: Evgeni Vachkov; +Cc: netfilter
On 28 Jun 2004, Evgeni Vachkov wrote:
> When I load test one of our firewalls, when the concurrent connections
> reach arround 230, I am getting a lot of error messages as shown below.
That's a very low number of connections...
> Mostly indicating that the server has sent an invalid SYN. This is a
> heavy load firewall.
...How can the firewall be then heavily loaded?
> I thought that increasing ip_conntrack_max and ip_conntrack_buckets
> would help, but this wasnt the case.
>
> The ip_conntrack version is 2.1. kernel is v 2.4.26
>
> Is that a problem with conntrack and its tunning or I am missing some
> patch? ...Or perhaps it is some other problem with other parts of the
> kernel?
Why are there so many SYN/ACK packets sent when there is already a
connection established trough the firewall between the same IP addresses
and same ports?
> Jun 25 16:38:51 myserver kernel: ip_conntrack_tcp: INVALID: invalid SYN
> (ignored) SRC=172.30.4.200 DST=192.168.30.3 LEN=60 TOS=0x00 PREC=0x00
> TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=43226 SEQ=461046254 ACK=654564425
> WINDOW=5792 RES=0x00 ECE ACK SYN URGP=0 OPT
> (020405B40402080A2D5584DD106BFE1501030300)
Best regards,
Jozsef
-
E-mail : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
H-1525 Budapest 114, POB. 49, Hungary
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: ip_conntrack_tcp Errors
2004-06-28 11:47 ip_conntrack_tcp Errors Evgeni Vachkov
2004-06-28 12:04 ` Jozsef Kadlecsik
@ 2004-06-28 12:05 ` Antony Stone
2004-06-28 12:30 ` Dimitar Katerinski
2 siblings, 0 replies; 16+ messages in thread
From: Antony Stone @ 2004-06-28 12:05 UTC (permalink / raw)
To: netfilter
On Monday 28 June 2004 12:47 pm, Evgeni Vachkov wrote:
> Hi all,
>
> When I load test one of our firewalls, when the concurrent connections
> reach arround 230, I am getting a lot of error messages as shown below.
230 is a very small number. You should be able to support several thousand
connections (depending on RAM) without problems.
> Mostly indicating that the server has sent an invalid SYN. This is a
> heavy load firewall.
Define "heavy load"?
230 connections is not much. What sort of bandwidth are they generating in
total?
What sort of network cards are you using? Anything changed about the
hardware around the time you started seeing the messages?
Regards,
Antony.
--
What is this talk of "software release"?
Our software evolves and matures until it is capable of escape, leaving a
bloody trail of designers and quality assurance people in its wake.
Please reply to the list;
please don't CC me.
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: ip_conntrack_tcp Errors
2004-06-28 12:04 ` Jozsef Kadlecsik
@ 2004-06-28 12:29 ` Evgeni Vachkov
2004-06-28 12:47 ` Jozsef Kadlecsik
0 siblings, 1 reply; 16+ messages in thread
From: Evgeni Vachkov @ 2004-06-28 12:29 UTC (permalink / raw)
To: netfilter
The test software connects to the web server with 200 connections at a
time to start with. When it receives the test data, it creates 210
connections, then 220... ...and so forth until something wrong happens.
The interesting fact is that most connections were in TIME_WAIT from
(/proc/net/ip_conntrack).
Doing a `wc -l /proc/net/ip_conntrack` returns a figure arround 25000,
which is well below the ip_conntrack_max treshold.
> Why are there so many SYN/ACK packets sent when there is already a
> connection established trough the firewall between the same IP addresses
> and same ports?
I'd love to know this, too. The software Im using is based on ab (apache
benchmark tool). I beleive ab is creating multiple separate connections,
over which it is getting data from server and therefore simulating a
large nimber of users.
> > Jun 25 16:38:51 myserver kernel: ip_conntrack_tcp: INVALID: invalid SYN
> > (ignored) SRC=172.30.4.200 DST=192.168.30.3 LEN=60 TOS=0x00 PREC=0x00
> > TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=43226 SEQ=461046254 ACK=654564425
> > WINDOW=5792 RES=0x00 ECE ACK SYN URGP=0 OPT
> > (020405B40402080A2D5584DD106BFE1501030300)
Regrads,
Evgeni Vachkov
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: ip_conntrack_tcp Errors
2004-06-28 11:47 ip_conntrack_tcp Errors Evgeni Vachkov
2004-06-28 12:04 ` Jozsef Kadlecsik
2004-06-28 12:05 ` Antony Stone
@ 2004-06-28 12:30 ` Dimitar Katerinski
2004-06-28 12:45 ` Dimitar Katerinski
` (3 more replies)
2 siblings, 4 replies; 16+ messages in thread
From: Dimitar Katerinski @ 2004-06-28 12:30 UTC (permalink / raw)
To: netfilter
Evgeni Vachkov wrote:
> Hi all,
>
Hello Evgeni,
> When I load test one of our firewalls, when the concurrent connections
> reach arround 230, I am getting a lot of error messages as shown below.
> Mostly indicating that the server has sent an invalid SYN. This is a
> heavy load firewall. I thought that increasing
> ip_conntrack_max and ip_conntrack_buckets would help, but this wasnt the
> case.
As stated with the previous posts, 230 concurent connections is very low number
indeed. Hence, manually tuning the ip_conntrack_max wont help either ;-).
>
> The ip_conntrack version is 2.1. kernel is v 2.4.26
>
> Is that a problem with conntrack and its tunning or I am missing some
> patch? ...Or perhaps it is some other problem with other parts of the
> kernel?
It seems to me that you have applied the tcp window tracking patch from pom-ng.
The problem is that the client and the server have done the first step of the
three way handshake, and are in sync, but the firewall for some reason is not.
So it drops the SYN/ACK, and thus forcing the client to retransmit its SYN and
initiate a new session (as descibed in the source code of the patch)
My advice is if you have applied this patch, to remove it, and test the load on the
firewall again.
>
> Your quick help is greatly appreciated.
Doing the best I can do ;-)
>
> Regards,
> Evgeni Vachkov
>
Regards,
Dimitar
--
"The only thing necessary for the triumph of evil is for good men to do nothing."
--Edmund Burke.
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: ip_conntrack_tcp Errors
2004-06-28 12:30 ` Dimitar Katerinski
@ 2004-06-28 12:45 ` Dimitar Katerinski
2004-06-28 12:55 ` Jozsef Kadlecsik
` (2 subsequent siblings)
3 siblings, 0 replies; 16+ messages in thread
From: Dimitar Katerinski @ 2004-06-28 12:45 UTC (permalink / raw)
To: netfilter
Hm, it seems that Jozsef Kadlecsik wrote tcp window tracking patch ;-)
So he should know better whats really going on, right Jozsef?
Regards,
Dimitar
--
"The only thing necessary for the triumph of evil is for good men to do nothing."
--Edmund Burke.
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: ip_conntrack_tcp Errors
2004-06-28 12:29 ` Evgeni Vachkov
@ 2004-06-28 12:47 ` Jozsef Kadlecsik
0 siblings, 0 replies; 16+ messages in thread
From: Jozsef Kadlecsik @ 2004-06-28 12:47 UTC (permalink / raw)
To: Evgeni Vachkov; +Cc: netfilter
On 28 Jun 2004, Evgeni Vachkov wrote:
> The test software connects to the web server with 200 connections at a
> time to start with. When it receives the test data, it creates 210
> connections, then 220... ...and so forth until something wrong happens.
So this is the number of new connections created.
> The interesting fact is that most connections were in TIME_WAIT from
> (/proc/net/ip_conntrack).
That is not problem. And the client may even reopen a connection in the
TIME_WAIT state.
> Doing a `wc -l /proc/net/ip_conntrack` returns a figure arround 25000,
> which is well below the ip_conntrack_max treshold.
That's not bad either.
> > Why are there so many SYN/ACK packets sent when there is already a
> > connection established trough the firewall between the same IP addresses
> > and same ports?
>
> I'd love to know this, too. The software Im using is based on ab (apache
> benchmark tool). I beleive ab is creating multiple separate connections,
> over which it is getting data from server and therefore simulating a
> large nimber of users.
Based on or it is ab itself?
> > > Jun 25 16:38:51 myserver kernel: ip_conntrack_tcp: INVALID: invalid SYN
> > > (ignored) SRC=172.30.4.200 DST=192.168.30.3 LEN=60 TOS=0x00 PREC=0x00
> > > TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=43226 SEQ=461046254 ACK=654564425
> > > WINDOW=5792 RES=0x00 ECE ACK SYN URGP=0 OPT
> > > (020405B40402080A2D5584DD106BFE1501030300)
The question still remains: where is the SYN packet for which this SYN/ACK
is sent as a reply? Could you run tcpdump on the interface of the firewall
connecting it to the client?
Best regards,
Jozsef
-
E-mail : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
H-1525 Budapest 114, POB. 49, Hungary
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: ip_conntrack_tcp Errors
2004-06-28 12:30 ` Dimitar Katerinski
2004-06-28 12:45 ` Dimitar Katerinski
@ 2004-06-28 12:55 ` Jozsef Kadlecsik
2004-06-28 14:40 ` Evgeni Vachkov
2004-06-28 15:45 ` Evgeni Vachkov
2004-06-30 9:25 ` ip_conntrack_tcp Errors - backing a patch Evgeni Vachkov
3 siblings, 1 reply; 16+ messages in thread
From: Jozsef Kadlecsik @ 2004-06-28 12:55 UTC (permalink / raw)
To: Dimitar Katerinski; +Cc: netfilter
On Mon, 28 Jun 2004, Dimitar Katerinski wrote:
> > Is that a problem with conntrack and its tunning or I am missing some
> > patch? ...Or perhaps it is some other problem with other parts of the
> > kernel?
> It seems to me that you have applied the tcp window tracking patch from pom-ng.
> The problem is that the client and the server have done the first step of the
> three way handshake, and are in sync, but the firewall for some reason is not.
Sorry, but I have to contradict: there must be a connection in conntrack
which overlaps with the SYN/ACK packet detected. But why the connection
initiating SYN was not detected the same way? That is the question which
should be answered somehow.
> So it drops the SYN/ACK, and thus forcing the client to retransmit its SYN and
> initiate a new session (as descibed in the source code of the patch)
No, it does not drop the packet but ignores it as the log says. Look at
the lines in the source code a few lines below.
> My advice is if you have applied this patch, to remove it, and test the load on the
> firewall again.
Yep, that's a solution. And there won't be an answer explaining the case
then.
Best regards,
Jozsef
-
E-mail : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
H-1525 Budapest 114, POB. 49, Hungary
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: ip_conntrack_tcp Errors
2004-06-28 12:55 ` Jozsef Kadlecsik
@ 2004-06-28 14:40 ` Evgeni Vachkov
2004-06-29 7:55 ` Jozsef Kadlecsik
0 siblings, 1 reply; 16+ messages in thread
From: Evgeni Vachkov @ 2004-06-28 14:40 UTC (permalink / raw)
To: netfilter
Jozsef,
I understand the need to identify the problem and I am more than willing
to help. It would be appreciated if you could adise me on the tcpdump
command switches to run, since I dont want to scan through thousands of
packets to identify the missing one. Also I am not an expert in tcpdump,
so I will need your help.
Note that this problem occurs only when ip_conntrack table reaches
arround 25000 connections. The software I am running is using ab (a
shell script incorporatring ab).
Regards,
Evgeni Vachkov
On Mon, 2004-06-28 at 13:55, Jozsef Kadlecsik wrote:
> On Mon, 28 Jun 2004, Dimitar Katerinski wrote:
>
> > > Is that a problem with conntrack and its tunning or I am missing some
> > > patch? ...Or perhaps it is some other problem with other parts of the
> > > kernel?
> > It seems to me that you have applied the tcp window tracking patch from pom-ng.
> > The problem is that the client and the server have done the first step of the
> > three way handshake, and are in sync, but the firewall for some reason is not.
>
> Sorry, but I have to contradict: there must be a connection in conntrack
> which overlaps with the SYN/ACK packet detected. But why the connection
> initiating SYN was not detected the same way? That is the question which
> should be answered somehow.
>
> > So it drops the SYN/ACK, and thus forcing the client to retransmit its SYN and
> > initiate a new session (as descibed in the source code of the patch)
>
> No, it does not drop the packet but ignores it as the log says. Look at
> the lines in the source code a few lines below.
>
> > My advice is if you have applied this patch, to remove it, and test the load on the
> > firewall again.
>
> Yep, that's a solution. And there won't be an answer explaining the case
> then.
>
> Best regards,
> Jozsef
> -
> E-mail : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu
> PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
> Address : KFKI Research Institute for Particle and Nuclear Physics
> H-1525 Budapest 114, POB. 49, Hungary
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: ip_conntrack_tcp Errors
2004-06-28 12:30 ` Dimitar Katerinski
2004-06-28 12:45 ` Dimitar Katerinski
2004-06-28 12:55 ` Jozsef Kadlecsik
@ 2004-06-28 15:45 ` Evgeni Vachkov
2004-06-29 7:46 ` Jozsef Kadlecsik
2004-06-30 9:25 ` ip_conntrack_tcp Errors - backing a patch Evgeni Vachkov
3 siblings, 1 reply; 16+ messages in thread
From: Evgeni Vachkov @ 2004-06-28 15:45 UTC (permalink / raw)
To: netfilter
> >
> > The ip_conntrack version is 2.1. kernel is v 2.4.26
> >
> > Is that a problem with conntrack and its tunning or I am missing some
> > patch? ...Or perhaps it is some other problem with other parts of the
> > kernel?
> It seems to me that you have applied the tcp window tracking patch from pom-ng.
> The problem is that the client and the server have done the first step of the
> three way handshake, and are in sync, but the firewall for some reason is not.
> So it drops the SYN/ACK, and thus forcing the client to retransmit its SYN and
> initiate a new session (as descibed in the source code of the patch)
>
> My advice is if you have applied this patch, to remove it, and test the load on the
> firewall again.
As far as I can figure out we are running with
patch-o-matic-ng-20040302. I can see that the latest is
patch-o-matic-ng-20040621. Is this problem present at the version we are
running and is the new version still having the window tracking issues?
Regards,
Evgeni Vachkov
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: ip_conntrack_tcp Errors
2004-06-28 15:45 ` Evgeni Vachkov
@ 2004-06-29 7:46 ` Jozsef Kadlecsik
0 siblings, 0 replies; 16+ messages in thread
From: Jozsef Kadlecsik @ 2004-06-29 7:46 UTC (permalink / raw)
To: Evgeni Vachkov; +Cc: netfilter
On 28 Jun 2004, Evgeni Vachkov wrote:
> > > patch? ...Or perhaps it is some other problem with other parts of the
> > > kernel?
> > It seems to me that you have applied the tcp window tracking patch from pom-ng.
> > The problem is that the client and the server have done the first step of the
> > three way handshake, and are in sync, but the firewall for some reason is not.
> > So it drops the SYN/ACK, and thus forcing the client to retransmit its SYN and
> > initiate a new session (as descibed in the source code of the patch)
> >
> > My advice is if you have applied this patch, to remove it, and test the load on the
> > firewall again.
>
> As far as I can figure out we are running with
> patch-o-matic-ng-20040302. I can see that the latest is
> patch-o-matic-ng-20040621. Is this problem present at the version we are
> running and is the new version still having the window tracking issues?
SACK support was added to the patch in patch-o-matic-ng-20040621 with some
minor fixes, compared to patch-o-matic-ng-20040302. It does not hurt to
upgrade but I don't expect that it'd behave differently in your test.
Best regards,
Jozsef
-
E-mail : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
H-1525 Budapest 114, POB. 49, Hungary
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: ip_conntrack_tcp Errors
2004-06-28 14:40 ` Evgeni Vachkov
@ 2004-06-29 7:55 ` Jozsef Kadlecsik
2004-06-29 9:30 ` Evgeni Vachkov
0 siblings, 1 reply; 16+ messages in thread
From: Jozsef Kadlecsik @ 2004-06-29 7:55 UTC (permalink / raw)
To: Evgeni Vachkov; +Cc: netfilter
On 28 Jun 2004, Evgeni Vachkov wrote:
> I understand the need to identify the problem and I am more than willing
> to help. It would be appreciated if you could adise me on the tcpdump
> command switches to run, since I dont want to scan through thousands of
> packets to identify the missing one. Also I am not an expert in tcpdump,
> so I will need your help.
Assuming that eth0 faces with the client machine, issue the
following command before starting the test:
# tcpdump -i eth0 -w capture.log tcp
If the machine is short in disk space you can compress the log on the fly
instead:
# tcpdump -i eth0 -w - tcp | gzip -c9 > capture.log.gz
and then you can stop tcpdump by
# killall -INT tcpdump
> Note that this problem occurs only when ip_conntrack table reaches
> arround 25000 connections. The software I am running is using ab (a
> shell script incorporatring ab).
I assume you run Linux on the client machine running ab as well, don't
you? What it the output of
# cat /proc/sys/net/ipv4/ip_local_port_range
on that machine?
Best regards,
Jozsef
-
E-mail : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
H-1525 Budapest 114, POB. 49, Hungary
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: ip_conntrack_tcp Errors
2004-06-29 7:55 ` Jozsef Kadlecsik
@ 2004-06-29 9:30 ` Evgeni Vachkov
2004-06-29 9:52 ` Jozsef Kadlecsik
0 siblings, 1 reply; 16+ messages in thread
From: Evgeni Vachkov @ 2004-06-29 9:30 UTC (permalink / raw)
To: netfilter
On Tue, 2004-06-29 at 08:55, Jozsef Kadlecsik wrote:
> Assuming that eth0 faces with the client machine, issue the
> following command before starting the test:
>
> # tcpdump -i eth0 -w capture.log tcp
>
I have completed the dump, file is quite big > 63MB. Is there a
possibility to limit the dump to the tested source and destination ip-s
only?
What shall I do with it next?
> I assume you run Linux on the client machine running ab as well, don't
> you?
>
Yes.
> What it the output of
>
> # cat /proc/sys/net/ipv4/ip_local_port_range
32768 61000
>
> on that machine?
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: ip_conntrack_tcp Errors
2004-06-29 9:30 ` Evgeni Vachkov
@ 2004-06-29 9:52 ` Jozsef Kadlecsik
0 siblings, 0 replies; 16+ messages in thread
From: Jozsef Kadlecsik @ 2004-06-29 9:52 UTC (permalink / raw)
To: Evgeni Vachkov; +Cc: netfilter
On 29 Jun 2004, Evgeni Vachkov wrote:
> On Tue, 2004-06-29 at 08:55, Jozsef Kadlecsik wrote:
> > Assuming that eth0 faces with the client machine, issue the
> > following command before starting the test:
> >
> > # tcpdump -i eth0 -w capture.log tcp
> >
> I have completed the dump, file is quite big > 63MB. Is there a
> possibility to limit the dump to the tested source and destination ip-s
> only?
I assumed there is no other traffic on the wire, just the test. You can
filter out the traffic between the two interesting parties by
tcpdump -r capture.log -w selected.log host a.b.c.d and host x.y.z.w
You can compress it as well. I'd need both the tcpdump log file and the
kernel log from the same time slice the tcpdump ran.
> > What it the output of
> >
> > # cat /proc/sys/net/ipv4/ip_local_port_range
>
> 32768 61000
That's 28232 ports available, over the 25000 connections you mentioned.
Best regards,
Jozsef
-
E-mail : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
H-1525 Budapest 114, POB. 49, Hungary
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: ip_conntrack_tcp Errors - backing a patch
2004-06-28 12:30 ` Dimitar Katerinski
` (2 preceding siblings ...)
2004-06-28 15:45 ` Evgeni Vachkov
@ 2004-06-30 9:25 ` Evgeni Vachkov
2004-06-30 12:58 ` Jozsef Kadlecsik
3 siblings, 1 reply; 16+ messages in thread
From: Evgeni Vachkov @ 2004-06-30 9:25 UTC (permalink / raw)
To: netfilter
On Mon, 2004-06-28 at 13:30, Dimitar Katerinski wrote:
> > Is that a problem with conntrack and its tunning or I am missing some
> > patch? ...Or perhaps it is some other problem with other parts of the
> > kernel?
> It seems to me that you have applied the tcp window tracking patch from pom-ng.
> The problem is that the client and the server have done the first step of the
> three way handshake, and are in sync, but the firewall for some reason is not.
> So it drops the SYN/ACK, and thus forcing the client to retransmit its SYN and
> initiate a new session (as descibed in the source code of the patch)
>
> My advice is if you have applied this patch, to remove it, and test the load on the
> firewall again.
> >
I would like to remove the patch, but unfortunately the person who
compiled last time didnt leave the code.
So....
Would it be possible to compile the relevent modules only (is it
conntrack only?) from scratch and to insmod them into the kernel?
Or there are parts of the kernel that I would need to recompile?
In any case, what code do I need to download and compile and where do I
find the relevant instructions?
In ideal situation I'd like to have a few versions of the conntrack (and
other related?) and be able to insmod them on the fly.
Also can I just rmmod the conntrack module or I need to unload iptables
and all the rest before I can insmod the new ones?
Many thanks,
Evgeni
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: ip_conntrack_tcp Errors - backing a patch
2004-06-30 9:25 ` ip_conntrack_tcp Errors - backing a patch Evgeni Vachkov
@ 2004-06-30 12:58 ` Jozsef Kadlecsik
0 siblings, 0 replies; 16+ messages in thread
From: Jozsef Kadlecsik @ 2004-06-30 12:58 UTC (permalink / raw)
To: Evgeni Vachkov; +Cc: netfilter
On 30 Jun 2004, Evgeni Vachkov wrote:
> On Mon, 2004-06-28 at 13:30, Dimitar Katerinski wrote:
> > > Is that a problem with conntrack and its tunning or I am missing some
> > > patch? ...Or perhaps it is some other problem with other parts of the
> > > kernel?
> > My advice is if you have applied this patch, to remove it, and test the load on the
> > firewall again.
>
> I would like to remove the patch, but unfortunately the person who
> compiled last time didnt leave the code.
That's bad: with pom-ng, you can remove installed patches as well.
> Would it be possible to compile the relevent modules only (is it
> conntrack only?) from scratch and to insmod them into the kernel?
Download the kernel source code and compile it, that's all. If there were
other pom-ng patches applied, figure out which one are required in your
setup and apply them before compiling the kernel.
> Or there are parts of the kernel that I would need to recompile?
Because you want to leave out the TCP window tracking patch, therefore no.
In the case of other way around, because the patch requires a new feature
in the networking stack, you'd need to recompile (and install) the whole
kernel.
> In ideal situation I'd like to have a few versions of the conntrack (and
> other related?) and be able to insmod them on the fly.
Usually it's doable, sometimes it's not. You have to know on which
services offered by the networking core do the given patches depend.
You can find the relevant info in the pom-ng info files, but otherwise
it's not documented.
> Also can I just rmmod the conntrack module or I need to unload iptables
> and all the rest before I can insmod the new ones?
You can rmmod ip_conntrack anytime with the price of losing the conntrack
info.
Best regards,
Jozsef
-
E-mail : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
H-1525 Budapest 114, POB. 49, Hungary
^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2004-06-30 12:58 UTC | newest]
Thread overview: 16+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2004-06-28 11:47 ip_conntrack_tcp Errors Evgeni Vachkov
2004-06-28 12:04 ` Jozsef Kadlecsik
2004-06-28 12:29 ` Evgeni Vachkov
2004-06-28 12:47 ` Jozsef Kadlecsik
2004-06-28 12:05 ` Antony Stone
2004-06-28 12:30 ` Dimitar Katerinski
2004-06-28 12:45 ` Dimitar Katerinski
2004-06-28 12:55 ` Jozsef Kadlecsik
2004-06-28 14:40 ` Evgeni Vachkov
2004-06-29 7:55 ` Jozsef Kadlecsik
2004-06-29 9:30 ` Evgeni Vachkov
2004-06-29 9:52 ` Jozsef Kadlecsik
2004-06-28 15:45 ` Evgeni Vachkov
2004-06-29 7:46 ` Jozsef Kadlecsik
2004-06-30 9:25 ` ip_conntrack_tcp Errors - backing a patch Evgeni Vachkov
2004-06-30 12:58 ` Jozsef Kadlecsik
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.