Weird Problem with TCP

Weird Problem with TCP

[Rohit Neupane]

Hi,
 I have a linux box which acts as gateway and b/w manager for our wireless
network. I have 3 Interfaces ( one not is use yet). I have created 3 aliases
on the interface connected to the wireless (with 3 different class C
address).
I have around 350-400 wireless clients with over 10Mbps traffic at peak
hours.
These days I have observed some abnormal behavior.

System:
Dell P4 Poweredge
Red Hat 8.0 with 2.4.25 kernel
Lan Cards:
 1. Intel 82540EM Gigabit Ethernet Controller (Intenet)
 2. 3com 3c905B Cyclone (Wireless)


The problem is:
* Everything works fine for about 5-10 mins then all of a sudden TCP services
are not accessable.
* For some reason TCP times out. However at the same time ping,traceroute and
dns trace works without any problem.
* The connected TCP sokets keeps working without any problem. I verified this
by using Msn chat. I observerd that I chat session ( which I had started
when everything was normal) continued without any problem however I was not
able to initiate a new chat session.

Result of tcpdum on eth0 (internet) and eth2(wireless)

* I can see tcp SYN request coming in from eth2 (from client) and going out
via eth0 to destination.
* I can see tcp SYN+ACK coming in from eth0. But it doesn't get out through
eth2 !!! However other udp,icmp pkts can.

src 69.56.37.146  , dst 64.236.16.52
this is tcpdump on eth0 (internet side)
17:14:32.003370 69.56.37.146.3170 > 64.236.16.52.http: S
2446361626:2446361626(0) win 64240 <mss 1460,nop,nop,sackOK> (DF)
17:14:32.004694 64.236.16.52.http > 69.56.37.146.3170: S
3641974278:3641974278(0) ack 2446361627 win 65535 <mss 1460> (DF)
17:14:34.920578 69.56.37.146.3170 > 64.236.16.52.http: S
2446361626:2446361626(0) win 64240 <mss 1460,nop,nop,sackOK> (DF)
17:14:34.921800 64.236.16.52.http > 69.56.37.146.3170: S
3641974278:3641974278(0) ack 2446361627 win 65535 <mss 1460> (DF)
17:14:37.915564 64.236.16.52.http > 69.56.37.146.3170: S
3641974278:3641974278(0) ack 2446361627 win 65535 <mss 1460> (DF)
17:14:40.955491 69.56.37.146.3170 > 64.236.16.52.http: S
2446361626:2446361626(0) win 64240 <mss 1460,nop,nop,sackOK> (DF)
17:14:40.957682 64.236.16.52.http > 69.56.37.146.3170: S
3641974278:3641974278(0) ack 2446361627 win 65535 <mss 1460> (DF)

this is tcpdump on eth2  (client side)
17:14:32.003179 69.56.37.146.3170 > 64.236.16.52.http: S
2446361626:2446361626(0) win 64240 <mss 1460,nop,nop,sackOK> (DF)
17:14:34.920212 69.56.37.146.3170 > 64.236.16.52.http: S
2446361626:2446361626(0) win 64240 <mss 1460,nop,nop,sackOK> (DF)
17:14:40.955257 69.56.37.146.3170 > 64.236.16.52.http: S
2446361626:2446361626(0) win 64240 <mss 1460,nop,nop,sackOK> (DF)

As you can see i am getting the ACK from source from seq. 2446361626 (with
ack 2446361627 ) on eth0. But it is not going out via eth2!!

After sometime everthing starts working normally again. So what may be the
problem? I have problem only with the tcp protocol and not with other. I
haven't changed any tcp memory related settings.

net.ipv4.tcp_rmem = 4096        87380   174760
net.ipv4.tcp_wmem = 4096        87380   174760
net.ipv4.tcp_mem = 195584       196096  196608
net.core.optmem_max = 10240
net.core.rmem_default = 65535
net.core.wmem_default = 65535
net.core.rmem_max = 131071
net.core.wmem_max = 131071


Has anyone come across this probelm? any suggestion?

Thanks
Rohit

Re: Weird Problem with TCP

[Alan Cox]

> * Everything works fine for about 5-10 mins then all of a sudden TCP services
> are not accessable.
> * For some reason TCP times out. However at the same time ping,traceroute and
> dns trace works without any problem.
> * The connected TCP sokets keeps working without any problem. I verified this
> by using Msn chat. I observerd that I chat session ( which I had started
> when everything was normal) continued without any problem however I was not
> able to initiate a new chat session.

Are you using session tracking. The symptoms you describe are
classically those of session tracking nat/firewalling/whatever running
out of table entries and being unable to allow new connections.

Re: Weird Problem with TCP

[Rohit Neupane]

On Thu, 02 Sep 2004 11:56:59 +0100, Alan Cox <alan@lxorguk.ukuu.org.uk> wrote:
> > * Everything works fine for about 5-10 mins then all of a sudden TCP services
> > are not accessable.
> > * For some reason TCP times out. However at the same time ping,traceroute and
> > dns trace works without any problem.
> > * The connected TCP sokets keeps working without any problem. I verified this
> > by using Msn chat. I observerd that I chat session ( which I had started
> > when everything was normal) continued without any problem however I was not
> > able to initiate a new chat session.
> 
> Are you using session tracking. The symptoms you describe are
> classically those of session tracking nat/firewalling/whatever running
> out of table entries and being unable to allow new connections.
> 
No, it is not running any session tracking (ip_conntrack) neither it
does nat. It is just a firewall with around 1600 rules in FORWARD
mangle table and around 1500 rules in FORWARD filter table. Out of
1500 rules , 1377 rules are MAC filter rules.
And it had 3 alias address for the interface conneted to the wirelss.

regards,
Rohit


>

Re: Weird Problem with TCP

[Stuart Young]

On Thu, 2 Sep 2004 23:33, Rohit Neupane wrote:
> On Thu, 02 Sep 2004 11:56:59 +0100, Alan Cox <alan@lxorguk.ukuu.org.uk> 
wrote:
> > > * Everything works fine for about 5-10 mins then all of a sudden TCP
> > > services are not accessable.
> > > * For some reason TCP times out. However at the same time
> > > ping,traceroute and dns trace works without any problem.
> > > * The connected TCP sokets keeps working without any problem. I
> > > verified this by using Msn chat. I observerd that I chat session (
> > > which I had started when everything was normal) continued without any
> > > problem however I was not able to initiate a new chat session.
> >
> > Are you using session tracking. The symptoms you describe are
> > classically those of session tracking nat/firewalling/whatever running
> > out of table entries and being unable to allow new connections.
>
> No, it is not running any session tracking (ip_conntrack) neither it
> does nat. It is just a firewall with around 1600 rules in FORWARD
> mangle table and around 1500 rules in FORWARD filter table. Out of
> 1500 rules , 1377 rules are MAC filter rules.
> And it had 3 alias address for the interface conneted to the wirelss.

EEEEK! 1600? That is insane!

Consider cutting your rules into sections, and jumping to other tables to do 
sections of the work. Perhaps you can filter on the start of the MAC address 
and break this into smaller sections?

Also of note: MAC addresses are easily spoofed, so if you're using this to 
lock out people on wireless, forget it, it doesn't work. Get them to use 
tunnels (eg: ipsec) instead. The only real way MAC addresses even sort of 
work is when you're providing a hotspot, ie: where you can't guarantee the 
client to have anything apart from base wireless, and you should therefore 
keep a tight leash on users connections by either timing them out regularly, 
or making them keep open a https:// page to a login/AAA server (ie: a page 
that auto-refreshes - when they stop refreshing the page, consider their 
connection stale).

-- 
 Stuart Young (aka Cef)
 cef-lkml@optusnet.com.au is for LKML and related email only

Re: Weird Problem with TCP

[Denis Vlasenko]

> > > Are you using session tracking. The symptoms you describe are
> > > classically those of session tracking nat/firewalling/whatever running
> > > out of table entries and being unable to allow new connections.
> >
> > No, it is not running any session tracking (ip_conntrack) neither it
> > does nat. It is just a firewall with around 1600 rules in FORWARD
> > mangle table and around 1500 rules in FORWARD filter table. Out of
> > 1500 rules , 1377 rules are MAC filter rules.
> > And it had 3 alias address for the interface conneted to the wirelss.
>
> EEEEK! 1600? That is insane!
>
> Consider cutting your rules into sections, and jumping to other tables to
> do sections of the work. Perhaps you can filter on the start of the MAC
> address and break this into smaller sections?
>
> Also of note: MAC addresses are easily spoofed, so if you're using this to
> lock out people on wireless, forget it, it doesn't work. Get them to use

Yes. I saw MAC filtering being hacked by a teenager with WinXP.
I inflicted OpenVPN on him. Now hack *that*, boy... ;)

> tunnels (eg: ipsec) instead. The only real way MAC addresses even sort of
> work is when you're providing a hotspot, ie: where you can't guarantee the
> client to have anything apart from base wireless, and you should therefore
> keep a tight leash on users connections by either timing them out
> regularly, or making them keep open a https:// page to a login/AAA server
> (ie: a page that auto-refreshes - when they stop refreshing the page,
> consider their connection stale).
--
vda

Re: Weird Problem with TCP

[Rohit Neupane]

On Thu, 02 Sep 2004 08:33:30 -0600, kwijibo@zianet.com
<kwijibo@zianet.com> wrote:
> Are you sure you don't have any rules that have "state" listed in them?
> It really does sound like you have a stateful firewall.  But in the case
> that you don't, try logging our TCP rules that don't appear to be working.

Yes i am sure I don't have any state in the firewall. Infact we were
aware of the state problem from the beginning hence we didn't include
in the firewall at all.
And the problem is not just with one IP. It happens with almost all
client but ramdomly.
If there is any test that I need to carry out then please suggest.
Also, please not that I am running HTB and SFQ for b/w shaping purpose
on the same machine.

Rohit

> 
> Steve
> 
> 
> 
> Rohit Neupane wrote:
> 
> >On Thu, 02 Sep 2004 11:56:59 +0100, Alan Cox <alan@lxorguk.ukuu.org.uk> wrote:
> >
> >
> >>>* Everything works fine for about 5-10 mins then all of a sudden TCP services
> >>>are not accessable.
> >>>* For some reason TCP times out. However at the same time ping,traceroute and
> >>>dns trace works without any problem.
> >>>* The connected TCP sokets keeps working without any problem. I verified this
> >>>by using Msn chat. I observerd that I chat session ( which I had started
> >>>when everything was normal) continued without any problem however I was not
> >>>able to initiate a new chat session.
> >>>
> >>>
> >>Are you using session tracking. The symptoms you describe are
> >>classically those of session tracking nat/firewalling/whatever running
> >>out of table entries and being unable to allow new connections.
> >>
> >>
> >>
> >No, it is not running any session tracking (ip_conntrack) neither it
> >does nat. It is just a firewall with around 1600 rules in FORWARD
> >mangle table and around 1500 rules in FORWARD filter table. Out of
> >1500 rules , 1377 rules are MAC filter rules.
> >And it had 3 alias address for the interface conneted to the wirelss.
> >
> >regards,
> >Rohit
> >
> >
> >
> >
> >-
> >To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> >the body of a message to majordomo@vger.kernel.org
> >More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >Please read the FAQ at  http://www.tux.org/lkml/
> >
> >
> >
> >
> 
>

Re: Weird Problem with TCP

[Andre Tomt]

Rohit Neupane wrote:
> No, it is not running any session tracking (ip_conntrack) neither it
> does nat. It is just a firewall with around 1600 rules in FORWARD
> mangle table and around 1500 rules in FORWARD filter table. Out of
> 1500 rules , 1377 rules are MAC filter rules.
> And it had 3 alias address for the interface conneted to the wirelss.

Ouch. Thats a lot of rules to traverse for each packet. Segment them 
into chains if possible. Also you may want to take a look at nf-hipac, 
http://www.hipac.org

-- 
André Tomt

Re: Weird Problem with TCP

[Rohit Neupane]

On Thu, 02 Sep 2004 17:38:07 -0600, Dax Kelson <dax@gurulabs.com> wrote:
> Can you run this command:
> 
> cat /proc/net/ip_conntrack  | wc -l
> 
> And email me the results?

I don't have the file. I don't have any rule that has "state" in it.
These are the modules under ip-tables
ip_tables              16800   7  [ipt_mark ipt_MARK iptable_mangle
ipt_limit ipt_mac ipt_LOG iptable_filter]

regards,
Rohit
> 
>