* Weird TCP flags?
@ 2003-12-12 4:11 Ian Hunter
2003-12-12 4:46 ` Jeffrey Laramie
` (4 more replies)
0 siblings, 5 replies; 28+ messages in thread
From: Ian Hunter @ 2003-12-12 4:11 UTC (permalink / raw)
To: netfilter
OK, I have a router (lucy) with a webserver (192.168.254.242) in a DMZ (off
eth1), and everything works fine -- when you hit my ip, you get the site,
all is well. However, I get STORMS of this nonsense in my logs:
Dec 11 22:58:52 lucy kernel: Fwd DMZ->Internet DROP: IN=eth1 OUT=ppp0
SRC=192.168.254.242 DST=204.157.6.223 LEN=60 TOS=0x00 PREC=0x00 TTL=63
ID=56169 DF PROTO=TCP SPT=80 DPT=56319 WINDOW=32476 RES=0x00 ACK SYN URGP=0
(That's responding to these rules at the end of the FORWARD chain of filter
table:)
iptables -A FORWARD -i eth1 -o ppp0 -j LOG --log-prefix "Fwd DMZ->Internet
DROP: "
iptables -A FORWARD -i eth1 -o ppp0 -j DROP
Now if web traffic is working, my "-m state --state ESTABLISHED,RELATED -j
ACCEPT" must be working on forwards from eth1 -> ppp0, so why am I seeing
these messages? Are these broken TCP connections that netfilter doesn't
consider established?
Where do I look now?
Thanks,
Ian
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: Weird TCP flags?
2003-12-12 4:11 Weird TCP flags? Ian Hunter
@ 2003-12-12 4:46 ` Jeffrey Laramie
2003-12-12 4:52 ` Ian Hunter
2003-12-12 4:51 ` Jeffrey Laramie
` (3 subsequent siblings)
4 siblings, 1 reply; 28+ messages in thread
From: Jeffrey Laramie @ 2003-12-12 4:46 UTC (permalink / raw)
To: Ian Hunter, netfilter
On Thursday 11 December 2003 23:11, Ian Hunter wrote:
> OK, I have a router (lucy) with a webserver (192.168.254.242) in a DMZ (off
> eth1), and everything works fine -- when you hit my ip, you get the site,
> all is well. However, I get STORMS of this nonsense in my logs:
>
> Dec 11 22:58:52 lucy kernel: Fwd DMZ->Internet DROP: IN=eth1 OUT=ppp0
> SRC=192.168.254.242 DST=204.157.6.223 LEN=60 TOS=0x00 PREC=0x00 TTL=63
> ID=56169 DF PROTO=TCP SPT=80 DPT=56319 WINDOW=32476 RES=0x00 ACK SYN URGP=0
>
These have ACK SYN set so I doubt they have anything to do with established
connections. I would guess that this is a browser on the webserver itself
trying to connect to an outside site which your rules don't allow. In this
case: bart.routesys.com.
Jeff
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: Weird TCP flags?
2003-12-12 4:11 Weird TCP flags? Ian Hunter
2003-12-12 4:46 ` Jeffrey Laramie
@ 2003-12-12 4:51 ` Jeffrey Laramie
[not found] ` <200312120105.01557.Alistair Tonner <>
` (2 subsequent siblings)
4 siblings, 0 replies; 28+ messages in thread
From: Jeffrey Laramie @ 2003-12-12 4:51 UTC (permalink / raw)
To: Ian Hunter, netfilter
On Thursday 11 December 2003 23:11, Ian Hunter wrote:
> OK, I have a router (lucy) with a webserver (192.168.254.242) in a DMZ (off
> eth1), and everything works fine -- when you hit my ip, you get the site,
> all is well. However, I get STORMS of this nonsense in my logs:
>
> Dec 11 22:58:52 lucy kernel: Fwd DMZ->Internet DROP: IN=eth1 OUT=ppp0
> SRC=192.168.254.242 DST=204.157.6.223 LEN=60 TOS=0x00 PREC=0x00 TTL=63
> ID=56169 DF PROTO=TCP SPT=80 DPT=56319 WINDOW=32476 RES=0x00 ACK SYN URGP=0
>
Let me correct my previous post. It's coming from port 80 but it's not trying
to connect to port 80 so it's probably not a browser. Disregard, I need
sleep.
Jeff
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: Weird TCP flags?
2003-12-12 4:46 ` Jeffrey Laramie
@ 2003-12-12 4:52 ` Ian Hunter
2003-12-12 5:20 ` Jeffrey Laramie
0 siblings, 1 reply; 28+ messages in thread
From: Ian Hunter @ 2003-12-12 4:52 UTC (permalink / raw)
To: netfilter
There are no users on that box though, and it's not web browser traffic
because the SOURCE port is 80, not the dest port, so this is web traffic
being served by that box for sure...
I've seen references to doing --reply-with tcp-reject to this sort of
packet... ???
----- Original Message -----
From: "Jeffrey Laramie" <JALaramie@Loudoun-Fairfax.com>
To: "Ian Hunter" <ihunter@hunterweb.net>; <netfilter@lists.netfilter.org>
Sent: Thursday, December 11, 2003 11:46 PM
Subject: Re: Weird TCP flags?
> On Thursday 11 December 2003 23:11, Ian Hunter wrote:
> > OK, I have a router (lucy) with a webserver (192.168.254.242) in a DMZ
(off
> > eth1), and everything works fine -- when you hit my ip, you get the
site,
> > all is well. However, I get STORMS of this nonsense in my logs:
> >
> > Dec 11 22:58:52 lucy kernel: Fwd DMZ->Internet DROP: IN=eth1 OUT=ppp0
> > SRC=192.168.254.242 DST=204.157.6.223 LEN=60 TOS=0x00 PREC=0x00 TTL=63
> > ID=56169 DF PROTO=TCP SPT=80 DPT=56319 WINDOW=32476 RES=0x00 ACK SYN
URGP=0
> >
>
> These have ACK SYN set so I doubt they have anything to do with
established
> connections. I would guess that this is a browser on the webserver itself
> trying to connect to an outside site which your rules don't allow. In this
> case: bart.routesys.com.
>
> Jeff
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: Weird TCP flags?
2003-12-12 4:52 ` Ian Hunter
@ 2003-12-12 5:20 ` Jeffrey Laramie
0 siblings, 0 replies; 28+ messages in thread
From: Jeffrey Laramie @ 2003-12-12 5:20 UTC (permalink / raw)
To: Ian Hunter, netfilter
On Thursday 11 December 2003 23:52, Ian Hunter wrote:
> There are no users on that box though, and it's not web browser traffic
> because the SOURCE port is 80, not the dest port, so this is web traffic
> being served by that box for sure...
>
> I've seen references to doing --reply-with tcp-reject to this sort of
> packet... ???
Hmmm, I wouldn't expect a SYN/ACK reply to be on a high port. I'm more of a
rules and logging kind of guy so I'm afraid you'll need to wait until the
tcp/ip (i.e. smart) people wake up.
Jeff
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: Weird TCP flags?
[not found] ` <200312120105.01557.Alistair Tonner <>
@ 2003-12-12 12:26 ` John A. Sullivan III
2003-12-12 13:14 ` Ian Hunter
0 siblings, 1 reply; 28+ messages in thread
From: John A. Sullivan III @ 2003-12-12 12:26 UTC (permalink / raw)
To: Ian Hunter; +Cc: netfilter
On Fri, 2003-12-12 at 01:05, an unknown sender wrote:
> On December 11, 2003 11:11 pm, Ian Hunter wrote:
> > OK, I have a router (lucy) with a webserver (192.168.254.242) in a DMZ (off
> > eth1), and everything works fine -- when you hit my ip, you get the site,
> > all is well. However, I get STORMS of this nonsense in my logs:
> >
> > Dec 11 22:58:52 lucy kernel: Fwd DMZ->Internet DROP: IN=eth1 OUT=ppp0
> > SRC=192.168.254.242 DST=204.157.6.223 LEN=60 TOS=0x00 PREC=0x00 TTL=63
> > ID=56169 DF PROTO=TCP SPT=80 DPT=56319 WINDOW=32476 RES=0x00 ACK SYN URGP=0
>
> Perhaps this is some sort of spoofing you are seeing. -- at least at this
> moment bart.routesys.com (204.157.6.223) does NOT appear to be alive ...
> (you too that snap at 10:58 your time .. I'm looking at 12:30 my time..EST)
>
>
> >
> > (That's responding to these rules at the end of the FORWARD chain of filter
> > table:)
> >
> > iptables -A FORWARD -i eth1 -o ppp0 -j LOG --log-prefix "Fwd DMZ->Internet
> > DROP: "
> > iptables -A FORWARD -i eth1 -o ppp0 -j DROP
> >
> >
> > Now if web traffic is working, my "-m state --state ESTABLISHED,RELATED -j
> > ACCEPT" must be working on forwards from eth1 -> ppp0, so why am I seeing
> > these messages? Are these broken TCP connections that netfilter doesn't
> > consider established?
> >
> > Where do I look now?
> >
> > Thanks,
> >
> > Ian
Is there any chance that port 80 traffic is being allowed in under a
rule that does not enter the connection in conntrack?
In other words, if you have a rule such as:
iptables -A FORWARD -d 192.168.254.242 -p tcp --dport 80 -j ACCEPT
The SYN packet will be sent to your web server and the web server will
respond with a SYN, ACK packet but it will not be related to anything in
the conntrack table. The connection may be being allowed by a more
general rule than one to the web services.
So, I suppose we should ask what your ACCEPT rules look like - John
--
John A. Sullivan III
Chief Technology Officer
Nexus Management
+1 207-985-7880
john.sullivan@nexusmgmt.com
---
If you are interested in helping to develop a GPL enterprise class
VPN/Firewall/Security device management console, please visit
http://iscs.sourceforge.net
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: Weird TCP flags?
2003-12-12 4:11 Weird TCP flags? Ian Hunter
` (2 preceding siblings ...)
[not found] ` <200312120105.01557.Alistair Tonner <>
@ 2003-12-12 13:13 ` Chris Brenton
2003-12-12 13:26 ` Ian Hunter
2003-12-12 14:05 ` Jeffrey Laramie
2003-12-13 14:00 ` Akos Szalkai
4 siblings, 2 replies; 28+ messages in thread
From: Chris Brenton @ 2003-12-12 13:13 UTC (permalink / raw)
To: Ian Hunter; +Cc: netfilter
On Thu, 2003-12-11 at 23:11, Ian Hunter wrote:
>
> Dec 11 22:58:52 lucy kernel: Fwd DMZ->Internet DROP: IN=eth1 OUT=ppp0
> SRC=192.168.254.242 DST=204.157.6.223 LEN=60 TOS=0x00 PREC=0x00 TTL=63
> ID=56169 DF PROTO=TCP SPT=80 DPT=56319 WINDOW=32476 RES=0x00 ACK SYN URGP=0
My "guess" is, you are receiving a SYN packet from 204.157.6.223. This
creates a state table entry with with a 60 second timer. Your system is
taking longer than 60 seconds to respond, so iptables is removing the
state table entry. Your system then responds causing the log entry shown
above.
HTH,
C
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: Weird TCP flags?
2003-12-12 12:26 ` John A. Sullivan III
@ 2003-12-12 13:14 ` Ian Hunter
2003-12-12 13:32 ` John A. Sullivan III
0 siblings, 1 reply; 28+ messages in thread
From: Ian Hunter @ 2003-12-12 13:14 UTC (permalink / raw)
To: netfilter
> > > Dec 11 22:58:52 lucy kernel: Fwd DMZ->Internet DROP: IN=eth1 OUT=ppp0
> > > SRC=192.168.254.242 DST=204.157.6.223 LEN=60 TOS=0x00 PREC=0x00 TTL=63
> > > ID=56169 DF PROTO=TCP SPT=80 DPT=56319 WINDOW=32476 RES=0x00 ACK SYN
URGP=0
> >
> Is there any chance that port 80 traffic is being allowed in under a
> rule that does not enter the connection in conntrack?
> In other words, if you have a rule such as:
>
> iptables -A FORWARD -d 192.168.254.242 -p tcp --dport 80 -j ACCEPT
>
> The SYN packet will be sent to your web server and the web server will
> respond with a SYN, ACK packet but it will not be related to anything in
> the conntrack table. The connection may be being allowed by a more
> general rule than one to the web services.
>
> So, I suppose we should ask what your ACCEPT rules look like - John
I'm absolutely doing that -- here are the relevant ACCEPTs:
iptables -A FORWARD -i ppp0 -o eth1 -p tcp --dport 80 -d 192.168.254.242 -j
ACCEPT
iptables -t nat -A PREROUTING -i ppp0 -d <my.public.ip.here> -p tcp --dport
80 -j DNAT --to-destination 192.168.254.242
If I want to forward port 80 to that box, isn't that the way to do it? How
would a connection escape conntrack?
Thanks MUCH!
Ian
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: Weird TCP flags?
2003-12-12 13:13 ` Chris Brenton
@ 2003-12-12 13:26 ` Ian Hunter
2003-12-12 14:05 ` Jeffrey Laramie
1 sibling, 0 replies; 28+ messages in thread
From: Ian Hunter @ 2003-12-12 13:26 UTC (permalink / raw)
To: netfilter
I'll log all web traffic in detail and see about the timings... thanks!
> My "guess" is, you are receiving a SYN packet from 204.157.6.223. This
> creates a state table entry with with a 60 second timer. Your system is
> taking longer than 60 seconds to respond, so iptables is removing the
> state table entry. Your system then responds causing the log entry shown
> above.
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: Weird TCP flags?
2003-12-12 13:14 ` Ian Hunter
@ 2003-12-12 13:32 ` John A. Sullivan III
2003-12-12 13:59 ` Antony Stone
0 siblings, 1 reply; 28+ messages in thread
From: John A. Sullivan III @ 2003-12-12 13:32 UTC (permalink / raw)
To: Ian Hunter; +Cc: netfilter
On Fri, 2003-12-12 at 08:14, Ian Hunter wrote:
> > > > Dec 11 22:58:52 lucy kernel: Fwd DMZ->Internet DROP: IN=eth1 OUT=ppp0
> > > > SRC=192.168.254.242 DST=204.157.6.223 LEN=60 TOS=0x00 PREC=0x00 TTL=63
> > > > ID=56169 DF PROTO=TCP SPT=80 DPT=56319 WINDOW=32476 RES=0x00 ACK SYN
> URGP=0
> > >
> > Is there any chance that port 80 traffic is being allowed in under a
> > rule that does not enter the connection in conntrack?
> > In other words, if you have a rule such as:
> >
> > iptables -A FORWARD -d 192.168.254.242 -p tcp --dport 80 -j ACCEPT
> >
> > The SYN packet will be sent to your web server and the web server will
> > respond with a SYN, ACK packet but it will not be related to anything in
> > the conntrack table. The connection may be being allowed by a more
> > general rule than one to the web services.
> >
> > So, I suppose we should ask what your ACCEPT rules look like - John
>
> I'm absolutely doing that -- here are the relevant ACCEPTs:
>
> iptables -A FORWARD -i ppp0 -o eth1 -p tcp --dport 80 -d 192.168.254.242 -j
> ACCEPT
> iptables -t nat -A PREROUTING -i ppp0 -d <my.public.ip.here> -p tcp --dport
> 80 -j DNAT --to-destination 192.168.254.242
>
> If I want to forward port 80 to that box, isn't that the way to do it? How
> would a connection escape conntrack?
>
> Thanks MUCH!
>
> Ian
You never told it to make an entry in conntrack. Try this:
iptables -A FORWARD -i ppp0 -o eth1 -p tcp --dport 80 -d 192.168.254.242
-m state --state NEW -j
ACCEPT
--
John A. Sullivan III
Chief Technology Officer
Nexus Management
+1 207-985-7880
john.sullivan@nexusmgmt.com
---
If you are interested in helping to develop a GPL enterprise class
VPN/Firewall/Security device management console, please visit
http://iscs.sourceforge.net
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: Weird TCP flags?
2003-12-12 14:05 ` Jeffrey Laramie
@ 2003-12-12 13:57 ` Antony Stone
2003-12-12 14:21 ` John A. Sullivan III
2003-12-12 14:26 ` Ralf Spenneberg
2 siblings, 0 replies; 28+ messages in thread
From: Antony Stone @ 2003-12-12 13:57 UTC (permalink / raw)
To: netfilter
On Friday 12 December 2003 2:05 pm, Jeffrey Laramie wrote:
> On Friday 12 December 2003 08:13, Chris Brenton wrote:
> > On Thu, 2003-12-11 at 23:11, Ian Hunter wrote:
> > > Dec 11 22:58:52 lucy kernel: Fwd DMZ->Internet DROP: IN=eth1 OUT=ppp0
> > > SRC=192.168.254.242 DST=204.157.6.223 LEN=60 TOS=0x00 PREC=0x00 TTL=63
> > > ID=56169 DF PROTO=TCP SPT=80 DPT=56319 WINDOW=32476 RES=0x00 ACK SYN
> > > URGP=0
> >
> > My "guess" is, you are receiving a SYN packet from 204.157.6.223. This
> > creates a state table entry with with a 60 second timer. Your system is
> > taking longer than 60 seconds to respond, so iptables is removing the
> > state table entry. Your system then responds causing the log entry shown
> > above.
>
> Hey Chris,
>
> Is it normal for the server to send the ACK SYN to a high dport? I wouldn't
> have expected that.
Yes - the server will respond to whatever port number the client chose to send
from. This is almost always >1024, and 56319 is perfectly reasonable.
Antony.
--
90% of networking problems are routing problems.
9 of the remaining 10% are routing problems in the other direction.
The remaining 1% might be something else, but check the routing anyway.
Please reply to the list;
please don't CC me.
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: Weird TCP flags?
2003-12-12 13:32 ` John A. Sullivan III
@ 2003-12-12 13:59 ` Antony Stone
2003-12-12 14:50 ` John A. Sullivan III
2003-12-12 16:37 ` Ian Hunter
0 siblings, 2 replies; 28+ messages in thread
From: Antony Stone @ 2003-12-12 13:59 UTC (permalink / raw)
To: netfilter
On Friday 12 December 2003 1:32 pm, John A. Sullivan III wrote:
> > > The SYN packet will be sent to your web server and the web server will
> > > respond with a SYN, ACK packet but it will not be related to anything
> > > in the conntrack table. The connection may be being allowed by a more
> > > general rule than one to the web services.
> > >
> > > So, I suppose we should ask what your ACCEPT rules look like - John
> >
> > I'm absolutely doing that -- here are the relevant ACCEPTs:
> >
> > iptables -A FORWARD -i ppp0 -o eth1 -p tcp --dport 80 -d 192.168.254.242
> > -j ACCEPT
> > iptables -t nat -A PREROUTING -i ppp0 -d <my.public.ip.here> -p tcp
> > --dport 80 -j DNAT --to-destination 192.168.254.242
> >
> > If I want to forward port 80 to that box, isn't that the way to do it?
> > How would a connection escape conntrack?
> You never told it to make an entry in conntrack. Try this:
> iptables -A FORWARD -i ppp0 -o eth1 -p tcp --dport 80 -d 192.168.254.242
> -m state --state NEW -j ACCEPT
I disagree with this.
Making the rule more explicit by matching state NEW is not a bad idea, but it
has nothing to do with whether a conntrack table entry gets created or not.
If you have compiled in connection tracking, or loaded the module, then all
connections which are ACCEPTed will result in a conntrack table entry.
There's no way to stop that happening on a connection-by-connection basis.
Antony.
--
Documentation is like sex.
When it's good, it's very very good.
When it's bad, it's still better than nothing.
Please reply to the list;
please don't CC me.
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: Weird TCP flags?
2003-12-12 13:13 ` Chris Brenton
2003-12-12 13:26 ` Ian Hunter
@ 2003-12-12 14:05 ` Jeffrey Laramie
2003-12-12 13:57 ` Antony Stone
` (2 more replies)
1 sibling, 3 replies; 28+ messages in thread
From: Jeffrey Laramie @ 2003-12-12 14:05 UTC (permalink / raw)
To: netfilter
On Friday 12 December 2003 08:13, Chris Brenton wrote:
> On Thu, 2003-12-11 at 23:11, Ian Hunter wrote:
> > Dec 11 22:58:52 lucy kernel: Fwd DMZ->Internet DROP: IN=eth1 OUT=ppp0
> > SRC=192.168.254.242 DST=204.157.6.223 LEN=60 TOS=0x00 PREC=0x00 TTL=63
> > ID=56169 DF PROTO=TCP SPT=80 DPT=56319 WINDOW=32476 RES=0x00 ACK SYN
> > URGP=0
>
> My "guess" is, you are receiving a SYN packet from 204.157.6.223. This
> creates a state table entry with with a 60 second timer. Your system is
> taking longer than 60 seconds to respond, so iptables is removing the
> state table entry. Your system then responds causing the log entry shown
> above.
>
Hey Chris,
Is it normal for the server to send the ACK SYN to a high dport? I wouldn't
have expected that.
Jeff
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: Weird TCP flags?
2003-12-12 14:05 ` Jeffrey Laramie
2003-12-12 13:57 ` Antony Stone
@ 2003-12-12 14:21 ` John A. Sullivan III
2003-12-12 14:26 ` Ralf Spenneberg
2 siblings, 0 replies; 28+ messages in thread
From: John A. Sullivan III @ 2003-12-12 14:21 UTC (permalink / raw)
To: Jeffrey Laramie; +Cc: netfilter
On Fri, 2003-12-12 at 09:05, Jeffrey Laramie wrote:
> On Friday 12 December 2003 08:13, Chris Brenton wrote:
> > On Thu, 2003-12-11 at 23:11, Ian Hunter wrote:
> > > Dec 11 22:58:52 lucy kernel: Fwd DMZ->Internet DROP: IN=eth1 OUT=ppp0
> > > SRC=192.168.254.242 DST=204.157.6.223 LEN=60 TOS=0x00 PREC=0x00 TTL=63
> > > ID=56169 DF PROTO=TCP SPT=80 DPT=56319 WINDOW=32476 RES=0x00 ACK SYN
> > > URGP=0
> >
> > My "guess" is, you are receiving a SYN packet from 204.157.6.223. This
> > creates a state table entry with with a 60 second timer. Your system is
> > taking longer than 60 seconds to respond, so iptables is removing the
> > state table entry. Your system then responds causing the log entry shown
> > above.
> >
>
> Hey Chris,
>
> Is it normal for the server to send the ACK SYN to a high dport? I wouldn't
> have expected that.
>
> Jeff
It will respond to whatever port the user chose we they initiated the
connection. That should indeed be a high port - John
--
John A. Sullivan III
Chief Technology Officer
Nexus Management
+1 207-985-7880
john.sullivan@nexusmgmt.com
---
If you are interested in helping to develop a GPL enterprise class
VPN/Firewall/Security device management console, please visit
http://iscs.sourceforge.net
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: Weird TCP flags?
2003-12-12 14:05 ` Jeffrey Laramie
2003-12-12 13:57 ` Antony Stone
2003-12-12 14:21 ` John A. Sullivan III
@ 2003-12-12 14:26 ` Ralf Spenneberg
2003-12-12 14:41 ` Jeffrey Laramie
2 siblings, 1 reply; 28+ messages in thread
From: Ralf Spenneberg @ 2003-12-12 14:26 UTC (permalink / raw)
To: Jeffrey Laramie; +Cc: Netfilter
Am Fre, 2003-12-12 um 15.05 schrieb Jeffrey Laramie:
> On Friday 12 December 2003 08:13, Chris Brenton wrote:
> > On Thu, 2003-12-11 at 23:11, Ian Hunter wrote:
> > > Dec 11 22:58:52 lucy kernel: Fwd DMZ->Internet DROP: IN=eth1 OUT=ppp0
> > > SRC=192.168.254.242 DST=204.157.6.223 LEN=60 TOS=0x00 PREC=0x00 TTL=63
> > > ID=56169 DF PROTO=TCP SPT=80 DPT=56319 WINDOW=32476 RES=0x00 ACK SYN
> > > URGP=0
> >
> > My "guess" is, you are receiving a SYN packet from 204.157.6.223. This
> > creates a state table entry with with a 60 second timer. Your system is
> > taking longer than 60 seconds to respond, so iptables is removing the
> > state table entry. Your system then responds causing the log entry shown
> > above.
> >
>
> Hey Chris,
>
> Is it normal for the server to send the ACK SYN to a high dport? I wouldn't
> have expected that.
Yes, of course:
client:56319 -SYN-> server:80
client:56319 <-ACK/SYN- server:80
client:56319 -ACK-> server:80
Connection established.
Cheers,
Ralf
--
Ralf Spenneberg
RHCE, RHCX
Book: VPN mit Linux
Book: Intrusion Detection für Linux Server http://www.spenneberg.com
IPsec-Howto http://www.ipsec-howto.org
Honeynet Project Mirror: http://honeynet.spenneberg.org
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: Weird TCP flags?
2003-12-12 14:50 ` John A. Sullivan III
@ 2003-12-12 14:37 ` Antony Stone
0 siblings, 0 replies; 28+ messages in thread
From: Antony Stone @ 2003-12-12 14:37 UTC (permalink / raw)
To: netfilter
On Friday 12 December 2003 2:50 pm, John A. Sullivan III wrote:
> On Fri, 2003-12-12 at 08:59, Antony Stone wrote:
>
> > I disagree with this.
> >
> > Making the rule more explicit by matching state NEW is not a bad idea,
> > but it has nothing to do with whether a conntrack table entry gets
> > created or not. If you have compiled in connection tracking, or loaded
> > the module, then all connections which are ACCEPTed will result in a
> > conntrack table entry. There's no way to stop that happening on a
> > connection-by-connection basis.
> I'm sure you know better than I. I always assumed that NEW was
> necessary before ESTABLISHED and RELATED were meaningful but I have
> never cracked open the netfilter code to confirm that assumption. Thank
> you for the correction - John
> PS - what, then, is the purpose of NEW? Is it just for identifying new
> connections for some other purpose than access control such as logging?
Some people like to specify NEW in their rules (and some people also like to
specify the TCP flags, to make sure only the SYN flag is set, and not
something silly like FIN or RST as well), just to be as specific as possible
about what is being allowed.
In my opinion, specifying NEW state for FORWARDING rules, when you already
have a general-purpose ESTABLISHED,RELATED rule doesn't add any extra
security to your ruleset, but nor does it do any harm. It's only ever going
to match the first packet of a connection, as well, so there's no significant
performance impact.
Antony.
--
Success is a lousy teacher. It seduces smart people into thinking they can't
lose.
- William H Gates III
Please reply to the list;
please don't CC me.
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: Weird TCP flags?
2003-12-12 14:41 ` Jeffrey Laramie
@ 2003-12-12 14:38 ` Antony Stone
2003-12-12 16:05 ` Jeffrey Laramie
0 siblings, 1 reply; 28+ messages in thread
From: Antony Stone @ 2003-12-12 14:38 UTC (permalink / raw)
To: Netfilter
On Friday 12 December 2003 2:41 pm, Jeffrey Laramie wrote:
> Duh! Boy do I feel like an idiot. Last night I blamed it on sleeplessness.
> Today I'll blame it on coffee. Yup, that's it, I haven't had enough coffee
> yet.
Time for a weekend, perhaps, Jeff?
Antony.
--
The difference between theory and practice is that in theory there is no
difference, whereas in practice there is.
Please reply to the list;
please don't CC me.
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: Weird TCP flags?
2003-12-12 14:26 ` Ralf Spenneberg
@ 2003-12-12 14:41 ` Jeffrey Laramie
2003-12-12 14:38 ` Antony Stone
0 siblings, 1 reply; 28+ messages in thread
From: Jeffrey Laramie @ 2003-12-12 14:41 UTC (permalink / raw)
To: Netfilter
On Friday 12 December 2003 09:26, Ralf Spenneberg wrote:
> Am Fre, 2003-12-12 um 15.05 schrieb Jeffrey Laramie:
> > On Friday 12 December 2003 08:13, Chris Brenton wrote:
> > > On Thu, 2003-12-11 at 23:11, Ian Hunter wrote:
> > > > Dec 11 22:58:52 lucy kernel: Fwd DMZ->Internet DROP: IN=eth1 OUT=ppp0
> > > > SRC=192.168.254.242 DST=204.157.6.223 LEN=60 TOS=0x00 PREC=0x00
> > > > TTL=63 ID=56169 DF PROTO=TCP SPT=80 DPT=56319 WINDOW=32476 RES=0x00
> > > > ACK SYN URGP=0
> > >
> > > My "guess" is, you are receiving a SYN packet from 204.157.6.223. This
> > > creates a state table entry with with a 60 second timer. Your system is
> > > taking longer than 60 seconds to respond, so iptables is removing the
> > > state table entry. Your system then responds causing the log entry
> > > shown above.
> >
> > Hey Chris,
> >
> > Is it normal for the server to send the ACK SYN to a high dport? I
> > wouldn't have expected that.
>
> Yes, of course:
>
> client:56319 -SYN-> server:80
> client:56319 <-ACK/SYN- server:80
> client:56319 -ACK-> server:80
> Connection established.
>
> Cheers,
>
> Ralf
Duh! Boy do I feel like an idiot. Last night I blamed it on sleeplessness.
Today I'll blame it on coffee. Yup, that's it, I haven't had enough coffee
yet.
Thanks Guys!
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: Weird TCP flags?
2003-12-12 13:59 ` Antony Stone
@ 2003-12-12 14:50 ` John A. Sullivan III
2003-12-12 14:37 ` Antony Stone
2003-12-12 16:37 ` Ian Hunter
1 sibling, 1 reply; 28+ messages in thread
From: John A. Sullivan III @ 2003-12-12 14:50 UTC (permalink / raw)
To: Antony Stone; +Cc: netfilter
On Fri, 2003-12-12 at 08:59, Antony Stone wrote:
> On Friday 12 December 2003 1:32 pm, John A. Sullivan III wrote:
>
> > > > The SYN packet will be sent to your web server and the web server will
> > > > respond with a SYN, ACK packet but it will not be related to anything
> > > > in the conntrack table. The connection may be being allowed by a more
> > > > general rule than one to the web services.
> > > >
> > > > So, I suppose we should ask what your ACCEPT rules look like - John
> > >
> > > I'm absolutely doing that -- here are the relevant ACCEPTs:
> > >
> > > iptables -A FORWARD -i ppp0 -o eth1 -p tcp --dport 80 -d 192.168.254.242
> > > -j ACCEPT
> > > iptables -t nat -A PREROUTING -i ppp0 -d <my.public.ip.here> -p tcp
> > > --dport 80 -j DNAT --to-destination 192.168.254.242
> > >
> > > If I want to forward port 80 to that box, isn't that the way to do it?
> > > How would a connection escape conntrack?
>
> > You never told it to make an entry in conntrack. Try this:
> > iptables -A FORWARD -i ppp0 -o eth1 -p tcp --dport 80 -d 192.168.254.242
> > -m state --state NEW -j ACCEPT
>
> I disagree with this.
>
> Making the rule more explicit by matching state NEW is not a bad idea, but it
> has nothing to do with whether a conntrack table entry gets created or not.
> If you have compiled in connection tracking, or loaded the module, then all
> connections which are ACCEPTed will result in a conntrack table entry.
> There's no way to stop that happening on a connection-by-connection basis.
>
> Antony.
I'm sure you know better than I. I always assumed that NEW was
necessary before ESTABLISHED and RELATED were meaningful but I have
never cracked open the netfilter code to confirm that assumption. Thank
you for the correction - John
PS - what, then, is the purpose of NEW? Is it just for identifying new
connections for some other purpose than access control such as logging?
--
John A. Sullivan III
Chief Technology Officer
Nexus Management
+1 207-985-7880
john.sullivan@nexusmgmt.com
---
If you are interested in helping to develop a GPL enterprise class
VPN/Firewall/Security device management console, please visit
http://iscs.sourceforge.net
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: Weird TCP flags?
2003-12-12 14:38 ` Antony Stone
@ 2003-12-12 16:05 ` Jeffrey Laramie
2003-12-12 17:41 ` Chris Brenton
2003-12-12 21:21 ` Antony Stone
0 siblings, 2 replies; 28+ messages in thread
From: Jeffrey Laramie @ 2003-12-12 16:05 UTC (permalink / raw)
To: Netfilter
On Friday 12 December 2003 09:38, Antony Stone wrote:
> On Friday 12 December 2003 2:41 pm, Jeffrey Laramie wrote:
> > Duh! Boy do I feel like an idiot. Last night I blamed it on
> > sleeplessness. Today I'll blame it on coffee. Yup, that's it, I haven't
> > had enough coffee yet.
>
> Time for a weekend, perhaps, Jeff?
>
Yeah, I definitely need a vacation. Next time I post something stupid like
that *smack* me and say "Shut Up Jeff, let me answer this!"
The TTL of his packet was about 60 secs lower than what I usually see which
makes Chris's explanation sound likely. How does TTL get calculated? Does
each host set a new TTL for it's packets or does it calculate it from the
time of the original SYN packet? In this situation which host is slow
responding, the server?
Jeff
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: Weird TCP flags?
2003-12-12 13:59 ` Antony Stone
2003-12-12 14:50 ` John A. Sullivan III
@ 2003-12-12 16:37 ` Ian Hunter
1 sibling, 0 replies; 28+ messages in thread
From: Ian Hunter @ 2003-12-12 16:37 UTC (permalink / raw)
To: netfilter
That's in line with what I thought. So I'm back to not having an
explanation for why some packets would get rejected as not part of an
existing connection.
So at this point best bet is >60 second delay somewhere? That doesn't seem
likely... if I log all port 80 traffic, and wait until I see another drop, I
should be able to correlate the two and verify that. Is that my best
option?
----- Original Message -----
From: "Antony Stone" <Antony@Soft-Solutions.co.uk>
To: <netfilter@lists.netfilter.org>
Sent: Friday, December 12, 2003 8:59 AM
Subject: Re: Weird TCP flags?
> > > iptables -A FORWARD -i ppp0 -o eth1 -p tcp --dport 80 -d
192.168.254.242
> > > -j ACCEPT
> > > iptables -t nat -A PREROUTING -i ppp0 -d <my.public.ip.here> -p tcp
> > > --dport 80 -j DNAT --to-destination 192.168.254.242
> > >
> > > If I want to forward port 80 to that box, isn't that the way to do it?
> > > How would a connection escape conntrack?
>
> > You never told it to make an entry in conntrack. Try this:
> > iptables -A FORWARD -i ppp0 -o eth1 -p tcp --dport 80 -d 192.168.254.242
> > -m state --state NEW -j ACCEPT
>
> I disagree with this.
>
> Making the rule more explicit by matching state NEW is not a bad idea, but
it
> has nothing to do with whether a conntrack table entry gets created or
not.
> If you have compiled in connection tracking, or loaded the module, then
all
> connections which are ACCEPTed will result in a conntrack table entry.
> There's no way to stop that happening on a connection-by-connection basis.
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: Weird TCP flags?
2003-12-12 16:05 ` Jeffrey Laramie
@ 2003-12-12 17:41 ` Chris Brenton
2003-12-12 21:21 ` Antony Stone
1 sibling, 0 replies; 28+ messages in thread
From: Chris Brenton @ 2003-12-12 17:41 UTC (permalink / raw)
To: Jeffrey Laramie; +Cc: netfilter
On Fri, 2003-12-12 at 11:05, Jeffrey Laramie wrote:
>
> Yeah, I definitely need a vacation. Next time I post something stupid like
> that *smack* me and say "Shut Up Jeff, let me answer this!"
Be careful what you wish for... ;-)
> The TTL of his packet was about 60 secs lower than what I usually see which
> makes Chris's explanation sound likely. How does TTL get calculated?
Actually, the TTL (63) is dead on if his server is running on Linux or
BSD. Based on the window and packet size, I would guess Linux.
Per the RFC, a host is to "suppose" to decrement the TTL by 1 for every
hop crossed, as well as every second the packet is queued. So for
example a router holding a packet for 5 seconds prior to transmitting
should decrement the TTL by 6.
In reality, most/all devices just decrement by one for hop and don't
look at the time component. IMHO this is useful for us as as a community
as it makes it easier to use TTL for passively fingerprinting packets.
HTH,
C
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: Weird TCP flags?
2003-12-12 16:05 ` Jeffrey Laramie
2003-12-12 17:41 ` Chris Brenton
@ 2003-12-12 21:21 ` Antony Stone
1 sibling, 0 replies; 28+ messages in thread
From: Antony Stone @ 2003-12-12 21:21 UTC (permalink / raw)
To: Netfilter
On Friday 12 December 2003 4:05 pm, Jeffrey Laramie wrote:
> The TTL of his packet was about 60 secs lower than what I usually see which
> makes Chris's explanation sound likely. How does TTL get calculated? Does
> each host set a new TTL for it's packets or does it calculate it from the
> time of the original SYN packet?
No, TTL doesn't mean "TIME to live" any more (it did when the Internet was
being invented, but pretty soon the meaning changed and TTL no longer has
anything to do with time).
TTL is actually a "maximum hop count", specifying the number of routers (and
remember, a netfilter firewall is a router too) a packet is allowed to pass
through before it "expires" and gets thrown away (with an ICMP "TTL expired"
packet being returned to the source address on the packet which just
expired).
The easiest way (I think) to explain how this is used is to consider the
traceroute program and what it does:
traceroute sends out three packets to some destination address with TTL=1.
The first router they come to decrements the TTL (all routers do this) and if
the TTL is now = 0, the router discards the packet, and returns an ICMP "TTL
expired" to the source address (ie: the machine which started the traceroute
command). The source address for this ICMP packet is that of the router,
which is how the traceroute program tells you the address (and, by reverse
DNS, also the name) of the first router the packet went through.
Then traceroute sends three packets to the same destination address, this time
with TTL = 2.
The first router decrements the TTL and passes the packets on (because that's
what routers do, and TTL is not = 0).
The second router decrements TTL, finds that it now = 0, and responds with the
ICMP packet (this time from the source address of the second router, so
traceroute can tell you that, too).
traceroute then just keeps on sending out packets with higher and higher TTLs,
waiting for all the ICMP responses to come back, and successive packets get
further and further, until traceroute receives either a "port unreachable" or
a genuine response packet, which means that the packets reached the real
destination. Then traceroute stops, and you have a record of all the
routers the packets went through on their way to the destination.
traceroute normally sends UDP packets to port numbers which are unlikely to be
listening (therefore the expected response from the final destination is ICMP
port unreachable).
Note that due to dynamic routing across the Internet, there's no guarantee
that every packet will follow the same route on its way to the destination,
so traceroute outputs can be misleading, but it's a good start.
Hope this helps explain a bit about TTL.....
Antony.
--
The truth is rarely pure, and never simple.
- Oscar Wilde
Please reply to the list;
please don't CC me.
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: Weird TCP flags?
2003-12-12 4:11 Weird TCP flags? Ian Hunter
` (3 preceding siblings ...)
2003-12-12 13:13 ` Chris Brenton
@ 2003-12-13 14:00 ` Akos Szalkai
2003-12-13 14:41 ` Antony Stone
4 siblings, 1 reply; 28+ messages in thread
From: Akos Szalkai @ 2003-12-13 14:00 UTC (permalink / raw)
To: Ian Hunter; +Cc: netfilter
> Dec 11 22:58:52 lucy kernel: Fwd DMZ->Internet DROP: IN=eth1 OUT=ppp0
> SRC=192.168.254.242 DST=204.157.6.223 LEN=60 TOS=0x00 PREC=0x00 TTL=63
> ID=56169 DF PROTO=TCP SPT=80 DPT=56319 WINDOW=32476 RES=0x00 ACK SYN URGP=0
I have also been seeing this kind of logs on our web server, and this is
my explanation. Probably it is an incoming HTTP connection (ie. SYN
packet) from a spoofed source address (204.157.6.223 in this case).
Your web server sends a syn-ack response, which the firewall in front of
204.157.6.223 silently drops (since it never saw the first packet of the
connection). Therefore your web server retransmits and keeps on trying
for at least the RFC-minimum 180 seconds. However, after 60 seconds
this kind of half-open connection is deleted from the connection table
on your firewall and then the syn-acks start hitting your "Fwd
DMZ->Internet DROP:" rule.
The scenario may be different (eg. no spoofing but some kind of
misconfiguration, etc.) but I think this is the reason. You can verify
this with tcpdump (which I have not bothered to do yet...)
Regards,
Akos
--
Akos Szalkai <szalkai@2f.hu>
IT Consultant, CISA
2F 2000 Szamitastechnikai es Szolgaltato Kft.
Tel: (+36-1)-4887700 Fax: (+36-1)-4887709 WWW: http://www.2f.hu/
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: Weird TCP flags?
2003-12-13 14:00 ` Akos Szalkai
@ 2003-12-13 14:41 ` Antony Stone
2003-12-13 14:50 ` Antony Stone
2003-12-13 14:53 ` Akos Szalkai
0 siblings, 2 replies; 28+ messages in thread
From: Antony Stone @ 2003-12-13 14:41 UTC (permalink / raw)
To: netfilter
On Saturday 13 December 2003 2:00 pm, Akos Szalkai wrote:
> > Dec 11 22:58:52 lucy kernel: Fwd DMZ->Internet DROP: IN=eth1 OUT=ppp0
> > SRC=192.168.254.242 DST=204.157.6.223 LEN=60 TOS=0x00 PREC=0x00 TTL=63
> > ID=56169 DF PROTO=TCP SPT=80 DPT=56319 WINDOW=32476 RES=0x00 ACK SYN
> > URGP=0
>
> I have also been seeing this kind of logs on our web server, and this is
> my explanation. Probably it is an incoming HTTP connection (ie. SYN
> packet) from a spoofed source address (204.157.6.223 in this case).
> Your web server sends a syn-ack response, which the firewall in front of
> 204.157.6.223 silently drops (since it never saw the first packet of the
> connection).
If the firewall never saw the first packet, how did it get to the web server?
It doesn't matter whether the source IP is spoofed or not, the only way for a
packet to get to the web server should be through the firewall, and both will
see the same source address, neither knows whether it's genuine or not.
Antony.
--
There are two possible outcomes:
If the result confirms the hypothesis, then you've made a measurement.
If the result is contrary to the hypothesis, then you've made a discovery.
- Enrico Fermi
Please reply to the list;
please don't CC me.
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: Weird TCP flags?
2003-12-13 14:41 ` Antony Stone
@ 2003-12-13 14:50 ` Antony Stone
2003-12-13 14:57 ` Akos Szalkai
2003-12-13 14:53 ` Akos Szalkai
1 sibling, 1 reply; 28+ messages in thread
From: Antony Stone @ 2003-12-13 14:50 UTC (permalink / raw)
To: netfilter
On Saturday 13 December 2003 2:41 pm, Antony Stone wrote:
> On Saturday 13 December 2003 2:00 pm, Akos Szalkai wrote:
> > > Dec 11 22:58:52 lucy kernel: Fwd DMZ->Internet DROP: IN=eth1 OUT=ppp0
> > > SRC=192.168.254.242 DST=204.157.6.223 LEN=60 TOS=0x00 PREC=0x00 TTL=63
> > > ID=56169 DF PROTO=TCP SPT=80 DPT=56319 WINDOW=32476 RES=0x00 ACK SYN
> > > URGP=0
> >
> > I have also been seeing this kind of logs on our web server, and this is
> > my explanation. Probably it is an incoming HTTP connection (ie. SYN
> > packet) from a spoofed source address (204.157.6.223 in this case).
> > Your web server sends a syn-ack response, which the firewall in front of
> > 204.157.6.223 silently drops (since it never saw the first packet of the
> > connection).
>
> If the firewall never saw the first packet, how did it get to the web
> server?
Sorry - I misread your posting at first - I realise now you were saying that
the firewall in front of the spoofed address never saw the first packet, so
it drops the second one.
However, the above log entry is from the firewall in front of the web server -
as far as it is concerned, it saw the first packet, and it saw the second
packet. I'm not sure there's an explanation yet for why it decided to drop
and log the second packet.
Antony.
--
If you want to be happy for an hour, get drunk.
If you want to be happy for a year, get married.
If you want to be happy for a lifetime, get a garden.
Please reply to the list;
please don't CC me.
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: Weird TCP flags?
2003-12-13 14:41 ` Antony Stone
2003-12-13 14:50 ` Antony Stone
@ 2003-12-13 14:53 ` Akos Szalkai
1 sibling, 0 replies; 28+ messages in thread
From: Akos Szalkai @ 2003-12-13 14:53 UTC (permalink / raw)
To: netfilter
> > Your web server sends a syn-ack response, which the firewall in front of
> > 204.157.6.223 silently drops (since it never saw the first packet of the
> > connection).
>
> If the firewall never saw the first packet, how did it get to the web server?
You probably misread my mail.
It is not the firewall in front of the webserver, but at the other end,
in front of the (probably spoofed) 204.157.6.223 machine. It does not
have to see the syn packet, but it will see the syn-ack coming back.
Akos
--
Akos Szalkai <szalkai@2f.hu>
IT Consultant, CISA
2F 2000 Szamitastechnikai es Szolgaltato Kft.
Tel: (+36-1)-4887700 Fax: (+36-1)-4887709 WWW: http://www.2f.hu/
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: Weird TCP flags?
2003-12-13 14:50 ` Antony Stone
@ 2003-12-13 14:57 ` Akos Szalkai
0 siblings, 0 replies; 28+ messages in thread
From: Akos Szalkai @ 2003-12-13 14:57 UTC (permalink / raw)
To: netfilter
> Sorry - I misread your posting at first - I realise now you were saying that
> the firewall in front of the spoofed address never saw the first packet, so
> it drops the second one.
Sorry, probably my writing is not clear enough...
> However, the above log entry is from the firewall in front of the web server -
> as far as it is concerned, it saw the first packet, and it saw the second
> packet. I'm not sure there's an explanation yet for why it decided to drop
> and log the second packet.
I am afraid now you did not misread my posting. You simply did not read
the rest of it, because it is in there. Summary: it drops the
retransmitted syn-acks after 60 secs.
Akos
--
Akos Szalkai <szalkai@2f.hu>
IT Consultant, CISA
2F 2000 Szamitastechnikai es Szolgaltato Kft.
Tel: (+36-1)-4887700 Fax: (+36-1)-4887709 WWW: http://www.2f.hu/
^ permalink raw reply [flat|nested] 28+ messages in thread
end of thread, other threads:[~2003-12-13 14:57 UTC | newest]
Thread overview: 28+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2003-12-12 4:11 Weird TCP flags? Ian Hunter
2003-12-12 4:46 ` Jeffrey Laramie
2003-12-12 4:52 ` Ian Hunter
2003-12-12 5:20 ` Jeffrey Laramie
2003-12-12 4:51 ` Jeffrey Laramie
[not found] ` <200312120105.01557.Alistair Tonner <>
2003-12-12 12:26 ` John A. Sullivan III
2003-12-12 13:14 ` Ian Hunter
2003-12-12 13:32 ` John A. Sullivan III
2003-12-12 13:59 ` Antony Stone
2003-12-12 14:50 ` John A. Sullivan III
2003-12-12 14:37 ` Antony Stone
2003-12-12 16:37 ` Ian Hunter
2003-12-12 13:13 ` Chris Brenton
2003-12-12 13:26 ` Ian Hunter
2003-12-12 14:05 ` Jeffrey Laramie
2003-12-12 13:57 ` Antony Stone
2003-12-12 14:21 ` John A. Sullivan III
2003-12-12 14:26 ` Ralf Spenneberg
2003-12-12 14:41 ` Jeffrey Laramie
2003-12-12 14:38 ` Antony Stone
2003-12-12 16:05 ` Jeffrey Laramie
2003-12-12 17:41 ` Chris Brenton
2003-12-12 21:21 ` Antony Stone
2003-12-13 14:00 ` Akos Szalkai
2003-12-13 14:41 ` Antony Stone
2003-12-13 14:50 ` Antony Stone
2003-12-13 14:57 ` Akos Szalkai
2003-12-13 14:53 ` Akos Szalkai
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.