How might incoming SMB probes from public IPs be ariving on the internal interfaces?

How might incoming SMB probes from public IPs be ariving on the internal interfaces?

[Whit Blauvelt]

Hi,

We've been running iptables v1.3.8 happily for a few years, and just
recently noticed that someone was managing to get probes to a Samba server
(which sits on the firewall) through the firewall. Since the Samba server
only allows logins from LAN addresses, the attempted connections only have
been refused. Still, Samba seeing the requests at all what we expected. This
was a brand new thing in the last few weeks. (We keep our logs for a _long_
time.)

After adding extra rules to make double sure that the smbd/nmbd ports were
blocked on the external interfaces, there were still failed logins happening
from external IPs. So I blocked everything on the smbd/nmbd ports regardless
of interface. That did the trick. But it leaves open the mystery of how
probes are managing to come in on the wrong ports. For example:

Jul 22 19:40:56 firewall2 kernel: [15358673.237154] Samba TCP: IN=eth5 OUT= MAC=00:1c:c4:48:5e:4e:00:0b:5f:fb:d0:80:08:00 SRC=206.126.124.39 DST=11.222.201.99 LEN=48 TOS=0x00 PREC=0x00 TTL=116 ID=16385 DF PROTO=TCP SPT=3303 DPT=445 WINDOW=65535 RES=0x00 SYN URGP=0 
Jul 22 19:40:56 firewall2 kernel: [15358673.631689] Samba TCP: IN=eth5 OUT= MAC=00:1c:c4:48:5e:4e:00:0b:5f:fb:d0:80:08:00 SRC=206.126.124.39 DST=11.222.201.100 LEN=48 TOS=0x00 PREC=0x00 TTL=116 ID=16472 DF PROTO=TCP SPT=3342 DPT=445 WINDOW=65535 RES=0x00 SYN URGP=0 
Jul 22 19:40:57 firewall2 kernel: [15358673.924561] Samba TCP: IN=eth1 OUT= MAC=00:1e:0b:5e:16:fc:00:0b:5f:fb:d0:80:08:00 SRC=206.126.124.39 DST=11.222.201.101 LEN=48 TOS=0x00 PREC=0x00 TTL=116 ID=16539 DF PROTO=TCP SPT=3366 DPT=445 WINDOW=65535 RES=0x00 SYN URGP=0 
Jul 22 19:40:57 firewall2 kernel: [15358674.359341] Samba TCP: IN=eth2 OUT= MAC=00:1c:c4:48:5e:4c:00:0b:5f:fb:d0:80:08:00 SRC=206.126.124.39 DST=11.222.201.103 LEN=48 TOS=0x00 PREC=0x00 TTL=116 ID=16661 DF PROTO=TCP SPT=3447 DPT=445 WINDOW=65535 RES=0x00 SYN URGP=0 
Jul 22 19:40:57 firewall2 kernel: [15358674.577988] Samba TCP: IN=eth5 OUT= MAC=00:1c:c4:48:5e:4e:00:0b:5f:fb:d0:80:08:00 SRC=206.126.124.39 DST=11.222.201.104 LEN=48 TOS=0x00 PREC=0x00 TTL=116 ID=16714 DF PROTO=TCP SPT=3478 DPT=445 WINDOW=65535 RES=0x00 SYN URGP=0 
Jul 22 19:40:58 firewall2 kernel: [15358674.845582] Samba TCP: IN=eth5 OUT= MAC=00:1c:c4:48:5e:4e:00:0b:5f:fb:d0:80:08:00 SRC=206.126.124.39 DST=11.222.201.105 LEN=48 TOS=0x00 PREC=0x00 TTL=116 ID=16799 DF PROTO=TCP SPT=3503 DPT=445 WINDOW=65535 RES=0x00 SYN URGP=0 
Jul 22 19:40:58 firewall2 kernel: [15358675.118500] Samba TCP: IN=eth5 OUT= MAC=00:1c:c4:48:5e:4e:00:0b:5f:fb:d0:80:08:00 SRC=206.126.124.39 DST=11.222.201.106 LEN=48 TOS=0x00 PREC=0x00 TTL=116 ID=16872 DF PROTO=TCP SPT=3532 DPT=445 WINDOW=65535 RES=0x00 SYN URGP=0 
Jul 22 19:40:59 firewall2 kernel: [15358676.152236] Samba TCP: IN=eth5 OUT= MAC=00:1c:c4:48:5e:4e:00:0b:5f:fb:d0:80:08:00 SRC=206.126.124.39 DST=11.222.201.99 LEN=48 TOS=0x00 PREC=0x00 TTL=116 ID=17145 DF PROTO=TCP SPT=3303 DPT=445 WINDOW=65535 RES=0x00 SYN URGP=0 
Jul 22 19:40:59 firewall2 kernel: [15358676.438063] Samba TCP: IN=eth5 OUT= MAC=00:1c:c4:48:5e:4e:00:0b:5f:fb:d0:80:08:00 SRC=206.126.124.39 DST=11.222.201.109 LEN=48 TOS=0x00 PREC=0x00 TTL=116 ID=17192 DF PROTO=TCP SPT=3641 DPT=445 WINDOW=65535 RES=0x00 SYN URGP=0 
Jul 22 19:40:59 firewall2 kernel: [15358676.611885] Samba TCP: IN=eth5 OUT= MAC=00:1c:c4:48:5e:4e:00:0b:5f:fb:d0:80:08:00 SRC=206.126.124.39 DST=11.222.201.100 LEN=48 TOS=0x00 PREC=0x00 TTL=116 ID=17228 DF PROTO=TCP SPT=3342 DPT=445 WINDOW=65535 RES=0x00 SYN URGP=0 
Jul 22 19:41:00 firewall2 kernel: [15358676.918421] Samba TCP: IN=eth1 OUT= MAC=00:1e:0b:5e:16:fc:00:0b:5f:fb:d0:80:08:00 SRC=206.126.124.39 DST=11.222.201.101 LEN=48 TOS=0x00 PREC=0x00 TTL=116 ID=17311 DF PROTO=TCP SPT=3366 DPT=445 WINDOW=65535 RES=0x00 SYN URGP=0 
Jul 22 19:41:00 firewall2 kernel: [15358677.258672] Samba TCP: IN=eth2 OUT= MAC=00:1c:c4:48:5e:4c:00:0b:5f:fb:d0:80:08:00 SRC=206.126.124.39 DST=11.222.201.103 LEN=48 TOS=0x00 PREC=0x00 TTL=116 ID=17435 DF PROTO=TCP SPT=3447 DPT=445 WINDOW=65535 RES=0x00 SYN URGP=0 
Jul 22 19:41:00 firewall2 kernel: [15358677.481513] Samba TCP: IN=eth5 OUT= MAC=00:1c:c4:48:5e:4e:00:0b:5f:fb:d0:80:08:00 SRC=206.126.124.39 DST=11.222.201.104 LEN=48 TOS=0x00 PREC=0x00 TTL=116 ID=17487 DF PROTO=TCP SPT=3478 DPT=445 WINDOW=65535 RES=0x00 SYN URGP=0 
Jul 22 19:41:01 firewall2 kernel: [15358677.900259] Samba TCP: IN=eth5 OUT= MAC=00:1c:c4:48:5e:4e:00:0b:5f:fb:d0:80:08:00 SRC=206.126.124.39 DST=11.222.201.105 LEN=48 TOS=0x00 PREC=0x00 TTL=116 ID=17586 DF PROTO=TCP SPT=3503 DPT=445 WINDOW=65535 RES=0x00 SYN URGP=0 
Jul 22 19:41:01 firewall2 kernel: [15358678.128048] Samba TCP: IN=eth5 OUT= MAC=00:1c:c4:48:5e:4e:00:0b:5f:fb:d0:80:08:00 SRC=206.126.124.39 DST=11.222.201.106 LEN=48 TOS=0x00 PREC=0x00 TTL=116 ID=17653 DF PROTO=TCP SPT=3532 DPT=445 WINDOW=65535 RES=0x00 SYN URGP=0 
Jul 22 19:41:02 firewall2 kernel: [15358679.430171] Samba TCP: IN=eth5 OUT= MAC=00:1c:c4:48:5e:4e:00:0b:5f:fb:d0:80:08:00 SRC=206.126.124.39 DST=11.222.201.109 LEN=48 TOS=0x00 PREC=0x00 TTL=116 ID=17950 DF PROTO=TCP SPT=3641 DPT=445 WINDOW=65535 RES=0x00 SYN URGP=0 
Jul 22 19:41:03 firewall2 kernel: [15358680.551399] Samba TCP: IN=eth5 OUT= MAC=00:1c:c4:48:5e:4e:00:0b:5f:fb:d0:80:08:00 SRC=206.126.124.39 DST=11.222.201.124 LEN=48 TOS=0x00 PREC=0x00 TTL=116 ID=18279 DF PROTO=TCP SPT=4061 DPT=445 WINDOW=65535 RES=0x00 SYN URGP=0 
Jul 22 19:41:05 firewall2 kernel: [15358682.163911] Samba TCP: IN=eth5 OUT= MAC=00:1c:c4:48:5e:4e:00:0b:5f:fb:d0:80:08:00 SRC=206.126.124.39 DST=11.222.201.99 LEN=48 TOS=0x00 PREC=0x00 TTL=116 ID=18664 DF PROTO=TCP SPT=3303 DPT=445 WINDOW=65535 RES=0x00 SYN URGP=0 
Jul 22 19:41:05 firewall2 kernel: [15358682.611803] Samba TCP: IN=eth5 OUT= MAC=00:1c:c4:48:5e:4e:00:0b:5f:fb:d0:80:08:00 SRC=206.126.124.39 DST=11.222.201.100 LEN=48 TOS=0x00 PREC=0x00 TTL=116 ID=18744 DF PROTO=TCP SPT=3342 DPT=445 WINDOW=65535 RES=0x00 SYN URGP=0 
Jul 22 19:41:06 firewall2 kernel: [15358682.941205] Samba TCP: IN=eth1 OUT= MAC=00:1e:0b:5e:16:fc:00:0b:5f:fb:d0:80:08:00 SRC=206.126.124.39 DST=11.222.201.101 LEN=48 TOS=0x00 PREC=0x00 TTL=116 ID=18856 DF PROTO=TCP SPT=3366 DPT=445 WINDOW=65535 RES=0x00 SYN URGP=0 
Jul 22 19:41:06 firewall2 kernel: [15358683.255435] Samba TCP: IN=eth2 OUT= MAC=00:1c:c4:48:5e:4c:00:0b:5f:fb:d0:80:08:00 SRC=206.126.124.39 DST=11.222.201.103 LEN=48 TOS=0x00 PREC=0x00 TTL=116 ID=18960 DF PROTO=TCP SPT=3447 DPT=445 WINDOW=65535 RES=0x00 SYN URGP=0 
Jul 22 19:41:06 firewall2 kernel: [15358683.478190] Samba TCP: IN=eth5 OUT= MAC=00:1c:c4:48:5e:4e:00:0b:5f:fb:d0:80:08:00 SRC=206.126.124.39 DST=11.222.201.104 LEN=48 TOS=0x00 PREC=0x00 TTL=116 ID=19001 DF PROTO=TCP SPT=3478 DPT=445 WINDOW=65535 RES=0x00 SYN URGP=0 
Jul 22 19:41:06 firewall2 kernel: [15358683.578231] Samba TCP: IN=eth5 OUT= MAC=00:1c:c4:48:5e:4e:00:0b:5f:fb:d0:80:08:00 SRC=206.126.124.39 DST=11.222.201.124 LEN=48 TOS=0x00 PREC=0x00 TTL=116 ID=19013 DF PROTO=TCP SPT=4061 DPT=445 WINDOW=65535 RES=0x00 SYN URGP=0 
Jul 22 19:41:07 firewall2 kernel: [15358683.912897] Samba TCP: IN=eth5 OUT= MAC=00:1c:c4:48:5e:4e:00:0b:5f:fb:d0:80:08:00 SRC=206.126.124.39 DST=11.222.201.105 LEN=48 TOS=0x00 PREC=0x00 TTL=116 ID=19089 DF PROTO=TCP SPT=3503 DPT=445 WINDOW=65535 RES=0x00 SYN URGP=0 
Jul 22 19:41:07 firewall2 kernel: [15358684.122432] Samba TCP: IN=eth5 OUT= MAC=00:1c:c4:48:5e:4e:00:0b:5f:fb:d0:80:08:00 SRC=206.126.124.39 DST=11.222.201.106 LEN=48 TOS=0x00 PREC=0x00 TTL=116 ID=19136 DF PROTO=TCP SPT=3532 DPT=445 WINDOW=65535 RES=0x00 SYN URGP=0 
Jul 22 19:41:08 firewall2 kernel: [15358685.439300] Samba TCP: IN=eth5 OUT= MAC=00:1c:c4:48:5e:4e:00:0b:5f:fb:d0:80:08:00 SRC=206.126.124.39 DST=11.222.201.109 LEN=48 TOS=0x00 PREC=0x00 TTL=116 ID=19407 DF PROTO=TCP SPT=3641 DPT=445 WINDOW=65535 RES=0x00 SYN URGP=0 
Jul 22 19:41:12 firewall2 kernel: [15358689.581183] Samba TCP: IN=eth5 OUT= MAC=00:1c:c4:48:5e:4e:00:0b:5f:fb:d0:80:08:00 SRC=206.126.124.39 DST=11.222.201.124 LEN=48 TOS=0x00 PREC=0x00 TTL=116 ID=20295 DF PROTO=TCP SPT=4061 DPT=445 WINDOW=65535 RES=0x00 SYN URGP=0 

To decode that, eth5 is an external interface, while eth1 and eth2 are
separate LANs. The (numbers altered here) "11.222.*" addresses are on eth5.
This is clearly a scan sequence, going over almost the same IPs in about the
same sequential order 3 times. But for two of the IPs in that sequence - the
same 2 each time - it looks to be coming in on eth1 and eth2 instead of
eth5.

This is the public-facing firewall. So how could these several probes show
up on the internal interfaces? (And these aren't the only ones. We're
getting them from IPs around the world.) Does this imply a compromised
internal machine that's relaying that part of the scan? Or some way to spoof
interfaces? It's new to us, and we've been running a stable firewall
configuration for a few years.

TIA,
Whit

Re: How might incoming SMB probes from public IPs be ariving on the internal interfaces?

[Whit Blauvelt]

Just to clarify and simplify the question my experience raises:

There's a system functioning as a firewall with two public lines coming in,
each with multiple IPs, and two LANs inside it. In this case the public
lines are on eth4 and eth5, the two LANs are on eth1 and eth2.

Rules for INPUT specify each of the ports which are to be open on eth4 and
eth5, with a separate rule for each IP/port combination, and with a rule
following them to reject all else on those interfaces. SMB ports are not
among the the accepted traffic for those interfaces at all, not for a single
public IP.

But there's no blocking of SMB traffic on eth1 and eth2. There's an SMB
service on the firewall by which workstations on the two LANs are able to
place files for public-facing (non-SMB) servers there. 

All SMB probes seen as coming in on eth4 and eth5 get blocked, just as
should be expected. But some SMB probes, of our public IPs, by outside
public IPs, are seen by iptables as coming in on eth1 and eth2, so not
blocked at that level. They can't connect successfully to our SMB server
anyway because of its own settings. But that's not the point.

The point is that some of the scans of our public IPs manage to appear to
the firewall as coming from the internal interfaces - and I provided an
example in the prior post where it is clearly a single, repeated scan that
hits the internal interfaces with several, but not all, of its instances.

Now, it's easy enough to block once it's recognized. It just takes adding
rules to block the SMB ports on _all_ interfaces to anything with an
external address, no matter which interface it appears to come in from. But
this also suggests that the apparent interface of a packet, perhaps, just
isn't to be trusted - that to be secure all rules must be 100% in terms of
IP ranges, with interface specifications only to be considered a secondary,
leaky line of defense.

Admittedly this is an older netfilter, on an older kernel, since firewalls
are the sort of thing where upgrade policies favor a conservative approach.
Is there some known bug where packets can "slip sideways" between
interfaces? What the heck can be going on here?

Thanks,
Whit

Could Cogent be doing packet mangling that would confuse Netfilter about interfaces?

[Whit Blauvelt]

Hi,

This is related to a couple of messages I sent to the list 3 weeks ago, with
some additional information. The basic problem is that Netfilter is seeing
some traffic lately as coming in on the wrong interface, which we notice
when that traffic gets blocked or gets by when it shouldn't, since some of
our rules specify interfaces.

We're now seeing the same behavior from both iptables 1.4.8 on Debian
Squeeze and 1.3.8 on Ubuntu 8.04 (on totally different hardware). The common
factor is that this is all traffic coming in via our Cogent pipe. When
traffic comes in via either of our two other Speakeasy/Megapath pipes
Netfilter sees all the interface specifications correctly.

This makes me wonder if Cogent has started doing some sort of packet
mangling that's not compatible with how Netfilter looks at packet headers to
determine which interface the packets have shown up from. Could something
like that be the case? Or what else could account for this? 

It does look like Cogent is on the list of providers who have been mucking
with stuff they shouldn't -
http://www.newscientist.com/article/dn20768-us-internet-providers-hijacking-users-search-queries.html

We can work around whatever the problem here is by specifying rules by IP
block rather than interface. Still, it's troubling that there can apparently
be some characteristic of incoming traffic that can get Netfilter confused
about the interface it's arrived on. There have to be evil uses for that.

Best,
Whit

Re: Could Cogent be doing packet mangling that would confuse Netfilter about interfaces?

[Tom Eastep]

On Aug 15, 2011, at 10:13 AM, Whit Blauvelt wrote:

> This is related to a couple of messages I sent to the list 3 weeks ago, with
> some additional information. The basic problem is that Netfilter is seeing
> some traffic lately as coming in on the wrong interface, which we notice
> when that traffic gets blocked or gets by when it shouldn't, since some of
> our rules specify interfaces.
> 
> We're now seeing the same behavior from both iptables 1.4.8 on Debian
> Squeeze and 1.3.8 on Ubuntu 8.04 (on totally different hardware). The common
> factor is that this is all traffic coming in via our Cogent pipe. When
> traffic comes in via either of our two other Speakeasy/Megapath pipes
> Netfilter sees all the interface specifications correctly.


This behavior is usually a sign that your Cogent external firewall interface and and your internal firewall interface have accidentally become part of the same broadcast domain. I would look for mis-cabling and for erroneous VLAN configuration.

-Tom

Tom Eastep        \ When I die, I want to go like my Grandfather who
Shoreline,         \ died peacefully in his sleep. Not screaming like
Washington, USA     \ all of the passengers in his car
http://shorewall.net \________________________________________________

Re: Could Cogent be doing packet mangling that would confuse Netfilter about interfaces?

[Whit Blauvelt]

On Mon, Aug 15, 2011 at 10:52:35AM -0700, Tom Eastep wrote:

> On Aug 15, 2011, at 10:13 AM, Whit Blauvelt wrote:

> > We're now seeing the same behavior from both iptables 1.4.8 on Debian
> > Squeeze and 1.3.8 on Ubuntu 8.04 (on totally different hardware). The common
> > factor is that this is all traffic coming in via our Cogent pipe. When
> > traffic comes in via either of our two other Speakeasy/Megapath pipes
> > Netfilter sees all the interface specifications correctly.
> 
> This behavior is usually a sign that your Cogent external firewall
> interface and and your internal firewall interface have accidentally
> become part of the same broadcast domain. I would look for mis-cabling and
> for erroneous VLAN configuration.

Tom,

Thanks. We'll double check our cabling. No VLANs are in use here. Would the
broadcast domain diagnosis fit our most recent problem? Here it is:

- We have an external IP on each outside pipe DNAT'd to an internal web
  server, on a DMZ address. This has been working perfectly for several
  years.

- Suddenly last week connections to the Cogent IP (on eth5) stopped working.
  The logs showed those packets coming in from the DMZ interface (eth2)
  destined for the DNAT-assigned internal IP (also on eth2) - getting
  blocked by Netfilter since that made no sense.

- Adding an iptables rule to allow forwarding from that interface (eth2) to
  that IP (on eth2) restored service

- The other two external pipes work as always.

Trying to picture how a cable in the wrong place would allow this. The
Cogent router is just a router, not doing firewalling, so it's the Linux
firewall with Cogent and Speakeasy routers attached on two interfaces (with
a switch in between in each case, since there's an active backup firewall),
and the LAN and DMZ attached on two more.

What kind of miscabling could cause a packet which makes it through the
Linux firewall's DNAT to then be seen by the same firewall as coming from
the DMZ's interface, complete with DNAT-translated destination IP, asking to
be forwarded to an IP on the DMZ?

You're probably right - I hope you are since a solution would be good. Just
trying to narrow down what we're looking for by way of miscabling.

Best,
Whit

Re: Could Cogent be doing packet mangling that would confuse Netfilter about interfaces?

[Whit Blauvelt]

On Mon, Aug 15, 2011 at 04:33:35PM -0400, Whit Blauvelt wrote:

> Trying to picture how a cable in the wrong place would allow this. The
> Cogent router is just a router, not doing firewalling, so it's the Linux
> firewall with Cogent and Speakeasy routers attached on two interfaces (with
> a switch in between in each case, since there's an active backup firewall),
> and the LAN and DMZ attached on two more.

Just to make it weirder, on the second system we tested this on we have
Cogent on eth3, DMZ on eth1, LAN on eth0 and Speakeasy on eth2. In this case
the Cogent traffic shows up on the wrong interface too - but a different
wrong interface - the LAN interface rather than the DMZ interface. It shows
up as coming in on eth0 headed for the DNAT translated address on eth1 -
despite that the traffic arrived on eth3.

What is consistent (probably coincidence) is that Cogent traffic coming in
on eth5 in the first case appears as if coming in on eth2, and Cogent
traffic coming in on eth3 in the second case appears as if it's coming in on
eth0 - so "subtract 3 from real interface number" would do it. 

Again, the same traffic coming in over through a Speakeasy pipe doesn't get
confused about interfaces at all.

Whit

Re: Could Cogent be doing packet mangling that would confuse Netfilter about interfaces?

[Tom Eastep]

[-- Attachment #1: Type: text/plain, Size: 1403 bytes --]

On Mon, 2011-08-15 at 16:33 -0400, Whit Blauvelt wrote:
> On Mon, Aug 15, 2011 at 10:52:35AM -0700, Tom Eastep wrote:
> 
> > On Aug 15, 2011, at 10:13 AM, Whit Blauvelt wrote:
> 
> > > We're now seeing the same behavior from both iptables 1.4.8 on Debian
> > > Squeeze and 1.3.8 on Ubuntu 8.04 (on totally different hardware). The common
> > > factor is that this is all traffic coming in via our Cogent pipe. When
> > > traffic comes in via either of our two other Speakeasy/Megapath pipes
> > > Netfilter sees all the interface specifications correctly.
> > 
> > This behavior is usually a sign that your Cogent external firewall
> > interface and and your internal firewall interface have accidentally
> > become part of the same broadcast domain. I would look for mis-cabling and
> > for erroneous VLAN configuration.

Whit,

I don't have time ATM to give you detailed help, but
http://www.shorewall.net/FoolsFirewall.html#id36131257 explains what
happens when two firewall interfaces are effectively connected to the
same ethernet network. That may help you figure out where the problem
is.

-Tom
-- 
Tom Eastep        \ When I die, I want to go like my Grandfather who
Shoreline,         \ died peacefully in his sleep. Not screaming like
Washington, USA     \ all of the passengers in his car
http://shorewall.net \________________________________________________


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Could Cogent be doing packet mangling that would confuse Netfilter about interfaces?

[Whit Blauvelt]

On Mon, Aug 15, 2011 at 02:10:33PM -0700, Tom Eastep wrote:

> I don't have time ATM to give you detailed help, but
> http://www.shorewall.net/FoolsFirewall.html#id36131257 explains what
> happens when two firewall interfaces are effectively connected to the
> same ethernet network. That may help you figure out where the problem
> is.

Tom,

I appreciate all suggestions. I'm pretty sure the guy in charge of our
switch-and-cable infrastructure hasn't connected any switch to more than one
zone - because I've specifically asked him before, he gave me that
assurance, and he's a smart guy. But I'll ask again.

Meanwhile, if anyone else here has a suggestion, the working assumption is
that we don't have an example of the "Fool's Firewall" (as it is very
clearly explained on Tom's page) so other suggestions will also be
appreciated.

Thanks,
Whit

Re: Could Cogent be doing packet mangling that would confuse Netfilter about interfaces?

[Grant Taylor]

On 08/15/11 16:25, Whit Blauvelt wrote:
> Meanwhile, if anyone else here has a suggestion, the working assumption is
> that we don't have an example of the "Fool's Firewall" (as it is very
> clearly explained on Tom's page) so other suggestions will also be
> appreciated.

For giggles have you tried looking for the mac addresses on eth1 and 
eth2 (from your first message)?

Does the traffic coming in to eth5 have the proper MAC address of your 
Cogent router?

Have you considered sniffing the traffic with another device before the 
traffic enters eth5 to make sure that the traffic really is on the wire 
like you think it is verses some odd bug that is causing the traffic to 
be mis-represented by the kernel?

Start gathering duplicate data from other locations in the network to 
see what adds up and checksums each other and what does not.  Follow the 
evidence.

It sounds like it's time to gather more data before you start filtering 
it down.



Grant. . . .