Strange logs...

Strange logs...

[Carlos Fernandez Sanz]

[-- Attachment #1: Type: text/plain, Size: 2101 bytes --]

Hi,

For some time I have been having strange problems, which mostly were noticed in Samba (so I assumed it was a Samba problem). Basically a connection (from a Windows box to a linux box, but that doesn't matter much now) would stop working (as if the server had gone down) and it was impossible to recover it for a few minutes.

Actually it happens with all other stuff (the linux box handling the internet connection and NAT), in fact when that happens you can't even telnet/ssh in to the linux box.

I got fed up so I started logging every dropped packet in iptables, just in case it was related.

And this is what I get:

Jan 11 11:52:12 fulanito kernel: [IPTABLES DROP NAT] : IN=eth1 OUT= MAC=00:01:03:27:83:4c:00:0c:6e:77:a9:92:08:00 SRC=192.168.20.5 DST=192.168.20.1 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=13013 PROTO=UDP SPT=137 DPT=137 LEN=58

eth1 is my external (connected to the internet router) interface, 192.168.20.5 is one of my window boxes, 192.168.20.1 is my linux box. These two boxes are connected via a switch (which has nothing else connected to it), and the interface is eth0. 

What could cause that the packet appears in eth1 instead of eth0? Of course that explains that it's being dropped, as I have a rule that drops everything coming in the external interface with private addresses.... 

I know the obvious answer would be "someone special made that packet and sent it", but the packet does come from the LAN. The MAC matches the IP it's supposes to come from (i.e. belongs to the NIC in my windows card), and most importantly, when I see that in the logs (happens from time to time, I can't figure out what triggers it, and the problem goes away by itself after a few minutes) the LAN computers can't connect to the linux box.

Any idea?

I'm currently using this kernel

Linux version 2.4.22-1.2096.nptl (bhcompile@porky.devel.redhat.com) (gcc version 3.2.3 20030422 (Red Hat Linux 3.2.3-6)) #1 Thu Oct 16 12:06:27 EDT 2003


but it happened as previous versions as well, both from redhat and mainstream (linus' tree locally compiled here).


[-- Attachment #2: Type: text/html, Size: 3585 bytes --]

Re: Strange logs...

[Antony Stone]

On Sunday 11 January 2004 11:40 am, Carlos Fernandez Sanz wrote:

> Jan 11 11:52:12 fulanito kernel: [IPTABLES DROP NAT] : IN=eth1 OUT=
> MAC=00:01:03:27:83:4c:00:0c:6e:77:a9:92:08:00 SRC=192.168.20.5
> DST=192.168.20.1 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=13013 PROTO=UDP
> SPT=137 DPT=137 LEN=58
>
> eth1 is my external (connected to the internet router) interface,
> 192.168.20.5 is one of my window boxes, 192.168.20.1 is my linux box. These
> two boxes are connected via a switch (which has nothing else connected to
> it), and the interface is eth0.
>
> What could cause that the packet appears in eth1 instead of eth0? Of course
> that explains that it's being dropped, as I have a rule that drops
> everything coming in the external interface with private addresses....
>
> I know the obvious answer would be "someone special made that packet and
> sent it", but the packet does come from the LAN. The MAC matches the IP
> it's supposes to come from (i.e. belongs to the NIC in my windows card),

The fact that the MAC address is correct means that the packet has surely come 
from the Windows machine, and has not come through any other router (because 
if it had, it would have the IP address of the Windows box and the MAC 
address of the router).

Tell us more about your network connections - you say you have a switch on 
eth0 connected to the Windows box and nothing else; how is eth1 connected to 
your Internet router?   Crossover cable?   Switch/hub?   What?

Also, do you have a nice simple, clean subnet arrangement - something like a 
single public IP on eth1, and a private class C on eth0, nothing fancy?

It would be good to try running tcpdump or ethereal on the netfilter machine, 
so that when a log entry such as this appears, you can check the tcpdump or 
ethereal log and see if it agrees that the packet really did only come in on 
eth1.

Not a solution to your probloem, I know, but maybe a help along the way?

Antony.

-- 
Software development can be quick, high quality, or low cost.

The customer gets to pick any two out of three.

                                                     Please reply to the list;
                                                           please don't CC me.

Re: Strange logs...

[Carlos Fernandez Sanz]

> The fact that the MAC address is correct means that the packet has surely
come
> from the Windows machine, and has not come through any other router
(because
> if it had, it would have the IP address of the Windows box and the MAC
> address of the router).

Yes. I'm playing a bit more with this, and after adding a "--match
mac --mac-source" rule, I started seeing the packets being dropped by the
input policy (which is obviously drop). Once I added the same "saving by
MAC" rule, everything works fine (except my conscience).
>
> Tell us more about your network connections - you say you have a switch on
> eth0 connected to the Windows box and nothing else; how is eth1 connected
to
> your Internet router?   Crossover cable?   Switch/hub?   What?

I'll explain and check cables at the same time.

The linux box has two ethernet interfaces.

eth0 is the internal LAN interface. 192.168.20.1, and it's connected to a
10/100 switch to which other computers go (192.168.20.5 which is the window
box, for example. there's others but not important here)
eth1 is the external interface, and it's is connected to my ADSL router
directly.
>
> Also, do you have a nice simple, clean subnet arrangement - something like
a
> single public IP on eth1, and a private class C on eth0, nothing fancy?

Yes.
>
> It would be good to try running tcpdump or ethereal on the netfilter
machine,
> so that when a log entry such as this appears, you can check the tcpdump
or
> ethereal log and see if it agrees that the packet really did only come in
on
> eth1.

How reliable is ethereal? I mean, does it see packets as they come from the
wire or after they have been touched by netfilter?

Anyway, I've added this now to the firewall:

iptables -A INPUT --match mac --mac-source 00:0c:6e:77:a9:92 -j
LOG --log-tcp-options --log-ip-options --log-prefix '[IPTABLES MAC INPUT] :
'
iptables -A INPUT --match mac --mac-source 00:0c:6e:77:a9:92 -j ACCEPT

So all packets that are saved by that rule are logged.

I've seen some packets already:

Jan 11 13:28:53 fulanito kernel: [IPTABLES MAC INPUT] : IN=eth1 OUT=
MAC=ff:ff:ff:ff:ff:ff:00:0c:6e:77:a9:92:08:00 SRC=192.168.20.5
DST=192.168.20.255 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=16661 PROTO=UDP
SPT=137 DPT=137 LEN=58

That packet shows up in ethereal as well (everything matches). However all
packets seen so far in both ethereal and the logs (4) are for broadcasts
(still, is that possible? How could a packet generated by the windows box,
which isn't connected to eth1, end up there?). I haven't seen anything else
yet.


> Not a solution to your probloem, I know, but maybe a help along the way?

Anything that makes me think of trying something else is a help somehow :-)
Thanks.

Re: Strange logs...

[Antony Stone]

On Sunday 11 January 2004 12:41 pm, Carlos Fernandez Sanz wrote:

> > The fact that the MAC address is correct means that the packet has surely
> > come from the Windows machine, and has not come through any other router
> > (because if it had, it would have the IP address of the Windows box and
> > the MAC address of the router).
>
> Yes. I'm playing a bit more with this, and after adding a "--match
> mac --mac-source" rule, I started seeing the packets being dropped by the
> input policy (which is obviously drop). Once I added the same "saving by
> MAC" rule, everything works fine (except my conscience).

Well, that's good progress.

> > It would be good to try running tcpdump or ethereal on the netfilter
> > machine, so that when a log entry such as this appears, you can check the
> > tcpdump or ethereal log and see if it agrees that the packet really did
> > only come in on eth1.
>
> How reliable is ethereal? I mean, does it see packets as they come from the
> wire or after they have been touched by netfilter?

Ethereal and tcpdump will both see packets off the wire before they get to 
netfilter.   Remember that these packet capturing programs and not 
IP-specific; they will show you ethernet, IPX, LAT, all sort of things, if 
they only come past the interface...

> I've seen some packets already:
>
> Jan 11 13:28:53 fulanito kernel: [IPTABLES MAC INPUT] : IN=eth1 OUT=
> MAC=ff:ff:ff:ff:ff:ff:00:0c:6e:77:a9:92:08:00 SRC=192.168.20.5
> DST=192.168.20.255 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=16661 PROTO=UDP
> SPT=137 DPT=137 LEN=58
>
> That packet shows up in ethereal as well (everything matches). However all
> packets seen so far in both ethereal and the logs (4) are for broadcasts

That's normal for Windows networking :)

> (still, is that possible? How could a packet generated by the windows box,
> which isn't connected to eth1, end up there?).

That's the one bit I can't think of an explanation for.   You don't have 
anything exotic like bridging or vlans enabled in your kernel do you?

Antony.

-- 
Wanted: telepath.   You know where to apply.

                                                     Please reply to the list;
                                                           please don't CC me.

Re: Strange logs...

[Carlos Fernandez Sanz]

> > How reliable is ethereal? I mean, does it see packets as they come from
the
> > wire or after they have been touched by netfilter?
>
> Ethereal and tcpdump will both see packets off the wire before they get to
> netfilter.   Remember that these packet capturing programs and not

After leaving it working for some more minutes, I started seeing normal
traffic being logged as well. Packets show up in eth1 (internet interface)
both in the logs and ethereal (there's a 100% match). Now, how are the linux
device (eth1) linked to the physical device (NIC) related?

I have to say that if I unplug any of the two cables (the one between the
linux and the switch or the one between the linux and the router) strange
things happen. So I'm really starting to believe that the packet does come
from the wire physically.... no matter how impossible that seems.

As I finished writing that I tried to ping from the windows box to the linux
box and both cables must be connected for ping responses to arrive. This
happens even with all the iptables tables flushed (ie firewall down; no
rules, and ACCEPT as policies for both input and policies).

> > (still, is that possible? How could a packet generated by the windows
box,
> > which isn't connected to eth1, end up there?).
>
> That's the one bit I can't think of an explanation for.   You don't have
> anything exotic like bridging or vlans enabled in your kernel do you?

Nothing (that I'm aware of, anyway). But this is starting to look more like
a hardware issue to me. Definitely not related to the wiring, but maybe
there's some kind of conflict between my NICs I haven't spotted yet.... so
in case it helps:

  Bus  2, device  11, function  0:
    Ethernet controller: 3Com Corporation 3c905C-TX/TX-M [Tornado] (rev 48).
      IRQ 9.
      Master Capable.  Latency=32.  Min Gnt=10.Max Lat=10.
      I/O at 0xd800 [0xd87f].
      Non-prefetchable 32 bit memory at 0xf4000000 [0xf400007f].
  Bus  2, device  14, function  0:
    Ethernet controller: 3Com Corporation 3c905C-TX/TX-M [Tornado] (#2) (rev
48).
      IRQ 3.
      Master Capable.  Latency=32.  Min Gnt=10.Max Lat=10.
      I/O at 0x8800 [0x887f].
      Non-prefetchable 32 bit memory at 0xf3800000 [0xf380007f].

Thanks for helping out :-)

Re: Strange logs...

[Antony Stone]

On Sunday 11 January 2004 1:29 pm, Carlos Fernandez Sanz wrote:

> I have to say that if I unplug any of the two cables (the one between the
> linux and the switch or the one between the linux and the router) strange
> things happen. So I'm really starting to believe that the packet does come
> from the wire physically.... no matter how impossible that seems.
>
> As I finished writing that I tried to ping from the windows box to the
> linux box and both cables must be connected for ping responses to arrive.

By "both cables", you mean the one connecting the Linux box to the switch (and 
thence to the Windows box) and also the one joining the Linux machine to the 
outside world!?!?

That is just plain wrong.   You should not need a connection to your external 
router in order to ping between two internal machines (assuming you are 
pinging the real IP addresses, and not relying on a DNS lookup or something).

> This happens even with all the iptables tables flushed (ie firewall down;
> no rules, and ACCEPT as policies for both input and policies).

Show us the routing tables on both machines.

> But this is starting to look more like
> a hardware issue to me. Definitely not related to the wiring, but maybe
> there's some kind of conflict between my NICs I haven't spotted yet.... so
> in case it helps:

Your IRQ/address etc setup looks fine to me, however there is clearly 
something weird about the network hardware arrangement for you to get the 
above-described ping behaviour.

Antony.

-- 
"Note: Windows 98, Windows 98SE and Windows 95 are not affected by [MS
Blaster].   However, these products are no longer supported.   Users of these
products are strongly encouraged to upgrade to later versions."

(which *are* affected by MS Blaster...)

http://www.microsoft.com/security/security_bulletins/ms03-026.asp

                                                     Please reply to the list;
                                                           please don't CC me.

Re: Strange logs...

[Carlos Fernandez Sanz]

> That is just plain wrong.   You should not need a connection to your
external
> router in order to ping between two internal machines (assuming you are
> pinging the real IP addresses, and not relying on a DNS lookup or
something).

Indeed....

> Your IRQ/address etc setup looks fine to me, however there is clearly
> something weird about the network hardware arrangement for you to get the
> above-described ping behaviour.

Such as the router and the switch being connected for no reason? I was
obviously on crack when I set that up.... just found out that was the
problem.... strange that everything would work perfectly for hours then stop
working for minutes, etc... probably a table filling up in the switch or the
router or both :-)

Well, I made you waste a few minutes and wasted a nice sunday morning doing
this instead of being with my girlfriend, on the bright side I got a spare
network cable, give me your address if you want a slice :-)

Thanks!

Re: Strange logs...

[Antony Stone]

On Sunday 11 January 2004 1:59 pm, Carlos Fernandez Sanz wrote:

> > Your IRQ/address etc setup looks fine to me, however there is clearly
> > something weird about the network hardware arrangement for you to get the
> > above-described ping behaviour.
>
> Such as the router and the switch being connected for no reason? I was
> obviously on crack when I set that up.... just found out that was the
> problem.... strange that everything would work perfectly for hours then
> stop working for minutes, etc... probably a table filling up in the switch
> or the router or both :-)

I'd put it down to the arp caches getting (mightily) confused about where to 
find things.   Never mind the degree to which a security engineer's eyebrow 
would rise upon realising your switch was connected to both sides of your 
firewall :)

Regards,

Antony.

-- 
This email is intended for the use of the individual addressee(s) named above 
and may contain information that is confidential, privileged or unsuitable 
for overly sensitive persons with low self-esteem, no sense of humour, or 
irrational religious beliefs.

If you have received this email in error, you are required to shred it 
immediately, add some nutmeg, three egg whites and a dessertspoonful of 
caster sugar.   Whisk until soft peaks form, then place in a warm oven for 40 
minutes.   Remove promptly and let stand for 2 hours before adding some 
decorative kiwi fruit and cream.   Then notify me immediately by return email 
and eat the original message.

                                                     Please reply to the list;
                                                           please don't CC me.

Re: Strange logs...

[Unknown, Alistair Tonner]

On January 11, 2004 08:59 am, Carlos Fernandez Sanz wrote:
> > That is just plain wrong.   You should not need a connection to your
>
> external
>
> > router in order to ping between two internal machines (assuming you are
> > pinging the real IP addresses, and not relying on a DNS lookup or
>
> something).
>
> Indeed....
>
> > Your IRQ/address etc setup looks fine to me, however there is clearly
> > something weird about the network hardware arrangement for you to get the
> > above-described ping behaviour.
>
> Such as the router and the switch being connected for no reason? I was
> obviously on crack when I set that up.... just found out that was the
> problem.... strange that everything would work perfectly for hours then
> stop working for minutes, etc... probably a table filling up in the switch
> or the router or both :-)
>
> Well, I made you waste a few minutes and wasted a nice sunday morning doing
> this instead of being with my girlfriend, on the bright side I got a spare
> network cable, give me your address if you want a slice :-)
>

	Getting outta bed late, I was about to suggest that there was an 802.11x interface on both the windersbox
	and the DSL router that was active ... *grin* -- but  yea extra wandering cable would certainly be the 
	problem.

	Just something to keep in mind -- I've seen it.

	Alistair.
> Thanks!

RE: Strange logs...

[Mark E. Donaldson]

If both the Linux Box and the Windows box on the same standard subnet of
192.168.20.0/24, and they are connected via a switch, then the packets
should never need to go through the router in the first place. This tells me
the switch is either configured wrong, or is faulty.  Question: Are you
dropping "spoofed IP's" on your router using the NAT table?  

-----Original Message-----
From: netfilter-admin@lists.netfilter.org
[mailto:netfilter-admin@lists.netfilter.org] On Behalf Of Carlos Fernandez
Sanz
Sent: Sunday, January 11, 2004 5:29 AM
To: netfilter
Subject: Re: Strange logs...


> > How reliable is ethereal? I mean, does it see packets as they come 
> > from
the
> > wire or after they have been touched by netfilter?
>
> Ethereal and tcpdump will both see packets off the wire before they get to
> netfilter.   Remember that these packet capturing programs and not

After leaving it working for some more minutes, I started seeing normal
traffic being logged as well. Packets show up in eth1 (internet interface)
both in the logs and ethereal (there's a 100% match). Now, how are the linux
device (eth1) linked to the physical device (NIC) related?

I have to say that if I unplug any of the two cables (the one between the
linux and the switch or the one between the linux and the router) strange
things happen. So I'm really starting to believe that the packet does come
from the wire physically.... no matter how impossible that seems.

As I finished writing that I tried to ping from the windows box to the linux
box and both cables must be connected for ping responses to arrive. This
happens even with all the iptables tables flushed (ie firewall down; no
rules, and ACCEPT as policies for both input and policies).

> > (still, is that possible? How could a packet generated by the 
> > windows
box,
> > which isn't connected to eth1, end up there?).
>
> That's the one bit I can't think of an explanation for.   You don't have
> anything exotic like bridging or vlans enabled in your kernel do you?

Nothing (that I'm aware of, anyway). But this is starting to look more like
a hardware issue to me. Definitely not related to the wiring, but maybe
there's some kind of conflict between my NICs I haven't spotted yet.... so
in case it helps:

  Bus  2, device  11, function  0:
    Ethernet controller: 3Com Corporation 3c905C-TX/TX-M [Tornado] (rev 48).
      IRQ 9.
      Master Capable.  Latency=32.  Min Gnt=10.Max Lat=10.
      I/O at 0xd800 [0xd87f].
      Non-prefetchable 32 bit memory at 0xf4000000 [0xf400007f].
  Bus  2, device  14, function  0:
    Ethernet controller: 3Com Corporation 3c905C-TX/TX-M [Tornado] (#2) (rev
48).
      IRQ 3.
      Master Capable.  Latency=32.  Min Gnt=10.Max Lat=10.
      I/O at 0x8800 [0x887f].
      Non-prefetchable 32 bit memory at 0xf3800000 [0xf380007f].

Thanks for helping out :-)

Re: Strange logs...

[Antony Stone]

On Sunday 11 January 2004 9:49 pm, Mark E. Donaldson wrote:

> If both the Linux Box and the Windows box on the same standard subnet of
> 192.168.20.0/24, and they are connected via a switch, then the packets
> should never need to go through the router in the first place. This tells
> me the switch is either configured wrong, or is faulty.

Don't worry about it - we found a spare cable where it shouldn't have been :)

Problem solved.

Antony.

-- 
Abandon hope, all ye who enter here.
You'll feel much better about things once you do.

                                                     Please reply to the list;
                                                           please don't CC me.