RE: ebtables on a stick

RE: ebtables on a stick

[Greg Scott]

I have a situation that needs to route mostly and bridge only a little bit.  

I have a private internal LAN, 192.168.10.nnn.  But one host in the internal side needs a real public IP Address, call it 1.2.115.157.  

I have a Linux firewall set up with ebtables.  Interface eth0 faces the Internet and has an IP Address of 1.2.115.146.  Interface eth1 is on the LAN side with IP Address  192.168.10.1.  Bridge br0 currently has no IP Address and bridges eth0 and eth1.  Everyone on the LAN side except my public IP Address host uses the private side of the firewall at 192.160.10.1 for default gateway.  The public IP Address host uses the public side of the firewall at 1.2.115.146 for default gateway.  

The challenge - I need my public IP Address host to communicate with everyone around the Internet, including the private LAN – and I’ll filter that appropriately with iptables.  Everything else should route traditionally.  

This mostly works, except I am having trouble communicating with hosts in the private LAN.  I can communicate anywhere else on the Internet as long as I don’t try to use my own DNS servers in the private LAN to translate names.   After studying ebtables examples and tons of trial and error, I still don’t get it.  Here are the ebtables rules:

[root@ehac-fw2011 firewall-scripts]# ebtables -t broute -Lc
Bridge table: broute

Bridge chain: BROUTING, entries: 9, policy: ACCEPT
-p IPv4 --ip-src 1.2.115.157 -j ACCEPT
-p IPv4 --ip-dst 1.2.115.157 -j ACCEPT
-p ARP --arp-ip-src 1.2.115.157 -j ACCEPT
-p ARP --arp-ip-dst 1.2.115.157 -j ACCEPT
-j redirect  --redirect-target DROP
[root@ehac-fw2011 firewall-scripts]#

When my public IP host pings the Internal LAN, the internal LAN host replies.   Watching tcpdump from the firewall, I can see the echo request come in and go out on both br0 and eth1 and  I see the echo reply come back on eth1.  But then the  reply dies and I never send it back out eth1 over the wire.   And when a private host pings the public host, the echo request dies at the firewall.    So somehow, I am having trouble getting private→public traffic out of br0 and onto eth1.  

What am I missing?

Thanks

- Greg Scott

ebtables on a stick

[Greg Scott]

I have a situation that needs to route mostly and bridge only a little bit.  

I have a private internal LAN, 192.168.10.nnn.  But one host in the internal side needs a real public IP Address, call it 1.2.115.157.  Everything except that public IP host needs to route.  The public host needs to bridge so it can interact with the world.  But it also needs to interact with the internal LAN.  

I have a Linux brouter set up with eth0 facing the Internet, eth1 facing the LAN as follows:

ifconfig eth0 1.2.115.146 mask 255.255.255.240
ifconfig eth1 192.168.10.1 mask 255.255.255.0
brctl addbr br0
brctl addif br0 eth0
brctl addif br0 eth1
ebtables -t broute -A BROUTING -p IPv4 --ip-source 1.2.115.157 -j ACCEPT
ebtables -t broute -A BROUTING -p IPv4 --ip-destination 1.2.115.157 -j ACCEPT
ebtables -t broute -A BROUTING -p arp --arp-ip-src 1.2.115.157 -j ACCEPT
ebtables -t broute -A BROUTING -p arp --arp-ip-dst 1.2.115.157 -j ACCEPT
ebtables -t broute -A BROUTING -j redirect --redirect-target DROP


This mostly works, except I am having trouble communicating with hosts in the private LAN.  I can communicate anywhere else on the Internet, just not in the private LAN.   After studying ebtables examples and tons of trial and error, I still don’t get it.  

When my public IP host pings the internal LAN, the internal LAN host replies, but the replies never get past the brouter.  Watching tcpdump from the firewall, I can see the echo request come in and go out on both br0 and eth1 and I see the echo reply come back on eth1.  But then the reply dies and I never send it back out eth1 over the wire.   And when a private host pings the public host, the echo request dies at the brouter.  So public --> private works, but private --> public including ICMP echo replies die at the brouter.  

I've tried various solutions such as "ip route add 1.2.115.157/32 via br0" without success.  What am I missing?

Thanks

- Greg Scott

Re: ebtables on a stick

[David Lamparter]

On Sun, Nov 27, 2011 at 09:10:08AM -0600, Greg Scott wrote:
> I have a situation that needs to route mostly and bridge only a little bit.  
> 
> I have a private internal LAN, 192.168.10.nnn.  But one host in the internal side needs a real public IP Address, call it 1.2.115.157.  Everything except that public IP host needs to route.  The public host needs to bridge so it can interact with the world.  But it also needs to interact with the internal LAN.  
> 
> I have a Linux brouter set up with eth0 facing the Internet, eth1 facing the LAN as follows:
> 
> ifconfig eth0 1.2.115.146 mask 255.255.255.240
> ifconfig eth1 192.168.10.1 mask 255.255.255.0
[...]

This doesn't answer your question, but your use case is better solved
with proxy arp.

ip route add 1.2.115.157/32 dev eth1
ip neigh add proxy 1.2.115.157 dev eth0
# ... adjust iptables rules to make sure traffic is allowed
# optional, but I'd recommend:
iptables -t raw -I PREROUTING -d 1.2.115.157 -j NOTRACK
iptables -t raw -I PREROUTING -s 1.2.115.157 -j NOTRACK

on the target host:

ip addr add 1.2.115.157/32 dev ethX
ip route add 192.168.10.1/24 dev ethX
ip route add default via 192.168.10.1

no bridge, no ebtables.

you may need to tweak shared_media/icmp redirect settings on the router,
should work as-is though.


-David

RE: ebtables on a stick

[Greg Scott]

> This doesn't answer your question, but your use case is better solved
> with proxy arp.

Proxy arp scares me to death.  I lived through a disaster a few years ago when I messed up a whole colo site when I put in a box with its public NIC set up with Proxy arp.  If I understand what happens, that proxy-arped NIC answers ARP requests for everyone with its own MAC Address.  Fortunately for me, it only lasted a little while around 4AM one morning before I realized what was going on and took it down.  After I went through that experience I promised myself never again for proxy arp.

That's why I started looking at bridging.  

...But maybe there's a way to make my NIC only answer ARPs for certain IP Addresses I care about?  That would nicely solve the problem.  If it works.  

In my earlier near-disaster, I did this:

echo 1 > /proc/sys/net/ipv4/conf/${INET_IFACE}/proxy_arp

to turn on proxy arp.  I wonder if that ip neighbor stuff does it selectively?  

- Greg

Re: ebtables on a stick

[David Lamparter]

On Mon, Nov 28, 2011 at 08:54:09AM -0600, Greg Scott wrote:
> > This doesn't answer your question, but your use case is better solved
> > with proxy arp.
>
> ...But maybe there's a way to make my NIC only answer ARPs for certain IP Addresses I care about?  That would nicely solve the problem.  If it works.  

"ip neigh add proxy 1.2.3.4 dev eth0". Please look at the example i gave
you.

> In my earlier near-disaster, I did this:
> 
> echo 1 > /proc/sys/net/ipv4/conf/${INET_IFACE}/proxy_arp

NO. You do not touch this switch. Be afraid of this switch.

The instructions i gave you are complete. This switch is not involved.

> to turn on proxy arp.  I wonder if that ip neighbor stuff does it selectively?  

We already had this discussion in July.

"ip neigh add proxy" is independent of /proc/.../proxy_arp.


-David

RE: ebtables on a stick

[Greg Scott]

> We already had this discussion in July.
>
> "ip neigh add proxy" is independent of /proc/.../proxy_arp.

We did discuss this back in July but never finished.  I posted a bunch
of questions about "ip neigh add proxy" because I haven't seen much in
the way of documentation.  When nobody answered, I left it alone for a
while.  

I found a discussion starting here:
http://lkml.indiana.edu/hypermail/linux/kernel/0110.2/0523.html

where some of the kernel folks are suggesting "ip neigh add proxy" is
deprecated.  By now that discussion is 10 years old and it apparently is
still around.  But for how long?

Appendix B of this 2007 document, http://linux-ip.net/html/index.html,
by Martin Brown also says "ip neigh add proxy" is deprecated.  

So now, in addition to my paranoia about proxy ARP in general, I have
this fear I will build a bunch of scripting around "ip neigh add proxy"
only to find it's gone in a future kernel, leaving me in a world of hurt
again.  

So humor me and convince me that "ip neigh add" is not on the chopping
block so I can try it with a little bit of confidence.  

Thanks

- Greg

Re: ebtables on a stick

[David Lamparter]

On Tue, Nov 29, 2011 at 04:05:18AM -0600, Greg Scott wrote:
> > We already had this discussion in July.
> >
> > "ip neigh add proxy" is independent of /proc/.../proxy_arp.
> 
> We did discuss this back in July but never finished.  I posted a bunch
> of questions about "ip neigh add proxy" because I haven't seen much in
> the way of documentation.  When nobody answered, I left it alone for a
> while.  
> 
> I found a discussion starting here:
> http://lkml.indiana.edu/hypermail/linux/kernel/0110.2/0523.html
> 
> where some of the kernel folks are suggesting "ip neigh add proxy" is
> deprecated.  By now that discussion is 10 years old and it apparently is
> still around.  But for how long?

While the feature is still around after 10 years, Alexey isn't touching
the network stack much anymore. Funny, that.

> Appendix B of this 2007 document, http://linux-ip.net/html/index.html,
> by Martin Brown also says "ip neigh add proxy" is deprecated.
>
> So now, in addition to my paranoia about proxy ARP in general, I have
> this fear I will build a bunch of scripting around "ip neigh add proxy"
> only to find it's gone in a future kernel, leaving me in a world of hurt
> again.
>
> So humor me and convince me that "ip neigh add" is not on the chopping
> block so I can try it with a little bit of confidence.

I'll humor myself instead. I'm shipping this as a feature to customers,
so if it comes up for chopping I'll have to maintain it myself
worst-case.

(I wasn't actually aware that this is supposed to be deprecated. Yes,
it's buggy in that you need "arp" to show the entries, but that's exacly
that - a low-severity bug.)


-David

RE: ebtables on a stick

[Greg Scott]

Well this seems like the only way to get done what I need, so if our
votes count, we'll vote to leave it in.  Let me get some sleep and I'll
try it and report back here later.  

- Greg



-----Original Message-----
From: David Lamparter [mailto:equinox@diac24.net] 
Sent: Tuesday, November 29, 2011 4:24 AM
To: Greg Scott
Cc: David Lamparter; netdev@vger.kernel.org
Subject: Re: ebtables on a stick

On Tue, Nov 29, 2011 at 04:05:18AM -0600, Greg Scott wrote:
> > We already had this discussion in July.
> >
> > "ip neigh add proxy" is independent of /proc/.../proxy_arp.
> 
> We did discuss this back in July but never finished.  I posted a bunch
> of questions about "ip neigh add proxy" because I haven't seen much in
> the way of documentation.  When nobody answered, I left it alone for a
> while.  
> 
> I found a discussion starting here:
> http://lkml.indiana.edu/hypermail/linux/kernel/0110.2/0523.html
> 
> where some of the kernel folks are suggesting "ip neigh add proxy" is
> deprecated.  By now that discussion is 10 years old and it apparently
is
> still around.  But for how long?

While the feature is still around after 10 years, Alexey isn't touching
the network stack much anymore. Funny, that.

> Appendix B of this 2007 document, http://linux-ip.net/html/index.html,
> by Martin Brown also says "ip neigh add proxy" is deprecated.
>
> So now, in addition to my paranoia about proxy ARP in general, I have
> this fear I will build a bunch of scripting around "ip neigh add
proxy"
> only to find it's gone in a future kernel, leaving me in a world of
hurt
> again.
>
> So humor me and convince me that "ip neigh add" is not on the chopping
> block so I can try it with a little bit of confidence.

I'll humor myself instead. I'm shipping this as a feature to customers,
so if it comes up for chopping I'll have to maintain it myself
worst-case.

(I wasn't actually aware that this is supposed to be deprecated. Yes,
it's buggy in that you need "arp" to show the entries, but that's exacly
that - a low-severity bug.)


-David

Re: ebtables on a stick

[Michal Soltys]

On 11-11-28 15:39, David Lamparter wrote:
> On Sun, Nov 27, 2011 at 09:10:08AM -0600, Greg Scott wrote:
>>  I have a situation that needs to route mostly and bridge only a
>>  little bit.
>>
>>  I have a private internal LAN, 192.168.10.nnn.  But one host in the
>>  internal side needs a real public IP Address, call it 1.2.115.157.
>>  Everything except that public IP host needs to route.  The public
>>  host needs to bridge so it can interact with the world.  But it also
>>  needs to interact with the internal LAN.
>>
>>  I have a Linux brouter set up with eth0 facing the Internet, eth1
>>  facing the LAN as follows:
>>
>>  ifconfig eth0 1.2.115.146 mask 255.255.255.240 ifconfig eth1
>>  192.168.10.1 mask 255.255.255.0
> [...]
> 
> This doesn't answer your question, but your use case is better solved
> with proxy arp.
> 
> ip route add 1.2.115.157/32 dev eth1
> ip neigh add proxy 1.2.115.157 dev eth0
> # ... adjust iptables rules to make sure traffic is allowed
> # optional, but I'd recommend:
> iptables -t raw -I PREROUTING -d 1.2.115.157 -j NOTRACK
> iptables -t raw -I PREROUTING -s 1.2.115.157 -j NOTRACK
> 
> on the target host:
> 
> ip addr add 1.2.115.157/32 dev ethX
> ip route add 192.168.10.1/24 dev ethX
> ip route add default via 192.168.10.1


In addition to what David wrote, you might want specify 'src' option on
certain routes, and/or specify scopes by hand.

For example if the lan host in question has something like:

ip add add 192.168.10.2/24 broad + dev eth0 scope link ip add add
1.2.115.157/32 dev eth0 scope global ip ro add unicast default via
192.168.10.1 dev eth0 src 1.2.115.157

Then the 'src' will prefer public ip for non-lan communication (even
without scopes).

Make sure, when you (likely) do snat with iptables on the router, to
skip this address in such case (in case of not using notrack).

BTW, scopes can help with masquerade target on router (not important in
your case as you have static addresses, but still), as it will choose
candidates with global scope.

RE: ebtables on a stick

[Greg Scott]

Well this is frustrating.  Now my public host can communicate anywhere it wants internally but nothing outside. Maddening - the exact opposite problem I had before.  

Here is the config as it sits right now:

Public IP host (a Windows XP system placeholder for now)
IP 1.2.115.157, default gw 1.2.115.146.

My test internal host - 192.168.10.2.  This one is an ordinary Windows server.

Firewall eth0 - 1.2.115.146/28
Firewall eth1 - 192.168.10.1. 

/sbin/ip neigh add proxy 1.2.115.157 dev eth0
/sbin/ip route add 1.2.115.157/32 dev eth1

As a troubleshooting step, I also put in:
/sbin/ip addr add 1.2.115.146/28 dev eth1; so now both eth0 and eth1 have the same IP Address.  This feels ugly and I think I'll take it out because it made no difference.  

All evidence of ebtables rules and brnn interfaces are gone.  

And the relevant iptables rules:

$IPTABLES -t nat -A POSTROUTING -s $1.2.115.157 -j ACCEPT

$IPTABLES -A FORWARD -s 1.2.115.157 -j ACCEPT
$IPTABLES -A FORWARD -s 192.168.10.0/24 -d 1.2.115.157 -j ACCEPT
$IPTABLES -A FORWARD -p TCP --dport 1720 -d $ADR -j allowed
$IPTABLES -A FORWARD -p TCP -s $MGMT_IP -d $ADR -j allowed

$MGMT_IP is the public side of my home office.  

And finally:

$IPTABLES -t nat -A PREROUTING -d 1.2.115.157 -j ACCEPT

When my public host pings, say, google, watching on the firewall with tcpdump, I see the ICMP echo request come in on eth1 and go out eth0.  So far so good. The ICMP echo reply comes back on eth0, still good.  But I never forward it over to eth1 and it dies right there.  The public host never sees the reply. 

Nothing nats to/from this address.  Tcpdump shows the IP Addresses I expect to see.  And I don't have any "-t raw -j NOTRACK" stuff because I will eventually want to use the H.323 conntrack helper with this.  

Maybe a good night's sleep will help.

- Greg

Re: ebtables on a stick

[David Lamparter]

On Wed, Nov 30, 2011 at 11:46:02PM -0600, Greg Scott wrote:
> Well this is frustrating.  Now my public host can communicate anywhere it wants internally but nothing outside. Maddening - the exact opposite problem I had before.  
> 
> Here is the config as it sits right now:
> 
> Public IP host (a Windows XP system placeholder for now)
> IP 1.2.115.157, default gw 1.2.115.146.

Note that since the IP should be 1.2.115.157/_32_, it doesn't make any
difference whether you use 1.2.115.146 for the defgw or 192.168.10.1,
since both are out-of-subnet.

> Firewall eth0 - 1.2.115.146/28
> Firewall eth1 - 192.168.10.1. 
> 
> /sbin/ip neigh add proxy 1.2.115.157 dev eth0
> /sbin/ip route add 1.2.115.157/32 dev eth1
> 
> As a troubleshooting step, I also put in:
> /sbin/ip addr add 1.2.115.146/28 dev eth1; so now both eth0 and eth1 have the same IP Address.  This feels ugly and I think I'll take it out because it made no difference.  

I agree, please remove.

> And the relevant iptables rules:
> 
> $IPTABLES -t nat -A POSTROUTING -s $1.2.115.157 -j ACCEPT
> $IPTABLES -t nat -A PREROUTING -d 1.2.115.157 -j ACCEPT
> 
> $IPTABLES -A FORWARD -s 1.2.115.157 -j ACCEPT

Where is the reverse rule of this? -d 1.2.115.157 -j ACCEPT

> $IPTABLES -A FORWARD -s 192.168.10.0/24 -d 1.2.115.157 -j ACCEPT
> $IPTABLES -A FORWARD -p TCP --dport 1720 -d $ADR -j allowed
> $IPTABLES -A FORWARD -p TCP -s $MGMT_IP -d $ADR -j allowed

(what's $ADR?)
[...]
> The ICMP echo reply comes back on eth0, still good.  But I never forward it over to eth1 and it dies right there.  The public host never sees the reply. 

Firewall rules?


-David

RE: ebtables on a stick

[Greg Scott]

Oh - woops:

>> IP 1.2.115.157, default gw 1.2.115.146.
>
> Note that since the IP should be 1.2.115.157/_32_, it doesn't make any
> difference whether you use 1.2.115.146 for the defgw or 192.168.10.1,
> since both are out-of-subnet.

Right this second, the whole thing really is 1.2.115.157/28, or mask
255.255.255.240.  So from that host's point of view the default gw makes
sense.  You really want me to set that up as a /32?  I don't get it.  
.
.
.
>> As a troubleshooting step, I also put in:
>> /sbin/ip addr add 1.2.115.146/28 dev eth1; so now both eth0 and eth1
have the same IP Address.  This feels >ugly and I think I'll take it out
because it made no difference.  
>
>I agree, please remove.

I put it in last night because this was the "trick" I had to do years
ago when I used proxy ARP and that scary switch turning it on for
everything.  Btw, there are still write-ups out there advocating that
approach.
.
.
.
>> $IPTABLES -A FORWARD -s 1.2.115.157 -j ACCEPT
>
> Where is the reverse rule of this? -d 1.2.115.157 -j ACCEPT
>
>> $IPTABLES -A FORWARD -s 192.168.10.0/24 -d 1.2.115.157 -j ACCEPT
>> $IPTABLES -A FORWARD -p TCP --dport 1720 -d $ADR -j allowed
>> $IPTABLES -A FORWARD -p TCP -s $MGMT_IP -d $ADR -j allowed
>
>(what's $ADR?)

I was having trouble keeping my eyes open last night. $ADR ends up being
1.2.115.157.  That's the reverse rule you were looking for.  And then
$MGMT_IP is the rule that lets me in no matter what protocol/port.
Allowed is a user defined chain that checks for the right TCP handshake.
I should probably change the $MGMT_IP rule to just ACCEPT.  This won't
matter though - I did my ping tests last night physically sitting in
front of that XP host and trying to ping google.  

> Firewall rules?

There are a bunch of 'em to handle a website and a few ftp sites and a
PPTP VPN and the H.323 stuff I'm trying to hook back up.  I have a
little script named "allow-all-with-nat" that turns all the filtering
rules off, leaving all the NAT rules in place.  Running that makes no
difference.  The last time I posted a whole ruleset I was roundly
chastised because it was too complicated and we went off into a rathole
about H.323.  

- Greg

RE: ebtables on a stick

[Greg Scott]

Well now, this is interesting.  Hopefully the emailer won't garble
below.  

3.4.2.129 is me.  I sent some pings from here to my public host.
Listening on eth0 on the firewall, here they come.  But the interesting
part is trying again, this time listening on eth1 on the firewall.  I
forwarded them! So why no echo reply? That makes me wonder if I did
something dumb on the test host I set up.  I'll probably hop in the car
and drive over there and take a look.  

[root@ehac-fw2011 firewall-scripts]#
[root@ehac-fw2011 firewall-scripts]# /usr/sbin/tcpdump -i eth0 net
3.4.2.128/27 and port not 22 -nn
tcpdump: verbose output suppressed, use -v or -vv for full protocol
decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
08:18:29.653384 IP 3.4.2.129 > 1.2.115.157: ICMP echo request, id 1, seq
72, length 40
08:18:34.201405 IP 3.4.2.129 > 1.2.115.157: ICMP echo request, id 1, seq
73, length 40
08:18:39.209183 IP 3.4.2.129 > 1.2.115.157: ICMP echo request, id 1, seq
74, length 40
08:18:44.202242 IP 3.4.2.129 > 1.2.115.157: ICMP echo request, id 1, seq
75, length 40
^C
4 packets captured
4 packets received by filter
0 packets dropped by kernel
[root@ehac-fw2011 firewall-scripts]# /usr/sbin/tcpdump -i eth1 net
3.4.2.128/27 and port not 22 -nn
tcpdump: verbose output suppressed, use -v or -vv for full protocol
decode
listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes
08:18:58.871185 IP 3.4.2.129 > 1.2.115.157: ICMP echo request, id 1, seq
76, length 40
08:19:03.719756 IP 3.4.2.129 > 1.2.115.157: ICMP echo request, id 1, seq
77, length 40
08:19:08.727342 IP 3.4.2.129 > 1.2.115.157: ICMP echo request, id 1, seq
78, length 40
08:19:13.704391 IP 3.4.2.129 > 1.2.115.157: ICMP echo request, id 1, seq
79, length 40
^C
4 packets captured
4 packets received by filter
0 packets dropped by kernel
[root@ehac-fw2011 firewall-scripts]#


Or - easier than hopping in the car - I did set it up to allow RDP in.
I wonder . . .

Yup, I can do an RDP session into it from here.  And sure enough, the
Windows personal firewall is turned on.  Turning it off - holy moley,
round trip pings from here!  And now I can also ping google from that
test workstation.  

I wonder what's different this morning?  A bunch of firewall conntrack
entries would have expired by now.  I was changing rules fast and
furious last night, maybe there were some stale conntrack entries that
messed with my head.  But this morning it's working as expected.

Must be an important project, otherwise it wouldn't give me all this
trouble.

- Greg

Re: ebtables on a stick

[David Lamparter]

On Thu, Dec 01, 2011 at 08:39:07AM -0600, Greg Scott wrote:
> I wonder what's different this morning?  A bunch of firewall conntrack
> entries would have expired by now.  I was changing rules fast and
> furious last night, maybe there were some stale conntrack entries that
> messed with my head.  But this morning it's working as expected.
> 
> Must be an important project, otherwise it wouldn't give me all this
> trouble.

Heh. Nice to see you got it to work.

A few last words about the subnet mask on the windows box:

The 1.2.115.144/28 subnet is on eth0 on your router. The windows host
with 1.2.115.157 is _not_ connected to that subnet. It is on eth1 on
your router, and it can't reach any hosts from 1.2.115.144/28 without
going through your router, so /32 is the correct configuration there.

That /32 just means "on my ethernet segment i'm alone with that
address". If the windows box has /28 as subnet mask, it will try to ARP
for other hosts from that subnet, instead of going through the router.
So, that'll break connectivity to them...


-David

RE: ebtables on a stick

[Greg Scott]

> That /32 just means "on my ethernet segment i'm alone with that
> address". If the windows box has /28 as subnet mask, it will try to
ARP
> for other hosts from that subnet, instead of going through the router.
> So, that'll break connectivity to them...

I never did get this. Right now, it's a test Windows box, but eventually
it will be something else.  It's connected to eth1 and needs to go
through the router - right - so how does it find its gateway at
1.2.115.146 on eth0?  And setting the mask to /32 makes it even
stranger.

As long as I can get to it - why don't I try setting it to /32 and let's
see what happens.  Worst case, I have to jump in the car I guess.  Well,
Windows won't allow a mask of 255.255.255.255.  I wonder how the real
stuff I'll eventually connect at that IP Address will behave with a /32
mask?

Digging a little deeper...

>From Windows, arp -a shows both 1.2.114.146 and 192.168.10.1 with a MAC
Address of 00-0d-88-31-d8-24.  Looking on the firewall with ip link show
- sure enough, that's the MAC Address of eth1.  

I wonder what happens with some of the stuff I'm NATing?  There's a
website at public IP 1.2.115.151, private 192.168.10.8.  Pinging
1.2.115.151 and then arp -a; it shows the firewall eth1 MAC Address.
Makes sense - it is NATed after all.

Launching IE from that host - nope - that NATed website doesn't come up.
But it doesn't time out, it errors right away, suggesting it was
rejected instead of dropped.  Well, OK - that's probably because I don't
have any firewall rules to handle this case (and probably don't need any
because this will never happen in production), so it went right to the
firewall itself and was properly rejected.  

But the eventual box at this address will probably have its own built-in
management website.  I wonder what happens with telnets on port 80 and
443 to it?  They both work; the firewall forwards it and the Windows box
rejects it.  So when there's a real website sitting there, it should be
OK.  

So the /28 mask feels OK so far...

- Greg

Re: ebtables on a stick

[David Lamparter]

On Thu, Dec 01, 2011 at 09:29:59AM -0600, Greg Scott wrote:
> > That /32 just means "on my ethernet segment i'm alone with that
> > address". If the windows box has /28 as subnet mask, it will try to
> ARP
> > for other hosts from that subnet, instead of going through the router.
> > So, that'll break connectivity to them...
> 
> I never did get this. Right now, it's a test Windows box, but eventually
> it will be something else.  It's connected to eth1 and needs to go
> through the router - right - so how does it find its gateway at
> 1.2.115.146 on eth0?  And setting the mask to /32 makes it even
> stranger.
> 
> As long as I can get to it - why don't I try setting it to /32 and let's
> see what happens.  Worst case, I have to jump in the car I guess.  Well,
> Windows won't allow a mask of 255.255.255.255.  I wonder how the real
> stuff I'll eventually connect at that IP Address will behave with a /32
> mask?

The default gateway is always assumed to be on-link / arp-able. Because
if it wasn't, it wouldn't be an usable default gateway...

On the box I'm writing this mail from right now:

# ip -4 a l eth0; ip r l match 0.0.0.0; ip r l exact 10.255.255.1
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
state UNKNOWN qlen 1000
    inet 87.106.131.203/32 scope global eth0
default via 10.255.255.1 dev eth0
10.255.255.1 dev eth0  scope link

(that config is pushed through DHCP)

> I wonder what happens with some of the stuff I'm NATing?  There's a
> website at public IP 1.2.115.151, private 192.168.10.8.  Pinging
> 1.2.115.151 and then arp -a; it shows the firewall eth1 MAC Address.
> Makes sense - it is NATed after all.

I don't work with windows and have no clue what's happening there :)

Well, as long as it works, I guess that's fine.


-David

RE: ebtables on a stick

[Greg Scott]

> # ip -4 a l eth0; ip r l match 0.0.0.0; ip r l exact 10.255.255.1
> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
> state UNKNOWN qlen 1000
>     inet 87.106.131.203/32 scope global eth0
> default via 10.255.255.1 dev eth0
> 10.255.255.1 dev eth0  scope link

Wow, how does that even work?  Every time I think I know something, I
get challenged.  I should make that a slogan.

- Greg

Re: ebtables on a stick

[Michal Soltys]

On 11-12-01 06:46, Greg Scott wrote:
> Well this is frustrating.  Now my public host can communicate anywhere
> it wants internally but nothing outside. Maddening - the exact
> opposite problem I had before.
> 
> 
> $IPTABLES -A FORWARD -s 1.2.115.157 -j ACCEPT
> $IPTABLES -A FORWARD -s 192.168.10.0/24 -d 1.2.115.157 -j ACCEPT
> $IPTABLES -A FORWARD -p TCP --dport 1720 -d $ADR -j allowed
> $IPTABLES -A FORWARD -p TCP -s $MGMT_IP -d $ADR -j allowed
> 

And accepting traffic to 1.2.115.157 from the outside ? Are there any -m
state / -m conntrack --ctstate entries in your rules ?

RE: ebtables on a stick

[Greg Scott]

Yes.  1.2.115.157 will eventually be an H.323 codec and will need to accept incoming calls.  For right now, it's just a test Windows system I had handy.  

I should have copied you on some of the other emails this morning.  They should be in the netdev list but I'll forward them privately too.

- Greg



-----Original Message-----
From: Michal Soltys [mailto:soltys@ziu.info] 
Sent: Thursday, December 01, 2011 11:44 AM
To: Greg Scott
Cc: David Lamparter; netdev@vger.kernel.org
Subject: Re: ebtables on a stick

On 11-12-01 06:46, Greg Scott wrote:
> Well this is frustrating.  Now my public host can communicate anywhere
> it wants internally but nothing outside. Maddening - the exact
> opposite problem I had before.
> 
> 
> $IPTABLES -A FORWARD -s 1.2.115.157 -j ACCEPT
> $IPTABLES -A FORWARD -s 192.168.10.0/24 -d 1.2.115.157 -j ACCEPT
> $IPTABLES -A FORWARD -p TCP --dport 1720 -d $ADR -j allowed
> $IPTABLES -A FORWARD -p TCP -s $MGMT_IP -d $ADR -j allowed
> 

And accepting traffic to 1.2.115.157 from the outside ? Are there any -m
state / -m conntrack --ctstate entries in your rules ?

Re: ebtables on a stick

[Michal Soltys]

On 11-12-01 17:56, Greg Scott wrote:
>>  # ip -4 a l eth0; ip r l match 0.0.0.0; ip r l exact 10.255.255.1
>>  2: eth0:<BROADCAST,MULTICAST,UP,LOWER_UP>  mtu 1500 qdisc pfifo_fast
>>  state UNKNOWN qlen 1000
>>      inet 87.106.131.203/32 scope global eth0
>>  default via 10.255.255.1 dev eth0
>>  10.255.255.1 dev eth0  scope link

Aside from addresses, same config I had when I was on adsl.

>
> Wow, how does that even work?  Every time I think I know something, I
> get challenged.  I should make that a slogan.
>
> - Greg
>

BTW:

You should be able to avoid whole proxy thing altogether (on your 
router), by doing:

ip add add 1.2.115.157/32 dev eth0
ip ro del table local 192.168.99.5/32 dev eth0
ip route add 1.2.115.157/32 dev eth1

instead of:

ip neigh add proxy 1.2.115.157 dev eth0

The former will still expose the address, but it will be not routable 
inside the router, so will get passed to your internal host. (this 
assumes your kernel is compiled with advanced routing).

Re: ebtables on a stick

[David Lamparter]

On Fri, Dec 02, 2011 at 04:40:13PM +0100, Michal Soltys wrote:
> You should be able to avoid whole proxy thing altogether (on your 
> router), by doing:
> 
> ip add add 1.2.115.157/32 dev eth0
> ip ro del table local 192.168.99.5/32 dev eth0
> ip route add 1.2.115.157/32 dev eth1

... and any application that looks at the local interface addresses
(e.g. ntpd, bind, etc.) will get thoroughly confused. Oh and worst-case
(i don't know/didn't check) source address selection as well.

I'd advise against it.

> instead of:
> 
> ip neigh add proxy 1.2.115.157 dev eth0


-David

RE: ebtables on a stick

[Greg Scott]

> You should be able to avoid whole proxy thing altogether (on your
router), by doing:
>
> ip add add 1.2.115.157/32 dev eth0
> ip ro del table local 192.168.99.5/32 dev eth0 ip route add
1.2.115.157/32 dev eth1

What's the 192.168.99.5/21 address?

- Greg

RE: ebtables on a stick

[Greg Scott]

..and as much as I went into the proxy thing kicking and screaming, it
does seem clean and simple now that I finally did it.

The routing still has me confused - my public host is on the router's
eth1, its gateway is on eth0, but it all still works.  I should probably
reboot that router at some point soon to make sure there aren't any
hidden legacies from my old bridge config left over.  

- Greg


-----Original Message-----
From: David Lamparter [mailto:equinox@diac24.net] 
Sent: Friday, December 02, 2011 10:05 AM
To: Michal Soltys
Cc: Greg Scott; David Lamparter; netdev@vger.kernel.org
Subject: Re: ebtables on a stick

On Fri, Dec 02, 2011 at 04:40:13PM +0100, Michal Soltys wrote:
> You should be able to avoid whole proxy thing altogether (on your 
> router), by doing:
> 
> ip add add 1.2.115.157/32 dev eth0
> ip ro del table local 192.168.99.5/32 dev eth0
> ip route add 1.2.115.157/32 dev eth1

... and any application that looks at the local interface addresses
(e.g. ntpd, bind, etc.) will get thoroughly confused. Oh and worst-case
(i don't know/didn't check) source address selection as well.

I'd advise against it.

> instead of:
> 
> ip neigh add proxy 1.2.115.157 dev eth0


-David

Re: ebtables on a stick

[Michal Soltys]

On 11-12-02 17:06, Greg Scott wrote:
>>  You should be able to avoid whole proxy thing altogether (on your
> router), by doing:
>>
>>  ip add add 1.2.115.157/32 dev eth0
>>  ip ro del table local 192.168.99.5/32 dev eth0 ip route add
> 1.2.115.157/32 dev eth1
>
> What's the 192.168.99.5/21 address?
>

Should be 1.2.115.157/32 (hasty copy paste).

RE: ebtables on a stick

[Greg Scott]

OK.  But I dunno....  I set eth0 on the router with the same address as
the real host behind it on eth1.  So something comes in on eth0 for
1.2.115.157.  The router has that as its own address now, plus a route
to somebody else with the same address on eth1.  But as far as the
router/firewall is concerned, that packet is already delivered - why
would it forward it out on eth1?

- Greg



-----Original Message-----
From: Michal Soltys [mailto:soltys@ziu.info] 
Sent: Friday, December 02, 2011 10:16 AM
To: Greg Scott
Cc: David Lamparter; netdev@vger.kernel.org
Subject: Re: ebtables on a stick

On 11-12-02 17:06, Greg Scott wrote:
>>  You should be able to avoid whole proxy thing altogether (on your
> router), by doing:
>>
>>  ip add add 1.2.115.157/32 dev eth0
>>  ip ro del table local 192.168.99.5/32 dev eth0 ip route add
> 1.2.115.157/32 dev eth1
>
> What's the 192.168.99.5/21 address?
>

Should be 1.2.115.157/32 (hasty copy paste).

Re: ebtables on a stick

[Michal Soltys]

On 11-12-02 17:20, Greg Scott wrote:
> OK.  But I dunno....  I set eth0 on the router with the same address as
> the real host behind it on eth1.  So something comes in on eth0 for
> 1.2.115.157.  The router has that as its own address now, plus a route
> to somebody else with the same address on eth1.  But as far as the
> router/firewall is concerned, that packet is already delivered - why
> would it forward it out on eth1?
>

Where the packet gets delivered is decided by the routing - and the very 
first table traversed is local - which is auto filled by the kernel. But 
that routing rule still can be forcibly removed, after which the next 
matching one is the one added manually - after which the packet will end 
in FORWARD, instead of INPUT.

(and keep in mind earlier David's warning about confusing 
programs/services - it's still doable, but requires more manual labor - 
proxy is certianly cleaner and just works)