fwmark routing and OpenVPN...

[Marco Gaiarin <gaio@sv.lnf.it>]

I use in my firewalls (debian stretch, kernel 4.9.0-12-amd64, iptables
1.6.0+snapshot20161117-6) by years different internet connection in
load balancing (route based) using fwmarks.

Recently (for COVID, of course) i've setup on this box an OpenVPN
server, listening on all connection/interfaces.
OpenVPN client are point towards an A record DNS-ip aliased on all the
three connection (eg, clients roandomly select one of the connection).

Users complain that have to retry many times before they connect.
Server side i see:

	Sep 17 12:07:14 tank ovpn-server[55840]: 45.13.80.130:45196 TLS: Initial packet from [AF_INET]45.13.80.130:45196, sidC3fc890 cad5507d

Googling around i've hit a page (i've lost link) that lit me a lamp.


Sniffing traffic on all three interfaces (currently) i get:

(line 1, inbound)
  127 270.659375004 45.13.80.130 → 88.147.114.200 OpenVPN 86 MessageType: P_CONTROL_HARD_RESET_CLIENT_V2
  128 271.896770524 45.13.80.130 → 88.147.114.200 OpenVPN 86 MessageType: P_CONTROL_HARD_RESET_CLIENT_V2
  129 276.834533759 45.13.80.130 → 88.147.114.200 OpenVPN 86 MessageType: P_CONTROL_HARD_RESET_CLIENT_V2
  130 285.709393774 45.13.80.130 → 88.147.114.200 OpenVPN 86 MessageType: P_CONTROL_HARD_RESET_CLIENT_V2
  131 301.564398903 45.13.80.130 → 88.147.114.200 OpenVPN 86 MessageType: P_CONTROL_HARD_RESET_CLIENT_V2
  132 355.779240076 45.13.80.130 → 88.147.114.200 OpenVPN 86 MessageType: P_CONTROL_HARD_RESET_CLIENT_V2
  133 356.937076259 45.13.80.130 → 88.147.114.200 OpenVPN 86 MessageType: P_CONTROL_HARD_RESET_CLIENT_V2
  134 361.549173084 45.13.80.130 → 88.147.114.200 OpenVPN 86 MessageType: P_CONTROL_HARD_RESET_CLIENT_V2
  135 368.724624378 45.13.80.130 → 88.147.114.200 OpenVPN 86 MessageType: P_CONTROL_HARD_RESET_CLIENT_V2

(line 2, outbound)
   33 432.043899703 88.37.116.142 → 45.13.80.130 OpenVPN 96 MessageType: P_CONTROL_HARD_RESET_SERVER_V2
   34 433.201556082 88.37.116.142 → 45.13.80.130 OpenVPN 96 MessageType: P_CONTROL_HARD_RESET_SERVER_V2
   35 437.286099130 88.37.116.142 → 45.13.80.130 OpenVPN 84 MessageType: P_CONTROL_HARD_RESET_SERVER_V2
   36 437.813630238 88.37.116.142 → 45.13.80.130 OpenVPN 92 MessageType: P_ACK_V1
   37 444.989645902 88.37.116.142 → 45.13.80.130 OpenVPN 96 MessageType: P_CONTROL_HARD_RESET_SERVER_V2

(line 3, outbound)
   28 350.793044884 37.186.212.162 → 45.13.80.130 OpenVPN 96 MessageType: P_CONTROL_HARD_RESET_SERVER_V2
   29 351.959606450 37.186.212.162 → 45.13.80.130 OpenVPN 84 MessageType: P_CONTROL_HARD_RESET_SERVER_V2
   30 352.030123423 37.186.212.162 → 45.13.80.130 OpenVPN 92 MessageType: P_ACK_V1
   31 356.846975139 37.186.212.162 → 45.13.80.130 OpenVPN 84 MessageType: P_CONTROL_HARD_RESET_SERVER_V2
   32 356.968050772 37.186.212.162 → 45.13.80.130 OpenVPN 92 MessageType: P_ACK_V1
   33 364.847600208 37.186.212.162 → 45.13.80.130 OpenVPN 84 MessageType: P_CONTROL_HARD_RESET_SERVER_V2
   34 365.842935732 37.186.212.162 → 45.13.80.130 OpenVPN 92 MessageType: P_ACK_V1
   35 379.981282619 37.186.212.162 → 45.13.80.130 OpenVPN 84 MessageType: P_CONTROL_HARD_RESET_SERVER_V2
   36 381.698151394 37.186.212.162 → 45.13.80.130 OpenVPN 92 MessageType: P_ACK_V1

and exactly what i suppose: openvpn receive on one interface, but
reply traffic flow to another.


My iptables rules are pretty clean my mangle PREROUTING chain is:

 Chain PREROUTING (policy ACCEPT 129M packets, 99G bytes)
  pkts bytes target     prot opt in     out     source               destination         
 5406M 3721G CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0            CONNMARK restore
   17M 3039M MARK       all  --  vlan192 *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0xf0 MARK xset 0x10/0xf0
   14M 2892M MARK       all  --  vlan193 *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0xf0 MARK xset 0x30/0xf0
  399K   98M MARK       all  --  ppp0   *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0xf0 MARK xset 0x20/0xf0

and AFAI've understood 'netfilter packet flow' schema,
mangle/PREROUTING get evaluated also for input traffic.


This is aside effect of using this for 'local terminated udp traffic'?
Eg:

 + for UDP and TCP traffic that flow in FORWARD, conntrack do their job.

 + for TCP traffic in INPUT, conntrack CAN track the connection, and so
   work.

 + for UDP traffic in INPUT, conntrack CANNOT track the connection, and
   so when the local 'openvpn' process inject th UDP response to the
   client, there's no tracking at all?

But last sentences does not match very well with the behaviour: when
OpenVPN connection is established (eg, when random input interface match
with random output interface ;), connection are pretty solid, so my
system 'remember' the interface... but this is probably OpenVPN
process itself, not the connection tracking?


What i'm missing? Thanks.

-- 
dott. Marco Gaiarin				        GNUPG Key ID: 240A3D66
  Associazione ``La Nostra Famiglia''          http://www.lanostrafamiglia.it/
  Polo FVG   -   Via della Bontà, 7 - 33078   -   San Vito al Tagliamento (PN)
  marco.gaiarin(at)lanostrafamiglia.it   t +39-0434-842711   f +39-0434-842797

		Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA!
      http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000
	(cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA)