Re: : getsockopt(fd, SOL_IP, SO_ORIGINAL_DST, sa, &salen) is in fact sometimes returning the source IP instead the destination IP

[Willy Tarreau <w@1wt.eu>]

Hi Lukas,

On Sat, Jan 12, 2019 at 07:01:34PM +0100, Lukas Tribus wrote:
> > Roobesh, do you use the destination address only for logging or
> > anywhere else in the request path ? And could you check if you have
> > nf_conntrack_tcp_loose set as Florian suggests ? I really think he
> > figured it right.
> 
> It's about what we send with the PROXY protocol to the backend server,
> Roobesh reported things like that (src and dst is the same):
> 
> PROXY TCP4 192.220.26.39 192.220.26.39 45066 45066
> PROXY TCP4 192.220.26.39 192.220.26.39 45075 45075
> 
> So the call would actually happen at the beginning of the TCP connection.

That sounds quite shocking to me then. Maybe we're facing such a sequence:

 1) first session comes from this port, and client closes first (FIN)
 2) haproxy then closes with (FIN)
 3) client doesn't respond with the final ACK (e.g. blocked by another
    firewall in between or the client's own conntrack)
 4) the socket on the haproxy side remains in LAST_ACK state and ACKs
    are periodically resent
 5) local conntrack is in TIME_WAIT and expires faster than the largest
    interval between two such ACKs
 6) one of these retransmitted ACKs reopens the connection in reverse
    direction due to nf_conntrack_tcp_loose. The connection is then
    seen in ESTABLISHED state and might be kept a bit longer.
 8) the connection finally expires in the local TCP stack but not yet
    in conntrack.
 7) later the client reuses the same source port while the connection
    is still present in the conntrack table.
 8) assuming tcp_be_liberal is also set, the packets can pass through
    conntrack and establish a new connection to haproxy.
 9) haproxy calls getsockopt(SO_ORIGINAL_DST) and gets the other end
    point since the connection was created at step 6 above in the
    other direction.

I could be wrong on certain specific points above but it looks plausible. 

> Initial report is here:
> https://discourse.haproxy.org/t/send-proxy-not-modifying-some-traffic-with-proxy-ip-port-details/3336

Ah cool, I didn't have all this, thank you!

> Let's see if disabling nf_conntrack_tcp_loose changes things.

Yes this really is the only thing I can think of, and in this case
noone is wrong in this chain (neither kernel nor haproxy). We'll
need to document it in my opinion.

Thanks,
Willy