nfqueue hashing on TCP/UDP port

nfqueue hashing on TCP/UDP port

[Jake Owen]

Hello!

tl;dr Is there a technical reason why nfqueue balance as implemented
does not use TCP/UDP ports as well as source/destination IP addresses?

We've been having trouble with the queue hashing algorithm used by
iptable's `--queue-balance` for traffic generated on-box (e.g. by a
squid proxy) where a large percentage of traffic would be TCP, source
IP of the proxy, and one google/microsoft/apple destination IP. This
is made worse if the random seed causes two or more of these high
traffic services to hash to the same queue. We are working on
preserving the original client IP as the source IP to provide
sufficient randomness to balance accurately, but in the meantime have
wondered if balancing by port was not implemented because it was
deemed unnecessary, or because of some technical reason which escapes
me.

I've been able to write a kernel patch which hashes based on protocol,
IP and port for the CentOS 7 kernel we have in production which
_appears_ to be stable. Is there an existing test harness I can use to
validate this implementation? I'm willing to propose a solution for
the latest 5.x kernel if other people think that this is a valid
solution/use case.

Kind Regards,
Jake Owen

Re: nfqueue hashing on TCP/UDP port

[Florian Westphal]

Jake Owen <jake.owen@superloop.com> wrote:
> Hello!
> 
> tl;dr Is there a technical reason why nfqueue balance as implemented
> does not use TCP/UDP ports as well as source/destination IP addresses?

To keep host-to-host comunication on the same queue, for ftp, sip and
other highlevel protocols where a logical connection consists of
multiple tcp/udp flows.

> We've been having trouble with the queue hashing algorithm used by
> iptable's `--queue-balance` for traffic generated on-box (e.g. by a
> squid proxy) where a large percentage of traffic would be TCP, source
> IP of the proxy, and one google/microsoft/apple destination IP. This
> is made worse if the random seed causes two or more of these high
> traffic services to hash to the same queue. We are working on
> preserving the original client IP as the source IP to provide
> sufficient randomness to balance accurately, but in the meantime have
> wondered if balancing by port was not implemented because it was
> deemed unnecessary, or because of some technical reason which escapes
> me.

The latter.  I will add arbitrary hash keying to nft, its currently
only missing from the frontend.

Will put you in CC when its done.

> I'm willing to propose a solution for
> the latest 5.x kernel if other people think that this is a valid
> solution/use case.

With nft this will soon be possible:

queue num jhash ip daddr . tcp sport . tcp dport mod 16

... which will queue to 0-15.

I don't think we need code changes to the xtables backend.

Re: nfqueue hashing on TCP/UDP port

[Jake Owen]

Thank you Florian. I appreciate your thoughtful reply.

> To keep host-to-host comunication on the same queue, for ftp, sip and
> other highlevel protocols where a logical connection consists of
> multiple tcp/udp flows.

That does make a lot of sense.

> I will add arbitrary hash keying to nft, its currently
> only missing from the frontend.
>
> Will put you in CC when its done.

Thank you!

> With nft this will soon be possible:
>
> queue num jhash ip daddr . tcp sport . tcp dport mod 16
>
> ... which will queue to 0-15.
>
> I don't think we need code changes to the xtables backend.

That amount of flexibility looks wonderful.