NFQUEUE usage and interaction with later chain rules

NFQUEUE usage and interaction with later chain rules

[Athanasius]

  I'm in the middle of trying to implement some firewall bans based on
GeoIP data.  It seemed that implementing an NFQUEUE userspace helper
to do the lookups and decision making was the most feasible solution,
but I have run into some issues.  I can probably work around them,
but it does make this setup more fragile than I'd like.

  So, I'd like to ensure I'm properly understanding some things.

  1. The only practical verdicts the userspace helper can return are
  ACCEPT or DROP.  Checking through some documentation, examples, and
  kernel code I've come across 'STOLEN', 'QUEUE' and 'REPEAT', but I
  cannot find any actual documentation on what these might achieve.

  2. There appears to be no way to have an NFQUEUE rule act in a manner
  where the userspace verdict can cause subsequent rules in the chain to
  be considered.

    I'd specifically like my userspace helper to be able to say 'DROP',
  but otherwise allow rule evaluation to continue in the chain, rather
  than blindly accepting all else.

    If I could cause rule evaluation to jump to another chain that would
  also be acceptable, simply placing rules that would normally be later
  in the current chain into that one.

As a result I'm going to have to very carefully consider the nature and
position of my iptables chain rules:

  1. Anything else that should cause a packet to be DROPPED must have its
  rules come before the NFQUEUE one, else the 'not DROP' decision from
  the helper will cause packets to be accepted that should not be.

  2. Any rules that would otherwise cause ACCEPT of a packet need to
  come after the NFQUEUE one, else they're going to negate the filtering
  it implements.

  3. Because once the NFQUEUE rule is matched no other rules will even
  be considered, I need to construct this rule so that it matches only
  packets that don't need to be considered by subsequent rules.

    In my case this would mean, e.g. only matching on specific port
  numbers and forgoing usually later rules that would affect such
  packets.

  Oh, also I found `--queue-bypass` to ensure it's not a "fail close" if
there's no userspace helper listening to the queue... but this literally
turns the rule into an ACCEPT, rather than passing evaluation to later
rules.  Presumably the 'fail open' socket option, to not cause all
packets to be dropped if the queue buffer is filled, has the same issue.
It's then blind ACCEPT rather than "let later rules look at the packet".

  Is any of my understanding in error?  Can I actually implement this
how I'd prefer, with later rules being evaulated upon some specific
verdict returned from userspace ?

P.S. The 'Mailing Lists' part of https://netfilter.org/ is out of date.
It still points to the old Majordomo pages on vger.kernel.org.
Attempting to subscribe there elicited a reply stating to use
https://subspace.kernel.org/vger.kernel.org.html instead.
-- 
- Athanasius (he/him) = Athanasius(at)miggy.org / https://miggy.org/
                  GPG/PGP Key: https://miggy.org/gpg-key
	   "And it's me who is my enemy. Me who beats me up.
Me who makes the monsters. Me who strips my confidence." Paula Cole - ME

Re: NFQUEUE usage and interaction with later chain rules

[Athanasius]

On Mon, Apr 15, 2024 at 01:09:30PM +0100, Athanasius wrote:
>   1. The only practical verdicts the userspace helper can return are
>   ACCEPT or DROP.  Checking through some documentation, examples, and
>   kernel code I've come across 'STOLEN', 'QUEUE' and 'REPEAT', but I
>   cannot find any actual documentation on what these might achieve.
> 
>   2. There appears to be no way to have an NFQUEUE rule act in a manner
>   where the userspace verdict can cause subsequent rules in the chain to
>   be considered.

  Well, of course I had an epiphany after sending this.  I can make use
of packet marks to do what I want, together with a REPEAT verdict.

  1. Have the NFQUEUE rule check that the specific mark value is NOT
  set.

  2. On not-DROP the userspace helper sets the mark value and returns a
  verdict of REPEAT.  REPEAT sends the packet through the netfilter
  chains and rules again, at least the INPUT chain.  So now it doesn't
  match the NFQUEUE rule and evaluation continues after.

I'm being a little vague about 'REPEAT' there because I've still not
found any documentation telling me what it's *supposed* to do.  I had
already tried 'STOP' (going through the 'other' possible verdicts), and
couldn't discern what it was causing to (not) happen.

  For the interested, I'm actually doing this in Python, using the
`fnfqueue` module.  The `NetfilterQueue` module appeared to offer no way
to use a more 'custom' verdict, or even set that 'fail open' socket
option.
  I'm using `impacket` to decode the packets in order to extract the
source IP.  The relevant code ends up being:

```python
def process_fnfqueue_packet(pkt):
    payload = pkt.payload
    rip = ImpactDecoder.IPDecoder().decode(payload)
    src_ip = rip.get_ip_src()
    if ipcheck.check_ip(src_ip):
        logger.debug(f"{src_ip=} should be banned")
        pkt.drop()

    else:
        pkt.mark = 2
        pkt.verdict(fnfqueue.REPEAT, fnfqueue.MANGLE_MARK)

```

`pkt` is the packet as `fnfqueue` provides it.

`ImpactDecoder...` is from `impacket`.

`ipcheck.check_ip()` is my code doing the GeoIP lookup and returning
a boolean, `True` for 'should drop'.

  I'll try to remember to write up a web page detailing my findings in
the hope that it might then be found useful by others in the future.

  Oh and that 'fail open' option is still a little problematic.  Without
it, because the NFQUEUE rule will initially match, I would just DROP
packets if the NFQUEUE buffer filled.  With it I'll instead end up with
blind ACCEPT rather than any subsequent rules being evaluated.
  I assume there are architectural reasons why evaluation couldn't be
made to continue after the NFQUEUE rule (e.g. a verdict of
'NF_CONTINUE'), but it would sure be helpful if that was possible.

-- 
- Athanasius (he/him) = Athanasius(at)miggy.org / https://miggy.org/
                  GPG/PGP Key: https://miggy.org/gpg-key
	   "And it's me who is my enemy. Me who beats me up.
Me who makes the monsters. Me who strips my confidence." Paula Cole - ME

Re: NFQUEUE usage and interaction with later chain rules

[Florian Westphal]

Athanasius <netfilter.org@miggy.org> wrote:
>   I'm in the middle of trying to implement some firewall bans based on
> GeoIP data.  It seemed that implementing an NFQUEUE userspace helper
> to do the lookups and decision making was the most feasible solution,

For IP matching?  Use ipset or xt_geoip from xtables addons, much
simpler and faster...

> but I have run into some issues.  I can probably work around them,
> but it does make this setup more fragile than I'd like.
> 
>   So, I'd like to ensure I'm properly understanding some things.
> 
>   1. The only practical verdicts the userspace helper can return are
>   ACCEPT or DROP.  Checking through some documentation, examples, and
>   kernel code I've come across 'STOLEN', 'QUEUE' and 'REPEAT', but I
>   cannot find any actual documentation on what these might achieve.

You can accept, drop, queue (pass packet to queue/program), or repeat,
which will make the packet reenter the same ruleset, i.e. its queued
again unless you use mark tricks to avoid that.

STOLEN cannot be used, its kernel internal.

>   2. There appears to be no way to have an NFQUEUE rule act in a manner
>   where the userspace verdict can cause subsequent rules in the chain to
>   be considered.

Yes.

>     I'd specifically like my userspace helper to be able to say 'DROP',
>   but otherwise allow rule evaluation to continue in the chain, rather
>   than blindly accepting all else.

Thats impossible to do.

>     If I could cause rule evaluation to jump to another chain that would
>   also be acceptable, simply placing rules that would normally be later
>   in the current chain into that one.

Same.

> there's no userspace helper listening to the queue... but this literally
> turns the rule into an ACCEPT, rather than passing evaluation to later
> rules.  Presumably the 'fail open' socket option, to not cause all
> packets to be dropped if the queue buffer is filled, has the same issue.
> It's then blind ACCEPT rather than "let later rules look at the packet".

Right.

>   Is any of my understanding in error?  Can I actually implement this
> how I'd prefer, with later rules being evaulated upon some specific
> verdict returned from userspace ?

No.  It would require significant rewrites on the kernel side, nfqueue
would have to be moved from core into xt_NFQUEUE and nft_queue,
respectively, with nftables/xtables specific changes to support what
you want, and such change would also break backwards compatibility.

I don't think this is going to happen.