nftables: bridge filter with queue to userspace

nftables: bridge filter with queue to userspace

[Martin Gröger]

I'm trying to build a transparent filter with application level 
filtering. First experiment with ip and output hook and queue to 
userspace was successful. Then I changed to bridge filtering with 
forward hook. With counter action I see that the packets match the rule, 
but the queue to the usersapce doesn't work.

Am I right, that this fuction should work?
I'm using Fedora 22 with nftables 0.4.

Re: nftables: bridge filter with queue to userspace

[Pablo Neira Ayuso]

On Thu, Oct 29, 2015 at 10:23:44PM +0100, Martin Gröger wrote:
> I'm trying to build a transparent filter with application level filtering.
> First experiment with ip and output hook and queue to userspace was
> successful. Then I changed to bridge filtering with forward hook. With
> counter action I see that the packets match the rule, but the queue to the
> usersapce doesn't work.
> 
> Am I right, that this fuction should work?

I guess you're using the 'bridge_netfilter' module?

Florian told me he will come up sooner or later with native queue
support for nft (ie. no bridge_netfilter required anymore).

> I'm using Fedora 22 with nftables 0.4.

Not related to this problem, but it's a good idea to stick to latest.
Lots of fixes and updates have happened between 0.4 and 0.5.

Re: nftables: bridge filter with queue to userspace

[Florian Westphal]

Pablo Neira Ayuso <pablo@netfilter.org> wrote:

[ CC Bernhard ]

> On Thu, Oct 29, 2015 at 10:23:44PM +0100, Martin Gröger wrote:
> > I'm trying to build a transparent filter with application level filtering.
> > First experiment with ip and output hook and queue to userspace was
> > successful. Then I changed to bridge filtering with forward hook. With
> > counter action I see that the packets match the rule, but the queue to the
> > usersapce doesn't work.
> > 
> > Am I right, that this fuction should work?

nfqueue backend only works with NFPROTO_IPV4 and _IPV6 at the moment,
i.e. nft ip and nft ip6, or via bridge_netfilter hack (which 'pushes'
packets though ipv4/ipv6 netfilter hooks).

> Florian told me he will come up sooner or later with native queue
> support for nft (ie. no bridge_netfilter required anymore).

Argh.  I'm a moron and forgot about this.

I still have the q&d hack that makes it work but no reroute (re-bridge,
cough) support, just dump-to-userspace.

Bernhard, did you have time to work on this?

If not, I think I can make time available soon since the other work
(nft bridge conntrack, nf netns hook stuff) is delayed at the moment
anyways.

Re: nftables: bridge filter with queue to userspace

[Martin Gröger]

Am 29.10.2015 23:23, schrieb Florian Westphal:
> Pablo Neira Ayuso <pablo@netfilter.org> wrote:
>
> [ CC Bernhard ]
>
>> On Thu, Oct 29, 2015 at 10:23:44PM +0100, Martin Gröger wrote:
>>> I'm trying to build a transparent filter with application level filtering.
>>> First experiment with ip and output hook and queue to userspace was
>>> successful. Then I changed to bridge filtering with forward hook. With
>>> counter action I see that the packets match the rule, but the queue to the
>>> usersapce doesn't work.
>>>
>>> Am I right, that this fuction should work?
> nfqueue backend only works with NFPROTO_IPV4 and _IPV6 at the moment,
> i.e. nft ip and nft ip6, or via bridge_netfilter hack (which 'pushes'
> packets though ipv4/ipv6 netfilter hooks).
Ok, this explains why my experiment with ip work, but not the one with 
bridge family.

>
>> Florian told me he will come up sooner or later with native queue
>> support for nft (ie. no bridge_netfilter required anymore).
> Argh.  I'm a moron and forgot about this.
How does a native (indepent on bridge) queue support should work ?
>
> I still have the q&d hack that makes it work but no reroute (re-bridge,
> cough) support, just dump-to-userspace.
As far as I understand this would be sufficient for my usecase, since I 
want simply to inspect the packets and then decide to accept or drop them.

>
> Bernhard, did you have time to work on this?
>
> If not, I think I can make time available soon since the other work
> (nft bridge conntrack, nf netns hook stuff) is delayed at the moment
> anyways.
>
Would be great, if you can help me!

Re: nftables: bridge filter with queue to userspace

[Florian Westphal]

Martin Gröger <mgroeger1@web.de> wrote:
> >>Florian told me he will come up sooner or later with native queue
> >>support for nft (ie. no bridge_netfilter required anymore).
> >Argh.  I'm a moron and forgot about this.
> How does a native (indepent on bridge) queue support should work ?

Highlevel?
nft add bridge filter forward queue

But on the backend/userspace side its a good question.

For instance, where should NFQA_PAYLOAD point to?

Start of ethernet header?
Start of network header?

If the former, what should NFQA_HWADDR contain?
Should it even be present?

What about ->hw_protocol in nfqnl_msg_packet_hdr ?

We also should definitely present VLAN headers to userspace
in some way.  Standard question is if that should be extra attribute
or if we should inject the vlan header into the packet data.

I'm leaning towards:
- NFQA_PAYLOAD starts at beginning of ethernet header
- nfqnl_msg_packet_hdr->hw_protocol is set to skb->protocol OR
  skb->vlan_proto in offload case (even though this information is
  redundant since its also present in the ethernet header ...)
- NFQA_PAYLOAD maintains illusion of disabled/non-existant VLAN hw
offload, i.e. we insert it into NFQA_PAYLOAD between mac and network
header.
- NFQA_HWADDR attribute is not present (redundant, we have this in
  NFQA_PAYLOAD).

> >I still have the q&d hack that makes it work but no reroute (re-bridge,
> >cough) support, just dump-to-userspace.
> As far as I understand this would be sufficient for my usecase,
> since I want simply to inspect the packets and then decide to accept
> or drop them.

Yes, in fact I think we should just ignore reroute (bad idea) or rebridge
(what would be the use case of this...?)

If someone really needs to be able to resend/relay packet they could do
this in userspace or just use PRE_ROUTING since thats before bridge
asks the FDB for the output port.

We could add bridge_me_harder to pick another output device for queueing
in BRIDGE OUTPUT but I have no idea why one would want such feature.

Re: nftables: bridge filter with queue to userspace

[Martin Gröger]

Am 30.10.2015 14:38, schrieb Florian Westphal:
> Martin Gröger <mgroeger1@web.de> wrote:
>>>> Florian told me he will come up sooner or later with native queue
>>>> support for nft (ie. no bridge_netfilter required anymore).
>>> Argh.  I'm a moron and forgot about this.
>> How does a native (indepent on bridge) queue support should work ?
> Highlevel?
> nft add bridge filter forward queue
That's exactly what I expected.
> But on the backend/userspace side its a good question.
>
> For instance, where should NFQA_PAYLOAD point to?
>
> Start of ethernet header?
> Start of network header?
However, the full information contained in the paket (incl. VLAN) should 
be given to the user.
So it wouold make sense to point to the ethernet header with NFQA_PAYLOAD.
>
> If the former, what should NFQA_HWADDR contain?
> Should it even be present?
>
> What about ->hw_protocol in nfqnl_msg_packet_hdr ?
>
> We also should definitely present VLAN headers to userspace
> in some way.  Standard question is if that should be extra attribute
> or if we should inject the vlan header into the packet data.
>
> I'm leaning towards:
> - NFQA_PAYLOAD starts at beginning of ethernet header
ok
> - nfqnl_msg_packet_hdr->hw_protocol is set to skb->protocol OR
>    skb->vlan_proto in offload case (even though this information is
>    redundant since its also present in the ethernet header ...)
> - NFQA_PAYLOAD maintains illusion of disabled/non-existant VLAN hw
> offload, i.e. we insert it into NFQA_PAYLOAD between mac and network
> header.
Sorry, I don't understand this. The VLAN header is (if exists) after the 
source MAC. If a paket is bridged, I would expect, that the VLAN header 
is kept unchanged. I would expect to find the VLAN header exactly there! 
Is this wrong?
> - NFQA_HWADDR attribute is not present (redundant, we have this in
>    NFQA_PAYLOAD).
>>> I still have the q&d hack that makes it work but no reroute (re-bridge,
>>> cough) support, just dump-to-userspace.
>> As far as I understand this would be sufficient for my usecase,
>> since I want simply to inspect the packets and then decide to accept
>> or drop them.
> Yes, in fact I think we should just ignore reroute (bad idea) or rebridge
> (what would be the use case of this...?)
>
> If someone really needs to be able to resend/relay packet they could do
> this in userspace or just use PRE_ROUTING since thats before bridge
> asks the FDB for the output port.
>
> We could add bridge_me_harder to pick another output device for queueing
> in BRIDGE OUTPUT but I have no idea why one would want such feature.
>
I'm currently trying to understand the structure.
So to add the missing parts:
- there is no change in the nftable kernel modules necessary?
- there are changes in the nftnl library necessary?
- there are changes in nft necessary?
Is this correct?

I didn't find much documentation conecrning the architecture.
Only Pablo_nftables-osd-userday-2013.pdf and 
Nftables-osd-2013-developer.pdf.
Is there somethimg more to read to get an better understanding of nftables?

Thanks,
Martin

Re: nftables: bridge filter with queue to userspace

[Florian Westphal]

Martin Gröger <mgroeger1@web.de> wrote:
> >- NFQA_PAYLOAD maintains illusion of disabled/non-existant VLAN hw
> >offload, i.e. we insert it into NFQA_PAYLOAD between mac and network
> >header.
> Sorry, I don't understand this. The VLAN header is (if exists) after
> the source MAC. If a paket is bridged, I would expect, that the VLAN
> header is kept unchanged. I would expect to find the VLAN header
> exactly there! Is this wrong?

Yes, there is no VLAN header, its removed (usually by hardware offloads)
and stored in meta data only.

Thats why I think we should transparently re-insert when sending
the copy to userspace.

(I.e. undo what hardware offloads did).

> >- NFQA_HWADDR attribute is not present (redundant, we have this in
> >   NFQA_PAYLOAD).
> >>>I still have the q&d hack that makes it work but no reroute (re-bridge,
> >>>cough) support, just dump-to-userspace.
> >>As far as I understand this would be sufficient for my usecase,
> >>since I want simply to inspect the packets and then decide to accept
> >>or drop them.
> >Yes, in fact I think we should just ignore reroute (bad idea) or rebridge
> >(what would be the use case of this...?)
> >
> >If someone really needs to be able to resend/relay packet they could do
> >this in userspace or just use PRE_ROUTING since thats before bridge
> >asks the FDB for the output port.
> >
> >We could add bridge_me_harder to pick another output device for queueing
> >in BRIDGE OUTPUT but I have no idea why one would want such feature.
> >
> I'm currently trying to understand the structure.
> So to add the missing parts:
> - there is no change in the nftable kernel modules necessary?

Not to nftables but to nfnetlink_queue module (and to bridge netfilter
kernel part).

> - there are changes in the nftnl library necessary?

No, unless we add extra attribute e.g. for vlan header but I'd like
to avoid it.

> - there are changes in nft necessary?

No, nft side should already work just fine.

> Only Pablo_nftables-osd-userday-2013.pdf and
> Nftables-osd-2013-developer.pdf.
> Is there somethimg more to read to get an better understanding of nftables?

Sorry, not that I know of.  Perhaps Patrick or Pablo have more
information available somewhere.

Re: nftables: bridge filter with queue to userspace

[Martin Gröger]

Am 30.10.2015 22:27, schrieb Florian Westphal:
> Martin Gröger <mgroeger1@web.de> wrote:
>>> - NFQA_PAYLOAD maintains illusion of disabled/non-existant VLAN hw
>>> offload, i.e. we insert it into NFQA_PAYLOAD between mac and network
>>> header.
>> Sorry, I don't understand this. The VLAN header is (if exists) after
>> the source MAC. If a paket is bridged, I would expect, that the VLAN
>> header is kept unchanged. I would expect to find the VLAN header
>> exactly there! Is this wrong?
> Yes, there is no VLAN header, its removed (usually by hardware offloads)
> and stored in meta data only.
>
> Thats why I think we should transparently re-insert when sending
> the copy to userspace.
>
> (I.e. undo what hardware offloads did).

Ok, now I understand.

So there are basically 2 options:
1.) reinsert the VLAN header
     pro: looks like the wire
     con: Is the insert measurable in terms of performance?
2.) pass der VLAN header also as meta data to the user.
     pro: network header is always on the same offset, so it might be 
easier for higher layer inspection
     con: not like the wire

So, although I would have expected option 1, I think option 2 has 
advantages too. So I think both shoud be ok.
>
>>> - NFQA_HWADDR attribute is not present (redundant, we have this in
>>>    NFQA_PAYLOAD).
>>>>> I still have the q&d hack that makes it work but no reroute (re-bridge,
>>>>> cough) support, just dump-to-userspace.
>>>> As far as I understand this would be sufficient for my usecase,
>>>> since I want simply to inspect the packets and then decide to accept
>>>> or drop them.
>>> Yes, in fact I think we should just ignore reroute (bad idea) or rebridge
>>> (what would be the use case of this...?)
>>>
>>> If someone really needs to be able to resend/relay packet they could do
>>> this in userspace or just use PRE_ROUTING since thats before bridge
>>> asks the FDB for the output port.
>>>
>>> We could add bridge_me_harder to pick another output device for queueing
>>> in BRIDGE OUTPUT but I have no idea why one would want such feature.
>>>
>> I'm currently trying to understand the structure.
>> So to add the missing parts:
>> - there is no change in the nftable kernel modules necessary?
> Not to nftables but to nfnetlink_queue module (and to bridge netfilter
> kernel part).
 From the presentation Nftables-osd-2013-developer.pdf I understood, 
that there is a generic instruction set part of the kernel. New 
functionality will be added by new combination of this instruction set. 
Additionally there must be some code for each hook, but I would expext, 
that this code provides the packet and some metadata to the engine, 
which process the instructions. Also for the interface from the main 
engine to the nfnetlink_queue module I would expect the packet and a 
metadata structure. Since the queueing for the IP hook works, I expected 
that all generic instructions for this purpose are available and just 
have to be used for the bridge hook.

So obviously somehow my expectation is wrong - but what's wrong?

>
>> - there are changes in the nftnl library necessary?
> No, unless we add extra attribute e.g. for vlan header but I'd like
>   avoid it.
Ok, is this again an argument for Option 1 above?
Is this, because there is currently in the generic instructions no 
option to add VLAN metadata somehow to the nfnetlink_queue?
>
>> - there are changes in nft necessary?
> No, nft side should already work just fine.
>
>> Only Pablo_nftables-osd-userday-2013.pdf and
>> Nftables-osd-2013-developer.pdf.
>> Is there somethimg more to read to get an better understanding of nftables?
> Sorry, not that I know of.  Perhaps Patrick or Pablo have more
> information available somewhere.
>