ebtables-nft can't delete complex rules by specifying complete rule with kernel 6.3+

ebtables-nft can't delete complex rules by specifying complete rule with kernel 6.3+

[Igor Raits]

Hello,

We started to observe the issue regarding ebtables-nft and how it
can't wipe rules when specifying full rule. Removing the rule by index
works fine, though. Also with kernel 6.1.y it works completely fine.

I've started with 1.8.8 provided in CentOS Stream 9, then tried the
latest git version and all behave exactly the same. See the behavior
below. As you can see, simple DROP works, but more complex one do not.

As bugzilla requires some special sign-up procedure, apologize for
reporting it directly here in the ML.

# ebtables -L
Bridge table: filter

Bridge chain: INPUT, entries: 0, policy: ACCEPT

Bridge chain: FORWARD, entries: 0, policy: ACCEPT

Bridge chain: OUTPUT, entries: 0, policy: ACCEPT
# ebtables -t nat -N barani
# ebtables -t nat -A barani -i br_public --among-src fa:16:3e:26:bf:42 -j RETURN
# ebtables -t nat -D barani -i br_public --among-src fa:16:3e:26:bf:42 -j RETURN
ebtables v1.8.9 (nf_tables):  RULE_DELETE failed (Invalid argument):
rule in chain barani
# ebtables -t nat -A barani -j DROP
# ebtables -t nat -D barani -j DROP
# ebtables -t nat -L barani
Bridge table: nat

Bridge chain: barani, entries: 1, policy: RETURN
-i br_public --among-src fa:16:3e:26:bf:42 -j RETURN
# ebtables -t nat -D barani 1
# ebtables -t nat -L barani
Bridge table: nat

Bridge chain: barani, entries: 0, policy: RETURN

Re: ebtables-nft can't delete complex rules by specifying complete rule with kernel 6.3+

[Florian Westphal]

Igor Raits <igor@gooddata.com> wrote:
> Hello,
> 
> We started to observe the issue regarding ebtables-nft and how it
> can't wipe rules when specifying full rule. Removing the rule by index
> works fine, though. Also with kernel 6.1.y it works completely fine.
> 
> I've started with 1.8.8 provided in CentOS Stream 9, then tried the
> latest git version and all behave exactly the same. See the behavior
> below. As you can see, simple DROP works, but more complex one do not.
> 
> As bugzilla requires some special sign-up procedure, apologize for
> reporting it directly here in the ML.

Thanks for the report, I'll look into it later today.

Re: ebtables-nft can't delete complex rules by specifying complete rule with kernel 6.3+

[Florian Westphal]

Florian Westphal <fw@strlen.de> wrote:
> Igor Raits <igor@gooddata.com> wrote:
> > Hello,
> > 
> > We started to observe the issue regarding ebtables-nft and how it
> > can't wipe rules when specifying full rule. Removing the rule by index
> > works fine, though. Also with kernel 6.1.y it works completely fine.
> > 
> > I've started with 1.8.8 provided in CentOS Stream 9, then tried the
> > latest git version and all behave exactly the same. See the behavior
> > below. As you can see, simple DROP works, but more complex one do not.
> > 
> > As bugzilla requires some special sign-up procedure, apologize for
> > reporting it directly here in the ML.
> 
> Thanks for the report, I'll look into it later today.

Its a bug in ebtables-nft, it fails to delete the rule since

938154b93be8cd611ddfd7bafc1849f3c4355201,
netfilter: nf_tables: reject unbound anonymous set before commit phase

But its possible do remove the rule via
nft delete rule .. handle $x

so the breakge is limited to ebtables-nft.

Re: ebtables-nft can't delete complex rules by specifying complete rule with kernel 6.3+

[Igor Raits]

Hello Florian,

On Mon, Jul 10, 2023 at 2:49 PM Florian Westphal <fw@strlen.de> wrote:
>
> Florian Westphal <fw@strlen.de> wrote:
> > Igor Raits <igor@gooddata.com> wrote:
> > > Hello,
> > >
> > > We started to observe the issue regarding ebtables-nft and how it
> > > can't wipe rules when specifying full rule. Removing the rule by index
> > > works fine, though. Also with kernel 6.1.y it works completely fine.
> > >
> > > I've started with 1.8.8 provided in CentOS Stream 9, then tried the
> > > latest git version and all behave exactly the same. See the behavior
> > > below. As you can see, simple DROP works, but more complex one do not.
> > >
> > > As bugzilla requires some special sign-up procedure, apologize for
> > > reporting it directly here in the ML.
> >
> > Thanks for the report, I'll look into it later today.
>
> Its a bug in ebtables-nft, it fails to delete the rule since
>
> 938154b93be8cd611ddfd7bafc1849f3c4355201,
> netfilter: nf_tables: reject unbound anonymous set before commit phase
>
> But its possible do remove the rule via
> nft delete rule .. handle $x
>
> so the breakge is limited to ebtables-nft.

Thanks for confirmation and additional information regarding where
exactly the issue was introduced.
The ebtables-nft (well, ebtables in general) is heavily used by the
OpenStack Neutron (in linuxbridge mode), so this breaks our setup
quite a bit. Would you recommend to revert kernel change or would you
have the actual fix soon (ebtables-nft or kernel)?

Re: ebtables-nft can't delete complex rules by specifying complete rule with kernel 6.3+

[Pablo Neira Ayuso]

Hi,

On Mon, Jul 10, 2023 at 04:41:27PM +0200, Igor Raits wrote:
> Hello Florian,
> 
> On Mon, Jul 10, 2023 at 2:49 PM Florian Westphal <fw@strlen.de> wrote:
> >
> > Florian Westphal <fw@strlen.de> wrote:
> > > Igor Raits <igor@gooddata.com> wrote:
> > > > Hello,
> > > >
> > > > We started to observe the issue regarding ebtables-nft and how it
> > > > can't wipe rules when specifying full rule. Removing the rule by index
> > > > works fine, though. Also with kernel 6.1.y it works completely fine.
> > > >
> > > > I've started with 1.8.8 provided in CentOS Stream 9, then tried the
> > > > latest git version and all behave exactly the same. See the behavior
> > > > below. As you can see, simple DROP works, but more complex one do not.
> > > >
> > > > As bugzilla requires some special sign-up procedure, apologize for
> > > > reporting it directly here in the ML.
> > >
> > > Thanks for the report, I'll look into it later today.
> >
> > Its a bug in ebtables-nft, it fails to delete the rule since
> >
> > 938154b93be8cd611ddfd7bafc1849f3c4355201,
> > netfilter: nf_tables: reject unbound anonymous set before commit phase
> >
> > But its possible do remove the rule via
> > nft delete rule .. handle $x
> >
> > so the breakge is limited to ebtables-nft.
> 
> Thanks for confirmation and additional information regarding where
> exactly the issue was introduced.
> The ebtables-nft (well, ebtables in general) is heavily used by the
> OpenStack Neutron (in linuxbridge mode), so this breaks our setup
> quite a bit. Would you recommend to revert kernel change or would you
> have the actual fix soon (ebtables-nft or kernel)?

Just to make sure this bug is not caused by something else.

Could you cherry-pick this kernel patch? It is currently missing 6.1.38:

commit 3e70489721b6c870252c9082c496703677240f53
Author: Pablo Neira Ayuso <pablo@netfilter.org>
Date:   Mon Jun 26 00:42:18 2023 +0200

    netfilter: nf_tables: unbind non-anonymous set if rule construction fails
    
    Otherwise a dangling reference to a rule object that is gone remains
    in the set binding list.

I have requested included to -stable already.

Thanks.

Re: ebtables-nft can't delete complex rules by specifying complete rule with kernel 6.3+

[Pablo Neira Ayuso]

On Mon, Jul 10, 2023 at 08:03:04PM +0200, Pablo Neira Ayuso wrote:
> Hi,
> 
> On Mon, Jul 10, 2023 at 04:41:27PM +0200, Igor Raits wrote:
> > Hello Florian,
> > 
> > On Mon, Jul 10, 2023 at 2:49 PM Florian Westphal <fw@strlen.de> wrote:
> > >
> > > Florian Westphal <fw@strlen.de> wrote:
> > > > Igor Raits <igor@gooddata.com> wrote:
> > > > > Hello,
> > > > >
> > > > > We started to observe the issue regarding ebtables-nft and how it
> > > > > can't wipe rules when specifying full rule. Removing the rule by index
> > > > > works fine, though. Also with kernel 6.1.y it works completely fine.
> > > > >
> > > > > I've started with 1.8.8 provided in CentOS Stream 9, then tried the
> > > > > latest git version and all behave exactly the same. See the behavior
> > > > > below. As you can see, simple DROP works, but more complex one do not.
> > > > >
> > > > > As bugzilla requires some special sign-up procedure, apologize for
> > > > > reporting it directly here in the ML.
> > > >
> > > > Thanks for the report, I'll look into it later today.
> > >
> > > Its a bug in ebtables-nft, it fails to delete the rule since
> > >
> > > 938154b93be8cd611ddfd7bafc1849f3c4355201,
> > > netfilter: nf_tables: reject unbound anonymous set before commit phase
> > >
> > > But its possible do remove the rule via
> > > nft delete rule .. handle $x
> > >
> > > so the breakge is limited to ebtables-nft.
> > 
> > Thanks for confirmation and additional information regarding where
> > exactly the issue was introduced.
> > The ebtables-nft (well, ebtables in general) is heavily used by the
> > OpenStack Neutron (in linuxbridge mode), so this breaks our setup
> > quite a bit. Would you recommend to revert kernel change or would you
> > have the actual fix soon (ebtables-nft or kernel)?
> 
> Just to make sure this bug is not caused by something else.
> 
> Could you cherry-pick this kernel patch? It is currently missing 6.1.38:
> 
> commit 3e70489721b6c870252c9082c496703677240f53
> Author: Pablo Neira Ayuso <pablo@netfilter.org>
> Date:   Mon Jun 26 00:42:18 2023 +0200
> 
>     netfilter: nf_tables: unbind non-anonymous set if rule construction fails
>     
>     Otherwise a dangling reference to a rule object that is gone remains
>     in the set binding list.
> 
> I have requested included to -stable already.

Oh wait, you mentioned this works fine for you with 6.1.x.

Re: ebtables-nft can't delete complex rules by specifying complete rule with kernel 6.3+

[Igor Raits]

Hello,

On Mon, Jul 10, 2023 at 8:05 PM Pablo Neira Ayuso <pablo@netfilter.org> wrote:
>
> On Mon, Jul 10, 2023 at 08:03:04PM +0200, Pablo Neira Ayuso wrote:
> > Hi,
> >
> > On Mon, Jul 10, 2023 at 04:41:27PM +0200, Igor Raits wrote:
> > > Hello Florian,
> > >
> > > On Mon, Jul 10, 2023 at 2:49 PM Florian Westphal <fw@strlen.de> wrote:
> > > >
> > > > Florian Westphal <fw@strlen.de> wrote:
> > > > > Igor Raits <igor@gooddata.com> wrote:
> > > > > > Hello,
> > > > > >
> > > > > > We started to observe the issue regarding ebtables-nft and how it
> > > > > > can't wipe rules when specifying full rule. Removing the rule by index
> > > > > > works fine, though. Also with kernel 6.1.y it works completely fine.
> > > > > >
> > > > > > I've started with 1.8.8 provided in CentOS Stream 9, then tried the
> > > > > > latest git version and all behave exactly the same. See the behavior
> > > > > > below. As you can see, simple DROP works, but more complex one do not.
> > > > > >
> > > > > > As bugzilla requires some special sign-up procedure, apologize for
> > > > > > reporting it directly here in the ML.
> > > > >
> > > > > Thanks for the report, I'll look into it later today.
> > > >
> > > > Its a bug in ebtables-nft, it fails to delete the rule since
> > > >
> > > > 938154b93be8cd611ddfd7bafc1849f3c4355201,
> > > > netfilter: nf_tables: reject unbound anonymous set before commit phase
> > > >
> > > > But its possible do remove the rule via
> > > > nft delete rule .. handle $x
> > > >
> > > > so the breakge is limited to ebtables-nft.
> > >
> > > Thanks for confirmation and additional information regarding where
> > > exactly the issue was introduced.
> > > The ebtables-nft (well, ebtables in general) is heavily used by the
> > > OpenStack Neutron (in linuxbridge mode), so this breaks our setup
> > > quite a bit. Would you recommend to revert kernel change or would you
> > > have the actual fix soon (ebtables-nft or kernel)?
> >
> > Just to make sure this bug is not caused by something else.
> >
> > Could you cherry-pick this kernel patch? It is currently missing 6.1.38:
> >
> > commit 3e70489721b6c870252c9082c496703677240f53
> > Author: Pablo Neira Ayuso <pablo@netfilter.org>
> > Date:   Mon Jun 26 00:42:18 2023 +0200
> >
> >     netfilter: nf_tables: unbind non-anonymous set if rule construction fails
> >
> >     Otherwise a dangling reference to a rule object that is gone remains
> >     in the set binding list.
> >
> > I have requested included to -stable already.
>
> Oh wait, you mentioned this works fine for you with 6.1.x.

I should have mentioned that we've tested 6.1.32 where it worked and
my colleague has tested 6.1.38 today where it is broken.

Re: ebtables-nft can't delete complex rules by specifying complete rule with kernel 6.3+

[Florian Westphal]

Pablo Neira Ayuso <pablo@netfilter.org> wrote:
> On Mon, Jul 10, 2023 at 04:41:27PM +0200, Igor Raits wrote:
> > Hello Florian,
> > 
> > On Mon, Jul 10, 2023 at 2:49 PM Florian Westphal <fw@strlen.de> wrote:
> > >
> > > Florian Westphal <fw@strlen.de> wrote:
> > > > Igor Raits <igor@gooddata.com> wrote:
> > > > > Hello,
> > > > >
> > > > > We started to observe the issue regarding ebtables-nft and how it
> > > > > can't wipe rules when specifying full rule. Removing the rule by index
> > > > > works fine, though. Also with kernel 6.1.y it works completely fine.
> > > > >
> > > > > I've started with 1.8.8 provided in CentOS Stream 9, then tried the
> > > > > latest git version and all behave exactly the same. See the behavior
> > > > > below. As you can see, simple DROP works, but more complex one do not.
> > > > >
> > > > > As bugzilla requires some special sign-up procedure, apologize for
> > > > > reporting it directly here in the ML.
> > > >
> > > > Thanks for the report, I'll look into it later today.
> > >
> > > Its a bug in ebtables-nft, it fails to delete the rule since
> > >
> > > 938154b93be8cd611ddfd7bafc1849f3c4355201,
> > > netfilter: nf_tables: reject unbound anonymous set before commit phase
> > >
> > > But its possible do remove the rule via
> > > nft delete rule .. handle $x
> > >
> > > so the breakge is limited to ebtables-nft.
> > 
> > Thanks for confirmation and additional information regarding where
> > exactly the issue was introduced.
> > The ebtables-nft (well, ebtables in general) is heavily used by the
> > OpenStack Neutron (in linuxbridge mode), so this breaks our setup
> > quite a bit. Would you recommend to revert kernel change or would you
> > have the actual fix soon (ebtables-nft or kernel)?
> 
> Just to make sure this bug is not caused by something else.

No no no, this is a userspace bug.
netfilter: nf_tables: reject unbound anonymous set before commit phase

ebtables-nft emits a DELRULE followed by creation of a (dangling!)
anon set, because backend code that handles add/delete is identical,
so 'delete' request for among schedules addition of the set.