Re: [PATCH v2 2/2] x86/asm/bitops: __ffs,ffz: use __builtin_ctzl to evaluate constant expressions

[Vincent MAILHOL <mailhol.vincent@wanadoo.fr>]

On Thu. 12 May 2022 at 07:20, Nick Desaulniers <ndesaulniers@google.com> wrote:
> On Wed, May 11, 2022 at 9:04 AM Vincent Mailhol
> <mailhol.vincent@wanadoo.fr> wrote:
> >
> > __ffs(x) is equivalent to (unsigned long)__builtin_ctzl(x) and ffz(x)
> > is equivalent to (unsigned long)__builtin_ctzl(~x). Because
> > __builting_ctzl() returns an int, a cast to (unsigned long) is
> > necessary to avoid potential warnings on implicit casts.
> >
> > For x86_64, the current __ffs() and ffz() implementations do not
> > produce optimized code when called with a constant expression. On the
> > contrary, the __builtin_ctzl() gets simplified into a single
> > instruction.
> >
> > However, for non constant expressions, the __ffs() and ffz() asm
> > versions of the kernel remains slightly better than the code produced
> > by GCC (it produces a useless instruction to clear eax).
> >
> > This patch uses the __builtin_constant_p() to select between the
> > kernel's __ffs()/ffz() and the __builtin_ctzl() depending on whether
> > the argument is constant or not.
> >
> > ** Statistics **
> >
> > On a allyesconfig, before applying this patch...:
> >
> > | $ objdump -d vmlinux.o | grep tzcnt | wc -l
> > | 3607
> >
> > ...and after:
> >
> > | $ objdump -d vmlinux.o | grep tzcnt | wc -l
> > | 2600
> >
> > So, roughly 27.9% of the call to either __ffs() or ffz() were using
> > constant expression and were optimized out.
> >
> > (tests done on linux v5.18-rc5 x86_64 using GCC 11.2.1)
> >
> > Note: on x86_64, the asm bsf instruction produces tzcnt when used with
> > the ret prefix (which is why we grep tzcnt instead of bsf in above
> > benchmark). c.f. [1]
> >
> > [1] commit e26a44a2d618 ("x86: Use REP BSF unconditionally")
> > http://lkml.kernel.org/r/5058741E020000780009C014@nat28.tlf.novell.com
> >
> > CC: Nick Desaulniers <ndesaulniers@google.com>
> > Signed-off-by: Vincent Mailhol <mailhol.vincent@wanadoo.fr>
>
> Patch LGTM, though I find the location of the double unscores in the
> names slightly against my taste.
>
> > ---
> >  arch/x86/include/asm/bitops.h | 38 ++++++++++++++++++++++-------------
> >  1 file changed, 24 insertions(+), 14 deletions(-)
> >
> > diff --git a/arch/x86/include/asm/bitops.h b/arch/x86/include/asm/bitops.h
> > index 6ed979547086..7cf5374ce403 100644
> > --- a/arch/x86/include/asm/bitops.h
> > +++ b/arch/x86/include/asm/bitops.h
> > @@ -224,13 +224,7 @@ static __always_inline bool variable_test_bit(long nr, volatile const unsigned l
> >          ? constant_test_bit((nr), (addr))      \
> >          : variable_test_bit((nr), (addr)))
> >
> > -/**
> > - * __ffs - find first set bit in word
> > - * @word: The word to search
> > - *
> > - * Undefined if no bit exists, so code should check against 0 first.
> > - */
> > -static __always_inline unsigned long __ffs(unsigned long word)
> > +static __always_inline unsigned long __variable_ffs(unsigned long word)
>
> How about `variable___ffs`? Patch 1/2 used `variable_ffs` for `ffs`?

On a first glance, having the triple underscores in the middle of the
name seemed odd. On second thought, it is more consistent with the
rest, and I finally like the idea. Will be adopted in v3.

> >  {
> >         asm("rep; bsf %1,%0"
> >                 : "=r" (word)
> > @@ -238,13 +232,18 @@ static __always_inline unsigned long __ffs(unsigned long word)
> >         return word;
> >  }
> >
> > -/**
> > - * ffz - find first zero bit in word
> > - * @word: The word to search
> > - *
> > - * Undefined if no zero exists, so code should check against ~0UL first.
> > - */
> > -static __always_inline unsigned long ffz(unsigned long word)
> > +/**
> > + * __ffs - find first set bit in word
> > + * @word: The word to search
> > + *
> > + * Undefined if no bit exists, so code should check against 0 first.
> > + */
> > +#define __ffs(word)                            \
> > +       (__builtin_constant_p(word) ?           \
> > +        (unsigned long)__builtin_ctzl(word) :  \
> > +        __variable_ffs(word))
> > +
> > +static __always_inline unsigned long __variable_ffz(unsigned long word)
>
> `ffz` had no underscore. Regardless of `__ffs`, this should definitely
> be `variable_ffz` IMO.

ACK.

> >  {
> >         asm("rep; bsf %1,%0"
> >                 : "=r" (word)
> > @@ -252,6 +251,17 @@ static __always_inline unsigned long ffz(unsigned long word)
> >         return word;
> >  }
> >
> > +/**
> > + * ffz - find first zero bit in word
> > + * @word: The word to search
> > + *
> > + * Undefined if no zero exists, so code should check against ~0UL first.
> > + */
> > +#define ffz(word)                              \
> > +       (__builtin_constant_p(word) ?           \
> > +        (unsigned long)__builtin_ctzl(~word) : \
> > +        __variable_ffz(word))
> > +
> >  /*
> >   * __fls: find last set bit in word
> >   * @word: The word to search
> > --
> > 2.35.1
> >
>
>
> --
> Thanks,
> ~Nick Desaulniers