[PATCH v3] ld.so.8: Describe glibc Hardware capabilities

[PATCH v3] ld.so.8: Describe glibc Hardware capabilities

[Adhemerval Zanella]

It was added on glibc 2.33 as a way to improve path search, since
legacy hardware capabilities combination scheme do not scale
properly with new hardware support.  The legacy support was removed
on glibc 2.37, so it is the only scheme currently supported.

Signed-off-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
---
 man8/ld.so.8 | 48 +++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 47 insertions(+), 1 deletion(-)

diff --git a/man8/ld.so.8 b/man8/ld.so.8
index cf03cb85e..5b02ae88f 100644
--- a/man8/ld.so.8
+++ b/man8/ld.so.8
@@ -208,6 +208,14 @@ The objects in
 .I list
 are delimited by colons.
 .TP
+.BI \-\-glibc-hwcaps-mask " list"
+only search built-in subdirectories if in
+.IR list .
+.TP
+.BI \-\-glibc-hwcaps-prepend " list"
+Search glibc-hwcaps subdirectories in
+.IR list .
+.TP
 .B \-\-inhibit\-cache
 Do not use
 .IR /etc/ld.so.cache .
@@ -808,7 +816,7 @@ as a temporary workaround to a library misconfiguration issue.)
 .I lib*.so*
 shared objects
 .SH NOTES
-.SS Hardware capabilities
+.SS Legacy Hardware capabilities (from glibc 2.5 to glibc 2.37)
 Some shared objects are compiled using hardware-specific instructions which do
 not exist on every CPU.
 Such objects should be installed in directories whose names define the
@@ -843,6 +851,44 @@ z900, z990, z9-109, z10, zarch
 .B x86 (32-bit only)
 acpi, apic, clflush, cmov, cx8, dts, fxsr, ht, i386, i486, i586, i686, mca, mmx,
 mtrr, pat, pbe, pge, pn, pse36, sep, ss, sse, sse2, tm
+.SS glibc Hardware capabilities (from glibc 2.33)
+The legacy hardware capabilities combinations has the drawback where
+each feature name incur in
+cascading extra paths added on the search path list,
+adding a lot of overhead on
+.B ld.so
+during library resolution.
+For instance, on x86 32-bit, if the hardware
+supports
+.B i686
+and
+.BR sse2
+, the resulting search path will be
+.BR i686/sse2:i686:sse2:. .
+A new capability
+.B newcap
+will set the search path to
+.BR newcap/i686/sse2:newcap/i686:newcap/sse2:newcap:i686/sse2:i686:sse2: .
+.PP
+glibc 2.33 added a new hardware capability scheme,
+where each ABI can define
+a set of paths based on expected hardware support.
+Each path is added depending of the hardware of the machine,
+and they are not combined together.
+They also have priority over the legacy hardware capabilities.
+The following paths are currently supported.
+.TP
+.B PowerPC (64-bit little-endian only)
+power9, power10
+.TP
+.B s390 (64-bit only)
+z13, z14, z15, z16
+.TP
+.B x86 (64-bit only)
+x86-64-v2, x86-64-v3, x86-64-v4
+.PP
+The glibc 2.37 removed support for the legacy hardware capabilities.
+.
 .SH SEE ALSO
 .BR ld (1),
 .BR ldd (1),
-- 
2.34.1

Re: [PATCH v3] ld.so.8: Describe glibc Hardware capabilities

[Stefan Puiu]

Hi and sorry to jump in so late in the discussion,

On Thu, Oct 19, 2023 at 8:23 PM Adhemerval Zanella
<adhemerval.zanella@linaro.org> wrote:
>
> It was added on glibc 2.33 as a way to improve path search, since
> legacy hardware capabilities combination scheme do not scale
> properly with new hardware support.  The legacy support was removed
> on glibc 2.37, so it is the only scheme currently supported.

I would suggest (caveat: non-native English speaker here):

s/It was added on glibc/The feature was added in glibc/
s/improve path search/improve the path search/
s/since legacy/since the legacy/
s/hardware capabilities combination scheme do not/hardware
capabilities scheme does not/
s/was removed on glibc/was removed in glibc/

>
> Signed-off-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
> ---
>  man8/ld.so.8 | 48 +++++++++++++++++++++++++++++++++++++++++++++++-
>  1 file changed, 47 insertions(+), 1 deletion(-)
>
> diff --git a/man8/ld.so.8 b/man8/ld.so.8
> index cf03cb85e..5b02ae88f 100644
> --- a/man8/ld.so.8
> +++ b/man8/ld.so.8
> @@ -208,6 +208,14 @@ The objects in
>  .I list
>  are delimited by colons.
>  .TP
> +.BI \-\-glibc-hwcaps-mask " list"
> +only search built-in subdirectories if in
> +.IR list .
> +.TP
> +.BI \-\-glibc-hwcaps-prepend " list"
> +Search glibc-hwcaps subdirectories in
> +.IR list .
> +.TP
>  .B \-\-inhibit\-cache
>  Do not use
>  .IR /etc/ld.so.cache .
> @@ -808,7 +816,7 @@ as a temporary workaround to a library misconfiguration issue.)
>  .I lib*.so*
>  shared objects
>  .SH NOTES
> -.SS Hardware capabilities
> +.SS Legacy Hardware capabilities (from glibc 2.5 to glibc 2.37)
>  Some shared objects are compiled using hardware-specific instructions which do
>  not exist on every CPU.
>  Such objects should be installed in directories whose names define the
> @@ -843,6 +851,44 @@ z900, z990, z9-109, z10, zarch
>  .B x86 (32-bit only)
>  acpi, apic, clflush, cmov, cx8, dts, fxsr, ht, i386, i486, i586, i686, mca, mmx,
>  mtrr, pat, pbe, pge, pn, pse36, sep, ss, sse, sse2, tm
> +.SS glibc Hardware capabilities (from glibc 2.33)
> +The legacy hardware capabilities combinations has the drawback where
> +each feature name incur in
> +cascading extra paths added on the search path list,

IMO, this part could use some rephrasing. How about:
The legacy hardware capabilities support has the drawback that each
feature grows the number of paths added to the search list.

Also, maybe this would better fit under the legacy capabilities section?

> +adding a lot of overhead on
I think "adding a lot of overhead to" sounds better here. "Adding to"
instead of "adding on".

> +.B ld.so
> +during library resolution.
> +For instance, on x86 32-bit, if the hardware
> +supports
> +.B i686
> +and
> +.BR sse2
> +, the resulting search path will be
> +.BR i686/sse2:i686:sse2:. .
> +A new capability
> +.B newcap
> +will set the search path to
> +.BR newcap/i686/sse2:newcap/i686:newcap/sse2:newcap:i686/sse2:i686:sse2: .
> +.PP
> +glibc 2.33 added a new hardware capability scheme,
> +where each ABI can define

Maybe s/each ABI/each ABI version/? I'm not familiar with the feature,
just guessing from the examples below; they were very helpful, IMO.

> +a set of paths based on expected hardware support.

a set of paths where to find the expected hardware support?

This is based on how I (mis)understood the feature, though, if that's
wrong, then the above might also be wrong. I guess the x86-64 glibc
would install the x86-64-v2, -v3 and -v4 directories on disk; when
running a program, glibc can then check what the current CPU supports
- say if it supports x86-64-v3, it loads the contents of that folder?

> +Each path is added depending of the hardware of the machine,
> +and they are not combined together.
> +They also have priority over the legacy hardware capabilities.
> +The following paths are currently supported.
> +.TP
> +.B PowerPC (64-bit little-endian only)
> +power9, power10
> +.TP
> +.B s390 (64-bit only)
> +z13, z14, z15, z16
> +.TP
> +.B x86 (64-bit only)
> +x86-64-v2, x86-64-v3, x86-64-v4
> +.PP
> +The glibc 2.37 removed support for the legacy hardware capabilities.
s/The glibc/glibc

Regards,
Stefan.

> +.
>  .SH SEE ALSO
>  .BR ld (1),
>  .BR ldd (1),
> --
> 2.34.1
>

Re: [PATCH v3] ld.so.8: Describe glibc Hardware capabilities

[Adhemerval Zanella Netto]

On 19/10/23 16:10, Stefan Puiu wrote:
> Hi and sorry to jump in so late in the discussion,

Thanks for the review!

> 
> On Thu, Oct 19, 2023 at 8:23 PM Adhemerval Zanella
> <adhemerval.zanella@linaro.org> wrote:
>>
>> It was added on glibc 2.33 as a way to improve path search, since
>> legacy hardware capabilities combination scheme do not scale
>> properly with new hardware support.  The legacy support was removed
>> on glibc 2.37, so it is the only scheme currently supported.
> 
> I would suggest (caveat: non-native English speaker here):
> 
> s/It was added on glibc/The feature was added in glibc/
> s/improve path search/improve the path search/
> s/since legacy/since the legacy/
> s/hardware capabilities combination scheme do not/hardware
> capabilities scheme does not/
> s/was removed on glibc/was removed in glibc/

Ack.

> 
>>
>> Signed-off-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
>> ---
>>  man8/ld.so.8 | 48 +++++++++++++++++++++++++++++++++++++++++++++++-
>>  1 file changed, 47 insertions(+), 1 deletion(-)
>>
>> diff --git a/man8/ld.so.8 b/man8/ld.so.8
>> index cf03cb85e..5b02ae88f 100644
>> --- a/man8/ld.so.8
>> +++ b/man8/ld.so.8
>> @@ -208,6 +208,14 @@ The objects in
>>  .I list
>>  are delimited by colons.
>>  .TP
>> +.BI \-\-glibc-hwcaps-mask " list"
>> +only search built-in subdirectories if in
>> +.IR list .
>> +.TP
>> +.BI \-\-glibc-hwcaps-prepend " list"
>> +Search glibc-hwcaps subdirectories in
>> +.IR list .
>> +.TP
>>  .B \-\-inhibit\-cache
>>  Do not use
>>  .IR /etc/ld.so.cache .
>> @@ -808,7 +816,7 @@ as a temporary workaround to a library misconfiguration issue.)
>>  .I lib*.so*
>>  shared objects
>>  .SH NOTES
>> -.SS Hardware capabilities
>> +.SS Legacy Hardware capabilities (from glibc 2.5 to glibc 2.37)
>>  Some shared objects are compiled using hardware-specific instructions which do
>>  not exist on every CPU.
>>  Such objects should be installed in directories whose names define the
>> @@ -843,6 +851,44 @@ z900, z990, z9-109, z10, zarch
>>  .B x86 (32-bit only)
>>  acpi, apic, clflush, cmov, cx8, dts, fxsr, ht, i386, i486, i586, i686, mca, mmx,
>>  mtrr, pat, pbe, pge, pn, pse36, sep, ss, sse, sse2, tm
>> +.SS glibc Hardware capabilities (from glibc 2.33)
>> +The legacy hardware capabilities combinations has the drawback where
>> +each feature name incur in
>> +cascading extra paths added on the search path list,
> 
> IMO, this part could use some rephrasing. How about:
> The legacy hardware capabilities support has the drawback that each
> feature grows the number of paths added to the search list.

The main problem was it did not only grows, but it did quadratically since
the new capability is combined with the others.  It was minimized because
the actually used capabilities was filtered out by the processor/kernel
advertise features. So maybe:

  The legacy hardware capabilities support has the drawback 
  it requires multiple search paths due the combined supported capabilities,
  and each new feature grows the number of paths added to the search list
  in quadratic manner. 

> 
> Also, maybe this would better fit under the legacy capabilities section?

Indeed, I will move to there.

> 
>> +adding a lot of overhead on
> I think "adding a lot of overhead to" sounds better here. "Adding to"
> instead of "adding on".
> 
>> +.B ld.so
>> +during library resolution.
>> +For instance, on x86 32-bit, if the hardware
>> +supports
>> +.B i686
>> +and
>> +.BR sse2
>> +, the resulting search path will be
>> +.BR i686/sse2:i686:sse2:. .
>> +A new capability
>> +.B newcap
>> +will set the search path to
>> +.BR newcap/i686/sse2:newcap/i686:newcap/sse2:newcap:i686/sse2:i686:sse2: .
>> +.PP
>> +glibc 2.33 added a new hardware capability scheme,
>> +where each ABI can define
> 
> Maybe s/each ABI/each ABI version/? I'm not familiar with the feature,
> just guessing from the examples below; they were very helpful, IMO.

I think it would be better to use 'architecture' here.


> 
>> +a set of paths based on expected hardware support.
> 
> a set of paths where to find the expected hardware support?
> 
> This is based on how I (mis)understood the feature, though, if that's
> wrong, then the above might also be wrong. I guess the x86-64 glibc
> would install the x86-64-v2, -v3 and -v4 directories on disk; when
> running a program, glibc can then check what the current CPU supports
> - say if it supports x86-64-v3, it loads the contents of that folder?

They are search paths, so glibc itself does not install them.  The
glibc Hardware capabilities search paths are constructed based on
pre-defined list (where only a handful architecture actually define
them) that are matched against the supported one by the hardware.
The initial patchset that actually implemented this feature has
more details [1].

It has the advantage that each glibc-hwcap path are not combined
with each other, so there is no quadratic increase when a new
patch is added.

So, let's say you have a x86_64-v3 chip (Haswell or Excavator) [2].
The result search patch will be 

  glibc-hwcaps/x86-64-v3:glibc-hwcaps/x86-64-v2:

It is also supported by ldconfig, so it will also check on all paths
defined on ld.so.conf along with combined one with glibc-hwcap.  So
let's say you have "/lib/x86_64-linux-gnu" on ld.so.conf, ldconfig will
check all possible subpaths based on the glibc-hwcap list:

  /lib/x86_64-linux-gnu/glibc-hwcaps/x86-64-v4
  /lib/x86_64-linux-gnu/glibc-hwcaps/x86-64-v3
  /lib/x86_64-linux-gnu/glibc-hwcaps/x86-64-v2
  /lib/x86_64-linux-gnu/

And only adds a possible candidate iff the file exists.  For instance
if you have:

  /lib/x86_64-linux-gnu/glibc-hwcaps/x86-64-v4/libsomething.so
  /lib/x86_64-linux-gnu/glibc-hwcaps/x86-64-v2/libsomething.so
  /lib/x86_64-linux-gnu/libsomething.so

The ldconfig will setup a ld.so.cache with all the entries, but
ld.so will only select the x86-64-v4/libsomething.so if the CPU supports
the x86-64-v4 (Skylake, Zen4); otherwise it will fallback to
x86-64-v2/libsomething.so if the cpu support x86_64-v2 (Nehalem, Jaguar),
or libsomething.so as the x86_64 baseline.

So maybe a better description would be

  glibc 2.33 added a new hardware capability scheme,
  where each architecture might define
  a set of paths based on expected hardware support.
  Each path is added on the search path list
  depending of the hardware of the machine.
  Each path is independent and not combined together,
  so it does have the drawbacks of legacy scheme.
  .PP
  For instance, on x86 64-bit, if the hardware supports
  .B x86_64-v3
  (for instance Intel Haswell or AMD Excavator)
  , the resulting search path will be
  .BR glibc-hwcaps/x86-64-v3:glibc-hwcaps/x86-64-v2:.
  The following paths are currently supported, in priority order.
  .TP
  .B PowerPC (64-bit little-endian only)
  power10, power9
  .TP
  .B s390 (64-bit only)
  z16, z15, z14, z13
  .TP
  .B x86 (64-bit only)
  x86-64-v4, x86-64-v3, x86-64-v2
  .PP
  glibc 2.37 removed support for the legacy hardware capabilities.

> 
>> +Each path is added depending of the hardware of the machine,
>> +and they are not combined together.
>> +They also have priority over the legacy hardware capabilities.
>> +The following paths are currently supported.
>> +.TP
>> +.B PowerPC (64-bit little-endian only)
>> +power9, power10
>> +.TP
>> +.B s390 (64-bit only)
>> +z13, z14, z15, z16
>> +.TP
>> +.B x86 (64-bit only)
>> +x86-64-v2, x86-64-v3, x86-64-v4
>> +.PP
>> +The glibc 2.37 removed support for the legacy hardware capabilities.
> s/The glibc/glibc
> 
> Regards,
> Stefan.
> 
>> +.
>>  .SH SEE ALSO
>>  .BR ld (1),
>>  .BR ldd (1),
>> --
>> 2.34.1
>>

[1] https://sourceware.org/pipermail/libc-alpha/2020-June/115250.html
[2] https://lists.llvm.org/pipermail/llvm-dev/2020-July/143289.html

Re: [PATCH v3] ld.so.8: Describe glibc Hardware capabilities

[Stefan Puiu]

Hi Adhemerval,

On Fri, Oct 20, 2023 at 8:50 PM Adhemerval Zanella Netto
<adhemerval.zanella@linaro.org> wrote:
>
>
>
> On 19/10/23 16:10, Stefan Puiu wrote:
> > Hi and sorry to jump in so late in the discussion,
>
> Thanks for the review!

Well, I'm glad to nitpi... um, help :).

>
> >
> > On Thu, Oct 19, 2023 at 8:23 PM Adhemerval Zanella
> > <adhemerval.zanella@linaro.org> wrote:
> >>
> >> It was added on glibc 2.33 as a way to improve path search, since
> >> legacy hardware capabilities combination scheme do not scale
> >> properly with new hardware support.  The legacy support was removed
> >> on glibc 2.37, so it is the only scheme currently supported.
> >
> > I would suggest (caveat: non-native English speaker here):
> >
> > s/It was added on glibc/The feature was added in glibc/
> > s/improve path search/improve the path search/
> > s/since legacy/since the legacy/
> > s/hardware capabilities combination scheme do not/hardware
> > capabilities scheme does not/
> > s/was removed on glibc/was removed in glibc/
>
> Ack.
>
> >
> >>
> >> Signed-off-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
> >> ---
> >>  man8/ld.so.8 | 48 +++++++++++++++++++++++++++++++++++++++++++++++-
> >>  1 file changed, 47 insertions(+), 1 deletion(-)
> >>
> >> diff --git a/man8/ld.so.8 b/man8/ld.so.8
> >> index cf03cb85e..5b02ae88f 100644
> >> --- a/man8/ld.so.8
> >> +++ b/man8/ld.so.8
> >> @@ -208,6 +208,14 @@ The objects in
> >>  .I list
> >>  are delimited by colons.
> >>  .TP
> >> +.BI \-\-glibc-hwcaps-mask " list"
> >> +only search built-in subdirectories if in
> >> +.IR list .
> >> +.TP
> >> +.BI \-\-glibc-hwcaps-prepend " list"
> >> +Search glibc-hwcaps subdirectories in
> >> +.IR list .
> >> +.TP
> >>  .B \-\-inhibit\-cache
> >>  Do not use
> >>  .IR /etc/ld.so.cache .
> >> @@ -808,7 +816,7 @@ as a temporary workaround to a library misconfiguration issue.)
> >>  .I lib*.so*
> >>  shared objects
> >>  .SH NOTES
> >> -.SS Hardware capabilities
> >> +.SS Legacy Hardware capabilities (from glibc 2.5 to glibc 2.37)
> >>  Some shared objects are compiled using hardware-specific instructions which do
> >>  not exist on every CPU.
> >>  Such objects should be installed in directories whose names define the
> >> @@ -843,6 +851,44 @@ z900, z990, z9-109, z10, zarch
> >>  .B x86 (32-bit only)
> >>  acpi, apic, clflush, cmov, cx8, dts, fxsr, ht, i386, i486, i586, i686, mca, mmx,
> >>  mtrr, pat, pbe, pge, pn, pse36, sep, ss, sse, sse2, tm
> >> +.SS glibc Hardware capabilities (from glibc 2.33)
> >> +The legacy hardware capabilities combinations has the drawback where
> >> +each feature name incur in
> >> +cascading extra paths added on the search path list,
> >
> > IMO, this part could use some rephrasing. How about:
> > The legacy hardware capabilities support has the drawback that each
> > feature grows the number of paths added to the search list.
>
> The main problem was it did not only grows, but it did quadratically since
> the new capability is combined with the others.  It was minimized because
> the actually used capabilities was filtered out by the processor/kernel
> advertise features. So maybe:
>
>   The legacy hardware capabilities support has the drawback
>   it requires multiple search paths due the combined supported capabilities,
>   and each new feature grows the number of paths added to the search list
>   in quadratic manner.

Maybe:
The legacy hardware capabilities support has the drawback that each
new feature added grows the search path exponentially, because it has
to be added to every combination of the other existing features.

And then you have the sse2 example below, which I think clarifies the
point very well.

>
> >
> > Also, maybe this would better fit under the legacy capabilities section?
>
> Indeed, I will move to there.
>
> >
> >> +adding a lot of overhead on
> > I think "adding a lot of overhead to" sounds better here. "Adding to"
> > instead of "adding on".
> >
> >> +.B ld.so
> >> +during library resolution.
> >> +For instance, on x86 32-bit, if the hardware
> >> +supports
> >> +.B i686
> >> +and
> >> +.BR sse2
> >> +, the resulting search path will be
> >> +.BR i686/sse2:i686:sse2:. .
> >> +A new capability
> >> +.B newcap
> >> +will set the search path to
> >> +.BR newcap/i686/sse2:newcap/i686:newcap/sse2:newcap:i686/sse2:i686:sse2: .
> >> +.PP
> >> +glibc 2.33 added a new hardware capability scheme,
> >> +where each ABI can define
> >
> > Maybe s/each ABI/each ABI version/? I'm not familiar with the feature,
> > just guessing from the examples below; they were very helpful, IMO.
>
> I think it would be better to use 'architecture' here.

Maybe 'architecture level' could work, based on Florian's mail to
llvm-dev? Architecture is too broad, I think, and Florian says that
the levels don't correspond to micro-architectures, since in the
x84-64 the progression is not so clear.

Anyway, the links were very helpful. Maybe you can leave a pointer to
them somewhere, at least in the change description?

>
>
> >
> >> +a set of paths based on expected hardware support.
> >
> > a set of paths where to find the expected hardware support?
> >
> > This is based on how I (mis)understood the feature, though, if that's
> > wrong, then the above might also be wrong. I guess the x86-64 glibc
> > would install the x86-64-v2, -v3 and -v4 directories on disk; when
> > running a program, glibc can then check what the current CPU supports
> > - say if it supports x86-64-v3, it loads the contents of that folder?
>
> They are search paths, so glibc itself does not install them.  The
> glibc Hardware capabilities search paths are constructed based on
> pre-defined list (where only a handful architecture actually define
> them) that are matched against the supported one by the hardware.
> The initial patchset that actually implemented this feature has
> more details [1].

By "glibc installs them", I meant that I would expect them to be part
of glibc, is that not the case?

>
> It has the advantage that each glibc-hwcap path are not combined
> with each other, so there is no quadratic increase when a new
> patch is added.
>
> So, let's say you have a x86_64-v3 chip (Haswell or Excavator) [2].
> The result search patch will be
>
>   glibc-hwcaps/x86-64-v3:glibc-hwcaps/x86-64-v2:
>
> It is also supported by ldconfig, so it will also check on all paths
> defined on ld.so.conf along with combined one with glibc-hwcap.  So
> let's say you have "/lib/x86_64-linux-gnu" on ld.so.conf, ldconfig will
> check all possible subpaths based on the glibc-hwcap list:
>
>   /lib/x86_64-linux-gnu/glibc-hwcaps/x86-64-v4
>   /lib/x86_64-linux-gnu/glibc-hwcaps/x86-64-v3
>   /lib/x86_64-linux-gnu/glibc-hwcaps/x86-64-v2
>   /lib/x86_64-linux-gnu/
>
> And only adds a possible candidate iff the file exists.  For instance
> if you have:
>
>   /lib/x86_64-linux-gnu/glibc-hwcaps/x86-64-v4/libsomething.so
>   /lib/x86_64-linux-gnu/glibc-hwcaps/x86-64-v2/libsomething.so
>   /lib/x86_64-linux-gnu/libsomething.so
>
> The ldconfig will setup a ld.so.cache with all the entries, but
> ld.so will only select the x86-64-v4/libsomething.so if the CPU supports
> the x86-64-v4 (Skylake, Zen4); otherwise it will fallback to
> x86-64-v2/libsomething.so if the cpu support x86_64-v2 (Nehalem, Jaguar),
> or libsomething.so as the x86_64 baseline.

Thanks, that makes it much clearer.

>
> So maybe a better description would be
>
>   glibc 2.33 added a new hardware capability scheme,
>   where each architecture might define
>   a set of paths based on expected hardware support.
>   Each path is added on the search path list
>   depending of the hardware of the machine.
>   Each path is independent and not combined together,
>   so it does have the drawbacks of legacy scheme.

How about:
glibc 2.33 added a new hardware capability scheme, where under each
CPU architecture, certain levels can be defined, grouping support for
certain features or special instructions. Each architecture level has
a fixed set of paths that it adds to the dynamic linker search list,
depending on the hardware of the machine. Since each new architecture
level is not combined with previously existing ones, the new scheme
does not have the drawback of growing the dynamic linker search list
uncontrollably.

>   .PP
>   For instance, on x86 64-bit, if the hardware supports
>   .B x86_64-v3
>   (for instance Intel Haswell or AMD Excavator)
>   , the resulting search path will be
>   .BR glibc-hwcaps/x86-64-v3:glibc-hwcaps/x86-64-v2:.

I think it would be useful if the existing levels are defined
somewhere; maybe Alex can suggest the proper manpage.

Regards,
Stefan.

>   The following paths are currently supported, in priority order.
>   .TP
>   .B PowerPC (64-bit little-endian only)
>   power10, power9
>   .TP
>   .B s390 (64-bit only)
>   z16, z15, z14, z13
>   .TP
>   .B x86 (64-bit only)
>   x86-64-v4, x86-64-v3, x86-64-v2
>   .PP
>   glibc 2.37 removed support for the legacy hardware capabilities.
>
> >
> >> +Each path is added depending of the hardware of the machine,
> >> +and they are not combined together.
> >> +They also have priority over the legacy hardware capabilities.
> >> +The following paths are currently supported.
> >> +.TP
> >> +.B PowerPC (64-bit little-endian only)
> >> +power9, power10
> >> +.TP
> >> +.B s390 (64-bit only)
> >> +z13, z14, z15, z16
> >> +.TP
> >> +.B x86 (64-bit only)
> >> +x86-64-v2, x86-64-v3, x86-64-v4
> >> +.PP
> >> +The glibc 2.37 removed support for the legacy hardware capabilities.
> > s/The glibc/glibc
> >
> > Regards,
> > Stefan.
> >
> >> +.
> >>  .SH SEE ALSO
> >>  .BR ld (1),
> >>  .BR ldd (1),
> >> --
> >> 2.34.1
> >>
>
> [1] https://sourceware.org/pipermail/libc-alpha/2020-June/115250.html
> [2] https://lists.llvm.org/pipermail/llvm-dev/2020-July/143289.html

Re: [PATCH v3] ld.so.8: Describe glibc Hardware capabilities

[Alejandro Colomar]

[-- Attachment #1: Type: text/plain, Size: 896 bytes --]

Hi Stefan,

On Tue, Oct 24, 2023 at 10:59:12AM +0300, Stefan Puiu wrote:
[...]

> > >> Signed-off-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
> > >> ---
> > >>  man8/ld.so.8 | 48 +++++++++++++++++++++++++++++++++++++++++++++++-
> > >>  1 file changed, 47 insertions(+), 1 deletion(-)
[...]
> 
> >   .PP
> >   For instance, on x86 64-bit, if the hardware supports
> >   .B x86_64-v3
> >   (for instance Intel Haswell or AMD Excavator)
> >   , the resulting search path will be
> >   .BR glibc-hwcaps/x86-64-v3:glibc-hwcaps/x86-64-v2:.
> 
> I think it would be useful if the existing levels are defined
> somewhere; maybe Alex can suggest the proper manpage.

I'm sorry; I didn't even know about these things until Adhemerval sent
the patch.  But yeah, I agree that if would be good to define them.  :)

Cheers,
Alex

-- 
<https://www.alejandro-colomar.es/>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v3] ld.so.8: Describe glibc Hardware capabilities

[Stefan Puiu]

Hi Alex,

On Tue, Oct 24, 2023 at 12:43 PM Alejandro Colomar <alx@kernel.org> wrote:
>
> Hi Stefan,
>
> On Tue, Oct 24, 2023 at 10:59:12AM +0300, Stefan Puiu wrote:
> [...]
>
> > > >> Signed-off-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
> > > >> ---
> > > >>  man8/ld.so.8 | 48 +++++++++++++++++++++++++++++++++++++++++++++++-
> > > >>  1 file changed, 47 insertions(+), 1 deletion(-)
> [...]
> >
> > >   .PP
> > >   For instance, on x86 64-bit, if the hardware supports
> > >   .B x86_64-v3
> > >   (for instance Intel Haswell or AMD Excavator)
> > >   , the resulting search path will be
> > >   .BR glibc-hwcaps/x86-64-v3:glibc-hwcaps/x86-64-v2:.
> >
> > I think it would be useful if the existing levels are defined
> > somewhere; maybe Alex can suggest the proper manpage.
>
> I'm sorry; I didn't even know about these things until Adhemerval sent
> the patch.  But yeah, I agree that if would be good to define them.  :)

I didn't know either. The email threads Adhemerval pointed to were
quite useful. That's what I think is one of the benefits of watching
this list, you learn a lot about new Linux features (and in my case,
features that are new to me).

The glibc-hwcaps support seems to be present in my Ubuntu 22.04
install, at least '/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 --help'
seems to imply that.

Regards,
Stefan.

>
> Cheers,
> Alex
>
> --
> <https://www.alejandro-colomar.es/>

Re: [PATCH v3] ld.so.8: Describe glibc Hardware capabilities

[Alejandro Colomar]

[-- Attachment #1: Type: text/plain, Size: 997 bytes --]

Hi Stefan,

On Tue, Oct 24, 2023 at 01:00:09PM +0300, Stefan Puiu wrote:
> Hi Alex,
> 
[...]
> >
> > I'm sorry; I didn't even know about these things until Adhemerval sent
> > the patch.  But yeah, I agree that if would be good to define them.  :)
> 
> I didn't know either. The email threads Adhemerval pointed to were
> quite useful. That's what I think is one of the benefits of watching
> this list, you learn a lot about new Linux features (and in my case,
> features that are new to me).

Indeed.  While doing random fixes to the pages, I learn about functions
or other features I didn't know existed, and later find very useful.

Cheers,
Alex

> 
> The glibc-hwcaps support seems to be present in my Ubuntu 22.04
> install, at least '/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 --help'
> seems to imply that.
> 
> Regards,
> Stefan.
> 
> >
> > Cheers,
> > Alex
> >
> > --
> > <https://www.alejandro-colomar.es/>

-- 
<https://www.alejandro-colomar.es/>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v3] ld.so.8: Describe glibc Hardware capabilities

[Adhemerval Zanella Netto]

On 24/10/23 04:59, Stefan Puiu wrote:
> Hi Adhemerval,
> 
> On Fri, Oct 20, 2023 at 8:50 PM Adhemerval Zanella Netto
> <adhemerval.zanella@linaro.org> wrote:
>>
>>
>>
>> On 19/10/23 16:10, Stefan Puiu wrote:
>>> Hi and sorry to jump in so late in the discussion,
>>
>> Thanks for the review!
> 
> Well, I'm glad to nitpi... um, help :).
> 
>>
>>>
>>> On Thu, Oct 19, 2023 at 8:23 PM Adhemerval Zanella
>>> <adhemerval.zanella@linaro.org> wrote:
>>>>
>>>> It was added on glibc 2.33 as a way to improve path search, since
>>>> legacy hardware capabilities combination scheme do not scale
>>>> properly with new hardware support.  The legacy support was removed
>>>> on glibc 2.37, so it is the only scheme currently supported.
>>>
>>> I would suggest (caveat: non-native English speaker here):
>>>
>>> s/It was added on glibc/The feature was added in glibc/
>>> s/improve path search/improve the path search/
>>> s/since legacy/since the legacy/
>>> s/hardware capabilities combination scheme do not/hardware
>>> capabilities scheme does not/
>>> s/was removed on glibc/was removed in glibc/
>>
>> Ack.
>>
>>>
>>>>
>>>> Signed-off-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
>>>> ---
>>>>  man8/ld.so.8 | 48 +++++++++++++++++++++++++++++++++++++++++++++++-
>>>>  1 file changed, 47 insertions(+), 1 deletion(-)
>>>>
>>>> diff --git a/man8/ld.so.8 b/man8/ld.so.8
>>>> index cf03cb85e..5b02ae88f 100644
>>>> --- a/man8/ld.so.8
>>>> +++ b/man8/ld.so.8
>>>> @@ -208,6 +208,14 @@ The objects in
>>>>  .I list
>>>>  are delimited by colons.
>>>>  .TP
>>>> +.BI \-\-glibc-hwcaps-mask " list"
>>>> +only search built-in subdirectories if in
>>>> +.IR list .
>>>> +.TP
>>>> +.BI \-\-glibc-hwcaps-prepend " list"
>>>> +Search glibc-hwcaps subdirectories in
>>>> +.IR list .
>>>> +.TP
>>>>  .B \-\-inhibit\-cache
>>>>  Do not use
>>>>  .IR /etc/ld.so.cache .
>>>> @@ -808,7 +816,7 @@ as a temporary workaround to a library misconfiguration issue.)
>>>>  .I lib*.so*
>>>>  shared objects
>>>>  .SH NOTES
>>>> -.SS Hardware capabilities
>>>> +.SS Legacy Hardware capabilities (from glibc 2.5 to glibc 2.37)
>>>>  Some shared objects are compiled using hardware-specific instructions which do
>>>>  not exist on every CPU.
>>>>  Such objects should be installed in directories whose names define the
>>>> @@ -843,6 +851,44 @@ z900, z990, z9-109, z10, zarch
>>>>  .B x86 (32-bit only)
>>>>  acpi, apic, clflush, cmov, cx8, dts, fxsr, ht, i386, i486, i586, i686, mca, mmx,
>>>>  mtrr, pat, pbe, pge, pn, pse36, sep, ss, sse, sse2, tm
>>>> +.SS glibc Hardware capabilities (from glibc 2.33)
>>>> +The legacy hardware capabilities combinations has the drawback where
>>>> +each feature name incur in
>>>> +cascading extra paths added on the search path list,
>>>
>>> IMO, this part could use some rephrasing. How about:
>>> The legacy hardware capabilities support has the drawback that each
>>> feature grows the number of paths added to the search list.
>>
>> The main problem was it did not only grows, but it did quadratically since
>> the new capability is combined with the others.  It was minimized because
>> the actually used capabilities was filtered out by the processor/kernel
>> advertise features. So maybe:
>>
>>   The legacy hardware capabilities support has the drawback
>>   it requires multiple search paths due the combined supported capabilities,
>>   and each new feature grows the number of paths added to the search list
>>   in quadratic manner.
> 
> Maybe:
> The legacy hardware capabilities support has the drawback that each
> new feature added grows the search path exponentially, because it has
> to be added to every combination of the other existing features.
> 

Ok, it works for me.

> And then you have the sse2 example below, which I think clarifies the
> point very well.
> 
>>
>>>
>>> Also, maybe this would better fit under the legacy capabilities section?
>>
>> Indeed, I will move to there.
>>
>>>
>>>> +adding a lot of overhead on
>>> I think "adding a lot of overhead to" sounds better here. "Adding to"
>>> instead of "adding on".
>>>
>>>> +.B ld.so
>>>> +during library resolution.
>>>> +For instance, on x86 32-bit, if the hardware
>>>> +supports
>>>> +.B i686
>>>> +and
>>>> +.BR sse2
>>>> +, the resulting search path will be
>>>> +.BR i686/sse2:i686:sse2:. .
>>>> +A new capability
>>>> +.B newcap
>>>> +will set the search path to
>>>> +.BR newcap/i686/sse2:newcap/i686:newcap/sse2:newcap:i686/sse2:i686:sse2: .
>>>> +.PP
>>>> +glibc 2.33 added a new hardware capability scheme,
>>>> +where each ABI can define
>>>
>>> Maybe s/each ABI/each ABI version/? I'm not familiar with the feature,
>>> just guessing from the examples below; they were very helpful, IMO.
>>
>> I think it would be better to use 'architecture' here.
> 
> Maybe 'architecture level' could work, based on Florian's mail to
> llvm-dev? Architecture is too broad, I think, and Florian says that
> the levels don't correspond to micro-architectures, since in the
> x84-64 the progression is not so clear.

Ack, architecture level I think works better here indeed.

> 
> Anyway, the links were very helpful. Maybe you can leave a pointer to
> them somewhere, at least in the change description?

I will add them as a comment.

> 
>>
>>
>>>
>>>> +a set of paths based on expected hardware support.
>>>
>>> a set of paths where to find the expected hardware support?
>>>
>>> This is based on how I (mis)understood the feature, though, if that's
>>> wrong, then the above might also be wrong. I guess the x86-64 glibc
>>> would install the x86-64-v2, -v3 and -v4 directories on disk; when
>>> running a program, glibc can then check what the current CPU supports
>>> - say if it supports x86-64-v3, it loads the contents of that folder?
>>
>> They are search paths, so glibc itself does not install them.  The
>> glibc Hardware capabilities search paths are constructed based on
>> pre-defined list (where only a handful architecture actually define
>> them) that are matched against the supported one by the hardware.
>> The initial patchset that actually implemented this feature has
>> more details [1].
> 
> By "glibc installs them", I meant that I would expect them to be part
> of glibc, is that not the case?

Not really, there are extra paths the loader might consulting when
loading shared libraries (either at startup or by dlopen).  It is
expected that package manager of something analogous to build and
install the optimized libraries on the expected folders.

> 
>>
>> It has the advantage that each glibc-hwcap path are not combined
>> with each other, so there is no quadratic increase when a new
>> patch is added.
>>
>> So, let's say you have a x86_64-v3 chip (Haswell or Excavator) [2].
>> The result search patch will be
>>
>>   glibc-hwcaps/x86-64-v3:glibc-hwcaps/x86-64-v2:
>>
>> It is also supported by ldconfig, so it will also check on all paths
>> defined on ld.so.conf along with combined one with glibc-hwcap.  So
>> let's say you have "/lib/x86_64-linux-gnu" on ld.so.conf, ldconfig will
>> check all possible subpaths based on the glibc-hwcap list:
>>
>>   /lib/x86_64-linux-gnu/glibc-hwcaps/x86-64-v4
>>   /lib/x86_64-linux-gnu/glibc-hwcaps/x86-64-v3
>>   /lib/x86_64-linux-gnu/glibc-hwcaps/x86-64-v2
>>   /lib/x86_64-linux-gnu/
>>
>> And only adds a possible candidate iff the file exists.  For instance
>> if you have:
>>
>>   /lib/x86_64-linux-gnu/glibc-hwcaps/x86-64-v4/libsomething.so
>>   /lib/x86_64-linux-gnu/glibc-hwcaps/x86-64-v2/libsomething.so
>>   /lib/x86_64-linux-gnu/libsomething.so
>>
>> The ldconfig will setup a ld.so.cache with all the entries, but
>> ld.so will only select the x86-64-v4/libsomething.so if the CPU supports
>> the x86-64-v4 (Skylake, Zen4); otherwise it will fallback to
>> x86-64-v2/libsomething.so if the cpu support x86_64-v2 (Nehalem, Jaguar),
>> or libsomething.so as the x86_64 baseline.
> 
> Thanks, that makes it much clearer.
> 
>>
>> So maybe a better description would be
>>
>>   glibc 2.33 added a new hardware capability scheme,
>>   where each architecture might define
>>   a set of paths based on expected hardware support.
>>   Each path is added on the search path list
>>   depending of the hardware of the machine.
>>   Each path is independent and not combined together,
>>   so it does have the drawbacks of legacy scheme.
> 
> How about:
> glibc 2.33 added a new hardware capability scheme, where under each
> CPU architecture, certain levels can be defined, grouping support for
> certain features or special instructions. Each architecture level has
> a fixed set of paths that it adds to the dynamic linker search list,
> depending on the hardware of the machine. Since each new architecture
> level is not combined with previously existing ones, the new scheme
> does not have the drawback of growing the dynamic linker search list
> uncontrollably.

Works for me, thanks.

> 
>>   .PP
>>   For instance, on x86 64-bit, if the hardware supports
>>   .B x86_64-v3
>>   (for instance Intel Haswell or AMD Excavator)
>>   , the resulting search path will be
>>   .BR glibc-hwcaps/x86-64-v3:glibc-hwcaps/x86-64-v2:.
> 
> I think it would be useful if the existing levels are defined
> somewhere; maybe Alex can suggest the proper manpage.

For x86_64, these are in fact defined by the ABI itself [1].
The powerpc and s390x names based on the IBM chips names,
but that also maps to specific ISA as well.  I will add a 
comment about it.

https://gitlab.com/x86-psABIs/x86-64-ABI/-/blob/master/x86-64-ABI/low-level-sys-info.tex

> 
> Regards,
> Stefan.
> 
>>   The following paths are currently supported, in priority order.
>>   .TP
>>   .B PowerPC (64-bit little-endian only)
>>   power10, power9
>>   .TP
>>   .B s390 (64-bit only)
>>   z16, z15, z14, z13
>>   .TP
>>   .B x86 (64-bit only)
>>   x86-64-v4, x86-64-v3, x86-64-v2
>>   .PP
>>   glibc 2.37 removed support for the legacy hardware capabilities.
>>
>>>
>>>> +Each path is added depending of the hardware of the machine,
>>>> +and they are not combined together.
>>>> +They also have priority over the legacy hardware capabilities.
>>>> +The following paths are currently supported.
>>>> +.TP
>>>> +.B PowerPC (64-bit little-endian only)
>>>> +power9, power10
>>>> +.TP
>>>> +.B s390 (64-bit only)
>>>> +z13, z14, z15, z16
>>>> +.TP
>>>> +.B x86 (64-bit only)
>>>> +x86-64-v2, x86-64-v3, x86-64-v4
>>>> +.PP
>>>> +The glibc 2.37 removed support for the legacy hardware capabilities.
>>> s/The glibc/glibc
>>>
>>> Regards,
>>> Stefan.
>>>
>>>> +.
>>>>  .SH SEE ALSO
>>>>  .BR ld (1),
>>>>  .BR ldd (1),
>>>> --
>>>> 2.34.1
>>>>
>>
>> [1] https://sourceware.org/pipermail/libc-alpha/2020-June/115250.html
>> [2] https://lists.llvm.org/pipermail/llvm-dev/2020-July/143289.html