[PATCH 0/1] glibc: re-package for libnss-db

[PATCH 0/1] glibc: re-package for libnss-db

[Chen Qi]

*** BLURB HERE ***
The following changes since commit 63a4ff7cf5f7d1671ab85800bc2212dd9cd9748d:

  bitbake: checksum: sanity check path when recursively checksumming (2018-08-14 16:32:48 +0100)

are available in the git repository at:

  git://git.pokylinux.org/poky-contrib ChenQi/libnss-db
  http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=ChenQi/libnss-db

Jackie Huang (1):
  glibc: re-package for libnss-db

 meta/recipes-core/glibc/glibc-package.inc | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

-- 
1.9.1

[PATCH 1/1] glibc: re-package for libnss-db

[Chen Qi]

From: Jackie Huang <jackie.huang@windriver.com>

On other distros like ubuntu/centos, libnss-db usually provides:
- The libraries
- The Makefile to create database
  (in /var/db for centos, /var/lib/misc/ for ubuntu)
- The makedb command (it's in glibc-common for centos7)

What we had is:
- The libraries are in glibc-extra-nss
- The Makefile is removed
- The makedb command is in glibc-utils (lack of dependency)

So when glibc-extra-nss is installed but glibc-utils is not,
we see error like:
nscd[165]: 165 checking for monitored file `/var/db/group.db': No such file or directory
nscd[165]: 165 checking for monitored file `/var/db/passwd.db': No such file or directory

And there is not an easy way to create these databases.

To fix the issue:
- Re-package the libraries into libnss-db
- Don't remove the Makefile and add it in libnss-db
- Add RDEPENDS for libnss-db on glibc-utils and make

Notes:
For external toolchain, an extra package 'libnss-db' need to be provided
If replacing glibc from core.

Signed-off-by: Jackie Huang <jackie.huang@windriver.com>
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
---
 meta/recipes-core/glibc/glibc-package.inc | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/meta/recipes-core/glibc/glibc-package.inc b/meta/recipes-core/glibc/glibc-package.inc
index 8ac01ce..305094f 100644
--- a/meta/recipes-core/glibc/glibc-package.inc
+++ b/meta/recipes-core/glibc/glibc-package.inc
@@ -1,6 +1,6 @@
 INHIBIT_SYSROOT_STRIP = "1"
 
-PACKAGES = "${PN}-dbg catchsegv sln nscd ldd tzcode glibc-thread-db ${PN}-pic libcidn libmemusage libsegfault ${PN}-pcprofile libsotruss ${PN} ${PN}-utils glibc-extra-nss ${PN}-dev ${PN}-staticdev ${PN}-doc"
+PACKAGES = "${PN}-dbg catchsegv sln nscd ldd tzcode glibc-thread-db ${PN}-pic libcidn libmemusage libnss-db libsegfault ${PN}-pcprofile libsotruss ${PN} ${PN}-utils glibc-extra-nss ${PN}-dev ${PN}-staticdev ${PN}-doc"
 
 # The ld.so in this glibc supports the GNU_HASH
 RPROVIDES_${PN} = "eglibc rtld(GNU_HASH)"
@@ -23,6 +23,8 @@ FILES_ldd = "${bindir}/ldd"
 FILES_libsegfault = "${base_libdir}/libSegFault*"
 FILES_libcidn = "${base_libdir}/libcidn-*.so ${base_libdir}/libcidn.so.*"
 FILES_libmemusage = "${base_libdir}/libmemusage.so"
+FILES_libnss-db = "${base_libdir}/libnss_db.so.* ${base_libdir}/libnss_db-*.so ${localstatedir}/db/Makefile"
+RDEPENDS_libnss-db = "${PN}-utils make"
 FILES_glibc-extra-nss = "${base_libdir}/libnss_*-*.so ${base_libdir}/libnss_*.so.*"
 FILES_sln = "${base_sbindir}/sln"
 FILES_${PN}-pic = "${libdir}/*_pic.a ${libdir}/*_pic.map ${libdir}/libc_pic/*.o"
@@ -59,7 +61,6 @@ inherit libc-common multilib_header
 
 do_install_append () {
 	rm -f ${D}${sysconfdir}/localtime
-	rm -rf ${D}${localstatedir}
 
 	# remove empty glibc dir
 	if [ -d ${D}${libexecdir} ]; then
-- 
1.9.1

Re: [PATCH 1/1] glibc: re-package for libnss-db

[Richard Purdie]

On Wed, 2018-08-15 at 15:00 +0800, Chen Qi wrote:
> From: Jackie Huang <jackie.huang@windriver.com>
> 
> On other distros like ubuntu/centos, libnss-db usually provides:
> - The libraries
> - The Makefile to create database
>   (in /var/db for centos, /var/lib/misc/ for ubuntu)
> - The makedb command (it's in glibc-common for centos7)
> 
> What we had is:
> - The libraries are in glibc-extra-nss
> - The Makefile is removed
> - The makedb command is in glibc-utils (lack of dependency)
> 
> So when glibc-extra-nss is installed but glibc-utils is not,
> we see error like:
> nscd[165]: 165 checking for monitored file `/var/db/group.db': No
> such file or directory
> nscd[165]: 165 checking for monitored file `/var/db/passwd.db': No
> such file or directory
> 
> And there is not an easy way to create these databases.
> 
> To fix the issue:
> - Re-package the libraries into libnss-db
> - Don't remove the Makefile and add it in libnss-db
> - Add RDEPENDS for libnss-db on glibc-utils and make
> 
> Notes:
> For external toolchain, an extra package 'libnss-db' need to be
> provided
> If replacing glibc from core.
> 
> Signed-off-by: Jackie Huang <jackie.huang@windriver.com>
> Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
> ---
>  meta/recipes-core/glibc/glibc-package.inc | 5 +++--
>  1 file changed, 3 insertions(+), 2 deletions(-)

This breaks on the autobuilder in oe-selftest:

2018-08-15 13:09:15,551 - oe-selftest - INFO - ======================================================================
2018-08-15 13:09:15,551 - oe-selftest - INFO - FAIL: bbtests.BitbakeTests.test_non_gplv3 (subunit.RemotedTestCase)
2018-08-15 13:09:15,552 - oe-selftest - INFO - ----------------------------------------------------------------------
2018-08-15 13:09:15,552 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/core/decorator/__init__.py", line 32, in wrapped_f
    return func(*args, **kwargs)
  File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/selftest/cases/bbtests.py", line 246, in test_non_gplv3
    self.assertEqual(result.status, 0, "Bitbake failed, exit code %s, output %s" % (result.status, result.output))
  File "/usr/lib64/python3.6/unittest/case.py", line 829, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/lib64/python3.6/unittest/case.py", line 822, in _baseAssertEqual
    raise self.failureException(msg)
AssertionError: 1 != 0 : Bitbake failed, exit code 1, output Parsing recipes...done.
Parsing of 811 .bb files complete (0 cached, 811 parsed). 1277 targets, 99 skipped, 0 masked, 0 errors.
Removing 10 recipes from the core2-64 sysroot...done.
Removing 10 recipes from the qemux86_64 sysroot...done.
NOTE: Resolving any missing task queue dependencies
ERROR: Nothing RPROVIDES 'make' (but /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/recipes-core/glibc/glibc_2.28.bb RDEPENDS on or otherwise requires it)
make was skipped: it has an incompatible license: GPLv3 & LGPLv2
NOTE: Runtime target 'make' is unbuildable, removing...
Missing or unbuildable dependency chain was: ['make']
ERROR: Required build target 'selftest-ed' has no buildable providers.
Missing or unbuildable dependency chain was: ['selftest-ed', 'virtual/libc', 'make']

Summary: There were 2 ERROR messages shown, returning a non-zero exit code.

https://autobuilder.yocto.io/builders/nightly-oe-selftest/builds/1281/steps/Running%20oe-selftest/logs/stdio

We really don't want a make dependency in glibc :(

Cheers,

Richard

Re: [PATCH 1/1] glibc: re-package for libnss-db

[Randy MacLeod]

On 08/15/2018 10:02 AM, Richard Purdie wrote:
> On Wed, 2018-08-15 at 15:00 +0800, Chen Qi wrote:
>> From: Jackie Huang <jackie.huang@windriver.com>
>>
>> On other distros like ubuntu/centos, libnss-db usually provides:
>> - The libraries
>> - The Makefile to create database
>>    (in /var/db for centos, /var/lib/misc/ for ubuntu)
>> - The makedb command (it's in glibc-common for centos7)
>>
>> What we had is:
>> - The libraries are in glibc-extra-nss
>> - The Makefile is removed
>> - The makedb command is in glibc-utils (lack of dependency)
>>
>> So when glibc-extra-nss is installed but glibc-utils is not,
>> we see error like:
>> nscd[165]: 165 checking for monitored file `/var/db/group.db': No
>> such file or directory
>> nscd[165]: 165 checking for monitored file `/var/db/passwd.db': No
>> such file or directory
>>
>> And there is not an easy way to create these databases.
>>
>> To fix the issue:
>> - Re-package the libraries into libnss-db
>> - Don't remove the Makefile and add it in libnss-db
>> - Add RDEPENDS for libnss-db on glibc-utils and make
>>
>> Notes:
>> For external toolchain, an extra package 'libnss-db' need to be
>> provided
>> If replacing glibc from core.
>>
>> Signed-off-by: Jackie Huang <jackie.huang@windriver.com>
>> Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
>> ---
>>   meta/recipes-core/glibc/glibc-package.inc | 5 +++--
>>   1 file changed, 3 insertions(+), 2 deletions(-)
> 
> This breaks on the autobuilder in oe-selftest:
> 
> 2018-08-15 13:09:15,551 - oe-selftest - INFO - ======================================================================


...

> NOTE: Resolving any missing task queue dependencies
> ERROR: Nothing RPROVIDES 'make' (but /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/recipes-core/glibc/glibc_2.28.bb RDEPENDS on or otherwise requires it)
> make was skipped: it has an incompatible license: GPLv3 & LGPLv2
> NOTE: Runtime target 'make' is unbuildable, removing...
> Missing or unbuildable dependency chain was: ['make']
> ERROR: Required build target 'selftest-ed' has no buildable providers.
> Missing or unbuildable dependency chain was: ['selftest-ed', 'virtual/libc', 'make']
> 
> Summary: There were 2 ERROR messages shown, returning a non-zero exit code.
> 
> https://autobuilder.yocto.io/builders/nightly-oe-selftest/builds/1281/steps/Running%20oe-selftest/logs/stdio
> 
> We really don't want a make dependency in glibc :(

Switching from RDEPENDS to RSUGGESTS  seems like a
reasonable solution. The GPLv2 build would work but
libnss-db would fail unless meta-gplv2 were used to
get an old version of make.

One way to avoid needing make is to demand that the bdb
files are generated during the build. That would be nice
but it shouldn't be a requirement since it's not the
typical workflow and since the db will usually be dynamic.

History and more details below.

../Randy

To support on target generation of the bdb files that
libnss-db uses then we need to:
  - separate the libnss-db files into a libnss-db pkg
  - *suggest* a 'make' as described below:

Back in January, Jackie had replied to:
    [OE-core] [PATCH 0/2 v2] glibc: fixes for nscd and libnss-db

with:

> Sorry, I didn't noticed that make is gplv3. I change the dependency to
> RSUGGESTS and the test passed:
> 
> $ oe-selftest -r bbtests.BitbakeTests.test_non_gplv3
> ...
> 2018-01-16 21:46:22,540 - oe-selftest - INFO - oe-selftest - OK - All required tests passed

but of course that papers over the failure by allowing the build to
pass even though runtime needs to include make to be able to
rebuild the db(s). And of course libnss-db isn't even being tested yet
so once we add that to oeqa, the test will fail for GPLv2-only images
where meta-gplv2 isn't added to the list of layers. I'm fine with
such oeqa failing when meta-gplv2 isn't around.


Qi, if no one comes up with a better idea,
please test that as well as:
   $ oe-selftest -r bbtests.BitbakeTests.test_non_gplv3
so that Richard doesn't burn another build cycle on it.

This isn't urgent but it would be nice to resolve the
issue properly for M3 without driving Richard around the bend!

Thanks,
../Randy


> 
> Cheers,
> 
> Richard
> 


-- 
# Randy MacLeod
# Wind River Linux

Re: [PATCH 1/1] glibc: re-package for libnss-db

[ChenQi]

On 08/16/2018 04:41 AM, Randy MacLeod wrote:
> On 08/15/2018 10:02 AM, Richard Purdie wrote:
>> On Wed, 2018-08-15 at 15:00 +0800, Chen Qi wrote:
>>> From: Jackie Huang <jackie.huang@windriver.com>
>>>
>>> On other distros like ubuntu/centos, libnss-db usually provides:
>>> - The libraries
>>> - The Makefile to create database
>>>    (in /var/db for centos, /var/lib/misc/ for ubuntu)
>>> - The makedb command (it's in glibc-common for centos7)
>>>
>>> What we had is:
>>> - The libraries are in glibc-extra-nss
>>> - The Makefile is removed
>>> - The makedb command is in glibc-utils (lack of dependency)
>>>
>>> So when glibc-extra-nss is installed but glibc-utils is not,
>>> we see error like:
>>> nscd[165]: 165 checking for monitored file `/var/db/group.db': No
>>> such file or directory
>>> nscd[165]: 165 checking for monitored file `/var/db/passwd.db': No
>>> such file or directory
>>>
>>> And there is not an easy way to create these databases.
>>>
>>> To fix the issue:
>>> - Re-package the libraries into libnss-db
>>> - Don't remove the Makefile and add it in libnss-db
>>> - Add RDEPENDS for libnss-db on glibc-utils and make
>>>
>>> Notes:
>>> For external toolchain, an extra package 'libnss-db' need to be
>>> provided
>>> If replacing glibc from core.
>>>
>>> Signed-off-by: Jackie Huang <jackie.huang@windriver.com>
>>> Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
>>> ---
>>>   meta/recipes-core/glibc/glibc-package.inc | 5 +++--
>>>   1 file changed, 3 insertions(+), 2 deletions(-)
>>
>> This breaks on the autobuilder in oe-selftest:
>>
>> 2018-08-15 13:09:15,551 - oe-selftest - INFO - 
>> ======================================================================
>
>
> ...
>
>> NOTE: Resolving any missing task queue dependencies
>> ERROR: Nothing RPROVIDES 'make' (but 
>> /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/recipes-core/glibc/glibc_2.28.bb 
>> RDEPENDS on or otherwise requires it)
>> make was skipped: it has an incompatible license: GPLv3 & LGPLv2
>> NOTE: Runtime target 'make' is unbuildable, removing...
>> Missing or unbuildable dependency chain was: ['make']
>> ERROR: Required build target 'selftest-ed' has no buildable providers.
>> Missing or unbuildable dependency chain was: ['selftest-ed', 
>> 'virtual/libc', 'make']
>>
>> Summary: There were 2 ERROR messages shown, returning a non-zero exit 
>> code.
>>
>> https://autobuilder.yocto.io/builders/nightly-oe-selftest/builds/1281/steps/Running%20oe-selftest/logs/stdio 
>>
>>
>> We really don't want a make dependency in glibc :(
>
> Switching from RDEPENDS to RSUGGESTS  seems like a
> reasonable solution. The GPLv2 build would work but
> libnss-db would fail unless meta-gplv2 were used to
> get an old version of make.
>
> One way to avoid needing make is to demand that the bdb
> files are generated during the build. That would be nice
> but it shouldn't be a requirement since it's not the
> typical workflow and since the db will usually be dynamic.
>
> History and more details below.
>
> ../Randy
>
> To support on target generation of the bdb files that
> libnss-db uses then we need to:
>  - separate the libnss-db files into a libnss-db pkg
>  - *suggest* a 'make' as described below:
>
> Back in January, Jackie had replied to:
>    [OE-core] [PATCH 0/2 v2] glibc: fixes for nscd and libnss-db
>
> with:
>
>> Sorry, I didn't noticed that make is gplv3. I change the dependency to
>> RSUGGESTS and the test passed:
>>
>> $ oe-selftest -r bbtests.BitbakeTests.test_non_gplv3
>> ...
>> 2018-01-16 21:46:22,540 - oe-selftest - INFO - oe-selftest - OK - All 
>> required tests passed
>
> but of course that papers over the failure by allowing the build to
> pass even though runtime needs to include make to be able to
> rebuild the db(s). And of course libnss-db isn't even being tested yet
> so once we add that to oeqa, the test will fail for GPLv2-only images
> where meta-gplv2 isn't added to the list of layers. I'm fine with
> such oeqa failing when meta-gplv2 isn't around.
>
>
> Qi, if no one comes up with a better idea,
> please test that as well as:
>   $ oe-selftest -r bbtests.BitbakeTests.test_non_gplv3
> so that Richard doesn't burn another build cycle on it.
>
> This isn't urgent but it would be nice to resolve the
> issue properly for M3 without driving Richard around the bend!
>
> Thanks,
> ../Randy
>
>
>>
>> Cheers,
>>
>> Richard
>>
>
>
Sorry for breaking the autobuilder.

I checked the Makefile and found that it's a bunch of `awd [expression] 
/etc/xxx | makedb -o /var/db/xxx.db -'.
I also checked the glibc repo and this Makefile is quite stable. So I 
think we can use a shell script to get rid of the make dependency. And 
it should not create maintenance problem.

I'll send out V2.
I've tested it locally and it should pass the oe-selftest.

Best Regards,
Chen Qi