Using users/groups from another recipe than the one creating them

Using users/groups from another recipe than the one creating them

[Peter Kjellerstedt]

Which assumption is correct: "a recipe A that depends on another 
recipe B can use users/groups that B creates" or "all recipes must 
create the users/groups they require themselves"?

The problem for us is that we have a lot of recipes that create 
users and groups, and subsequently a number of other related recipes 
that need to use those users and groups. 

If the first assumption is correct then the useradd.bbclass needs to 
be corrected so that it adds a dependency from do_install to 
base-passwd:do_populate_sysroot and 
base-passwd:do_populate_sysroot_setscene, because if either of those 
tasks execute they will overwrite /etc/passwd and /etc/group, 
effectively removing any users/groups that were created earlier...

On the other hand, if it is the second assumption that is correct, 
then there should be QA tests in place to make sure all recipes 
create the users/groups they use.

//Peter

Re: Using users/groups from another recipe than the one creating them

[Peter Kjellerstedt]

> -----Original Message-----
> From: openembedded-core-bounces@lists.openembedded.org
> [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf Of
> Peter Kjellerstedt
> Sent: den 19 maj 2014 10:15
> To: OE Core (openembedded-core@lists.openembedded.org)
> Subject: [OE-core] Using users/groups from another recipe than the one
> creating them
> 
> Which assumption is correct: "a recipe A that depends on another
> recipe B can use users/groups that B creates" or "all recipes must
> create the users/groups they require themselves"?
> 
> The problem for us is that we have a lot of recipes that create
> users and groups, and subsequently a number of other related recipes
> that need to use those users and groups.
> 
> If the first assumption is correct then the useradd.bbclass needs to
> be corrected so that it adds a dependency from do_install to
> base-passwd:do_populate_sysroot and
> base-passwd:do_populate_sysroot_setscene, because if either of those
> tasks execute they will overwrite /etc/passwd and /etc/group,
> effectively removing any users/groups that were created earlier...
> 
> On the other hand, if it is the second assumption that is correct,
> then there should be QA tests in place to make sure all recipes
> create the users/groups they use.
> 
> //Peter

*ping* 

Doesn't anyone know how users and groups are supposed to work?

//Peter

Re: Using users/groups from another recipe than the one creating them

[Peter Kjellerstedt]

> -----Original Message-----
> From: openembedded-core-bounces@lists.openembedded.org
> [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf Of
> Peter Kjellerstedt
> Sent: den 23 maj 2014 12:38
> To: OE Core (openembedded-core@lists.openembedded.org)
> Subject: Re: [OE-core] Using users/groups from another recipe than the
> one creating them
> 
> > -----Original Message-----
> > From: openembedded-core-bounces@lists.openembedded.org
> > [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf
> > Of Peter Kjellerstedt
> > Sent: den 19 maj 2014 10:15
> > To: OE Core (openembedded-core@lists.openembedded.org)
> > Subject: [OE-core] Using users/groups from another recipe than the
> > one creating them
> >
> > Which assumption is correct: "a recipe A that depends on another
> > recipe B can use users/groups that B creates" or "all recipes must
> > create the users/groups they require themselves"?
> >
> > The problem for us is that we have a lot of recipes that create
> > users and groups, and subsequently a number of other related recipes
> > that need to use those users and groups.
> >
> > If the first assumption is correct then the useradd.bbclass needs to
> > be corrected so that it adds a dependency from do_install to
> > base-passwd:do_populate_sysroot and
> > base-passwd:do_populate_sysroot_setscene, because if either of those
> > tasks execute they will overwrite /etc/passwd and /etc/group,
> > effectively removing any users/groups that were created earlier...
> >
> > On the other hand, if it is the second assumption that is correct,
> > then there should be QA tests in place to make sure all recipes
> > create the users/groups they use.
> >
> > //Peter
> 
> *ping*
> 
> Doesn't anyone know how users and groups are supposed to work?
> 
> //Peter

*ping again*

Well, this is somewhat discouraging. Three weeks and not a single 
response. Are we really the only ones that care about users and 
groups and how they are created? Doesn't anyone know which of my 
two assumptions above are correct?

Personally I would prefer that all recipes create the users and 
groups they require themselves as this keeps them more self 
contained. I have no idea how to write a QA test to verify this, 
but I assume it should be possible...

//Peter

Re: Using users/groups from another recipe than the one creating them

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 2869 bytes --]

On Mon, Jun 09, 2014 at 03:39:46PM +0200, Peter Kjellerstedt wrote:
> > -----Original Message-----
> > From: openembedded-core-bounces@lists.openembedded.org
> > [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf Of
> > Peter Kjellerstedt
> > Sent: den 23 maj 2014 12:38
> > To: OE Core (openembedded-core@lists.openembedded.org)
> > Subject: Re: [OE-core] Using users/groups from another recipe than the
> > one creating them
> > 
> > > -----Original Message-----
> > > From: openembedded-core-bounces@lists.openembedded.org
> > > [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf
> > > Of Peter Kjellerstedt
> > > Sent: den 19 maj 2014 10:15
> > > To: OE Core (openembedded-core@lists.openembedded.org)
> > > Subject: [OE-core] Using users/groups from another recipe than the
> > > one creating them
> > >
> > > Which assumption is correct: "a recipe A that depends on another
> > > recipe B can use users/groups that B creates" or "all recipes must
> > > create the users/groups they require themselves"?
> > >
> > > The problem for us is that we have a lot of recipes that create
> > > users and groups, and subsequently a number of other related recipes
> > > that need to use those users and groups.
> > >
> > > If the first assumption is correct then the useradd.bbclass needs to
> > > be corrected so that it adds a dependency from do_install to
> > > base-passwd:do_populate_sysroot and
> > > base-passwd:do_populate_sysroot_setscene, because if either of those
> > > tasks execute they will overwrite /etc/passwd and /etc/group,
> > > effectively removing any users/groups that were created earlier...
> > >
> > > On the other hand, if it is the second assumption that is correct,
> > > then there should be QA tests in place to make sure all recipes
> > > create the users/groups they use.
> > >
> > > //Peter
> > 
> > *ping*
> > 
> > Doesn't anyone know how users and groups are supposed to work?
> > 
> > //Peter
> 
> *ping again*
> 
> Well, this is somewhat discouraging. Three weeks and not a single 
> response. Are we really the only ones that care about users and 
> groups and how they are created? Doesn't anyone know which of my 
> two assumptions above are correct?
> 
> Personally I would prefer that all recipes create the users and 
> groups they require themselves as this keeps them more self 
> contained. I have no idea how to write a QA test to verify this, 
> but I assume it should be possible...

My experience from Dylan release is that only users/groups created in
base-passwd work reliably, with useradd.bbclass we were seeing random
files getting random user/group owners (comparing buildhistory report
files-in-image.txt from multiple builds which weren't using sstate).

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 188 bytes --]

Re: Using users/groups from another recipe than the one creating them

[Mark Hatle]

On 6/9/14, 8:39 AM, Peter Kjellerstedt wrote:
>> -----Original Message-----
>> From: openembedded-core-bounces@lists.openembedded.org
>> [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf Of
>> Peter Kjellerstedt
>> Sent: den 23 maj 2014 12:38
>> To: OE Core (openembedded-core@lists.openembedded.org)
>> Subject: Re: [OE-core] Using users/groups from another recipe than the
>> one creating them
>>
>>> -----Original Message-----
>>> From: openembedded-core-bounces@lists.openembedded.org
>>> [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf
>>> Of Peter Kjellerstedt
>>> Sent: den 19 maj 2014 10:15
>>> To: OE Core (openembedded-core@lists.openembedded.org)
>>> Subject: [OE-core] Using users/groups from another recipe than the
>>> one creating them
>>>
>>> Which assumption is correct: "a recipe A that depends on another
>>> recipe B can use users/groups that B creates" or "all recipes must
>>> create the users/groups they require themselves"?
>>>
>>> The problem for us is that we have a lot of recipes that create
>>> users and groups, and subsequently a number of other related recipes
>>> that need to use those users and groups.
>>>
>>> If the first assumption is correct then the useradd.bbclass needs to
>>> be corrected so that it adds a dependency from do_install to
>>> base-passwd:do_populate_sysroot and
>>> base-passwd:do_populate_sysroot_setscene, because if either of those
>>> tasks execute they will overwrite /etc/passwd and /etc/group,
>>> effectively removing any users/groups that were created earlier...
>>>
>>> On the other hand, if it is the second assumption that is correct,
>>> then there should be QA tests in place to make sure all recipes
>>> create the users/groups they use.
>>>
>>> //Peter
>>
>> *ping*
>>
>> Doesn't anyone know how users and groups are supposed to work?
>>
>> //Peter
>
> *ping again*

I never saw the original emails, either of them.

> Well, this is somewhat discouraging. Three weeks and not a single
> response. Are we really the only ones that care about users and
> groups and how they are created? Doesn't anyone know which of my
> two assumptions above are correct?
>
> Personally I would prefer that all recipes create the users and
> groups they require themselves as this keeps them more self
> contained. I have no idea how to write a QA test to verify this,
> but I assume it should be possible...

The rule is:

There is a limited number of base users and groups, if your package uses a 
user/group in that base set nothing else is needed.

If your package needs an additional user/group, then it must either:

*) Add the user/group itself.  Multiple packages are allowed to add the same 
users/groups, with the understanding that the options -must- be identical!

*) Must -require- (not recommend) a package that adds the group that they 
require.  It's a good idea in the recipe to comment that the 'require' adds the 
user/group as well.

This is pretty common where a single recipe provides multiple packages.  If a 
common package creates the user/group, then all of the other packages [that need 
that user/group] should then require the common package.


As far as the QA resources go.  There are actually two sides to the QA.  The 
first is in the do_install step, if the users/groups are being used there -- and 
don't exist -- a failure should be generated.  That exists just by the nature of 
the system build processes.

The second is that once a package has been generated (the package directory that 
is), it should be verified that all files are owned by in the base set of 
users/groups, the user/group has been added, or a dependency on the package that 
added them exist.  This is difficult to do though.  There is currently no 
tracking mechanism in the system to say which recipes added which users/groups 
to the system.  A mechanism similar to the sysroot file population would be 
required to capture the user and group changes and by which package(s).  This of 
course would also have to work with the sstate-cache mechanism.

The first step in implementing this would be to capture the user/group changes 
and record them in a database.  (again, I think similar to the populate sysroot 
file).  Then during the QA process, you can iterate over the files and find out 
where each user/group in use comes from.  If the provider is not in the 
'required' set of packages generate a QA -warning- (since this is new and 
unproven code).. We can upgrade it to an error once it's stable.

--Mark


> //Peter
>

Re: Using users/groups from another recipe than the one creating them

[Mark Hatle]

On 6/9/14, 8:52 AM, Martin Jansa wrote:
> On Mon, Jun 09, 2014 at 03:39:46PM +0200, Peter Kjellerstedt wrote:
>>> -----Original Message-----
>>> From: openembedded-core-bounces@lists.openembedded.org
>>> [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf Of
>>> Peter Kjellerstedt
>>> Sent: den 23 maj 2014 12:38
>>> To: OE Core (openembedded-core@lists.openembedded.org)
>>> Subject: Re: [OE-core] Using users/groups from another recipe than the
>>> one creating them
>>>
>>>> -----Original Message-----
>>>> From: openembedded-core-bounces@lists.openembedded.org
>>>> [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf
>>>> Of Peter Kjellerstedt
>>>> Sent: den 19 maj 2014 10:15
>>>> To: OE Core (openembedded-core@lists.openembedded.org)
>>>> Subject: [OE-core] Using users/groups from another recipe than the
>>>> one creating them
>>>>
>>>> Which assumption is correct: "a recipe A that depends on another
>>>> recipe B can use users/groups that B creates" or "all recipes must
>>>> create the users/groups they require themselves"?
>>>>
>>>> The problem for us is that we have a lot of recipes that create
>>>> users and groups, and subsequently a number of other related recipes
>>>> that need to use those users and groups.
>>>>
>>>> If the first assumption is correct then the useradd.bbclass needs to
>>>> be corrected so that it adds a dependency from do_install to
>>>> base-passwd:do_populate_sysroot and
>>>> base-passwd:do_populate_sysroot_setscene, because if either of those
>>>> tasks execute they will overwrite /etc/passwd and /etc/group,
>>>> effectively removing any users/groups that were created earlier...
>>>>
>>>> On the other hand, if it is the second assumption that is correct,
>>>> then there should be QA tests in place to make sure all recipes
>>>> create the users/groups they use.
>>>>
>>>> //Peter
>>>
>>> *ping*
>>>
>>> Doesn't anyone know how users and groups are supposed to work?
>>>
>>> //Peter
>>
>> *ping again*
>>
>> Well, this is somewhat discouraging. Three weeks and not a single
>> response. Are we really the only ones that care about users and
>> groups and how they are created? Doesn't anyone know which of my
>> two assumptions above are correct?
>>
>> Personally I would prefer that all recipes create the users and
>> groups they require themselves as this keeps them more self
>> contained. I have no idea how to write a QA test to verify this,
>> but I assume it should be possible...
>
> My experience from Dylan release is that only users/groups created in
> base-passwd work reliably, with useradd.bbclass we were seeing random
> files getting random user/group owners (comparing buildhistory report
> files-in-image.txt from multiple builds which weren't using sstate).
>

Which package management type?

The only issue I'm aware of is when the requires aren't in the correct order, a 
package can fall back and try to use the host-system's passwd/group entries 
instead of the sysroot version.  This can lead to cases that during the 
installation process the dynamic users/groups have not been generated due to 
missing dependencies.

For the RPM type, you get a message that says "user foo can not be found, using 
root".  I'm not sure what deb/ipk do in these cases.  (RPM always handles users 
and groups by name, never by gid/uid number.)

The whole users and groups system has been working very well in my systems based 
on 'dora'.  I haven't experienced issues with dylan, but I also haven't used it 
as extensively.

--Mark

Re: Using users/groups from another recipe than the one creating them

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 4412 bytes --]

On Mon, Jun 09, 2014 at 09:50:23AM -0500, Mark Hatle wrote:
> On 6/9/14, 8:52 AM, Martin Jansa wrote:
> > On Mon, Jun 09, 2014 at 03:39:46PM +0200, Peter Kjellerstedt wrote:
> >>> -----Original Message-----
> >>> From: openembedded-core-bounces@lists.openembedded.org
> >>> [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf Of
> >>> Peter Kjellerstedt
> >>> Sent: den 23 maj 2014 12:38
> >>> To: OE Core (openembedded-core@lists.openembedded.org)
> >>> Subject: Re: [OE-core] Using users/groups from another recipe than the
> >>> one creating them
> >>>
> >>>> -----Original Message-----
> >>>> From: openembedded-core-bounces@lists.openembedded.org
> >>>> [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf
> >>>> Of Peter Kjellerstedt
> >>>> Sent: den 19 maj 2014 10:15
> >>>> To: OE Core (openembedded-core@lists.openembedded.org)
> >>>> Subject: [OE-core] Using users/groups from another recipe than the
> >>>> one creating them
> >>>>
> >>>> Which assumption is correct: "a recipe A that depends on another
> >>>> recipe B can use users/groups that B creates" or "all recipes must
> >>>> create the users/groups they require themselves"?
> >>>>
> >>>> The problem for us is that we have a lot of recipes that create
> >>>> users and groups, and subsequently a number of other related recipes
> >>>> that need to use those users and groups.
> >>>>
> >>>> If the first assumption is correct then the useradd.bbclass needs to
> >>>> be corrected so that it adds a dependency from do_install to
> >>>> base-passwd:do_populate_sysroot and
> >>>> base-passwd:do_populate_sysroot_setscene, because if either of those
> >>>> tasks execute they will overwrite /etc/passwd and /etc/group,
> >>>> effectively removing any users/groups that were created earlier...
> >>>>
> >>>> On the other hand, if it is the second assumption that is correct,
> >>>> then there should be QA tests in place to make sure all recipes
> >>>> create the users/groups they use.
> >>>>
> >>>> //Peter
> >>>
> >>> *ping*
> >>>
> >>> Doesn't anyone know how users and groups are supposed to work?
> >>>
> >>> //Peter
> >>
> >> *ping again*
> >>
> >> Well, this is somewhat discouraging. Three weeks and not a single
> >> response. Are we really the only ones that care about users and
> >> groups and how they are created? Doesn't anyone know which of my
> >> two assumptions above are correct?
> >>
> >> Personally I would prefer that all recipes create the users and
> >> groups they require themselves as this keeps them more self
> >> contained. I have no idea how to write a QA test to verify this,
> >> but I assume it should be possible...
> >
> > My experience from Dylan release is that only users/groups created in
> > base-passwd work reliably, with useradd.bbclass we were seeing random
> > files getting random user/group owners (comparing buildhistory report
> > files-in-image.txt from multiple builds which weren't using sstate).
> >
> 
> Which package management type?

ipk

> The only issue I'm aware of is when the requires aren't in the correct order, a 
> package can fall back and try to use the host-system's passwd/group entries 
> instead of the sysroot version.  This can lead to cases that during the 
> installation process the dynamic users/groups have not been generated due to 
> missing dependencies.

Those random files were also the files which are usually owned by
root:root, so it wasn't like incorrectly using users/groups from host
system. Maybe something in pseudo db got broken, but after finding the
work around with creating all required users from base-passwd I didn't
try to debug it more.

> For the RPM type, you get a message that says "user foo can not be found, using 
> root".  I'm not sure what deb/ipk do in these cases.  (RPM always handles users 
> and groups by name, never by gid/uid number.)
> 
> The whole users and groups system has been working very well in my systems based 
> on 'dora'.  I haven't experienced issues with dylan, but I also haven't used it 
> as extensively.
> 
> --Mark
> -- 
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 188 bytes --]

Re: Using users/groups from another recipe than the one creating them

[Peter Kjellerstedt]

> -----Original Message-----
> From: openembedded-core-bounces@lists.openembedded.org
> [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf Of
> Mark Hatle
> Sent: den 9 juni 2014 16:47
> To: Peter Kjellerstedt; OE Core (openembedded-
> core@lists.openembedded.org)
> Subject: Re: [OE-core] Using users/groups from another recipe than the
> one creating them
> 
> On 6/9/14, 8:39 AM, Peter Kjellerstedt wrote:
> >> -----Original Message-----
> >> From: openembedded-core-bounces@lists.openembedded.org
> >> [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf
> >> Of Peter Kjellerstedt
> >> Sent: den 23 maj 2014 12:38
> >> To: OE Core (openembedded-core@lists.openembedded.org)
> >> Subject: Re: [OE-core] Using users/groups from another recipe than
> >> the one creating them
> >>
> >>> -----Original Message-----
> >>> From: openembedded-core-bounces@lists.openembedded.org
> >>> [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf
> >>> Of Peter Kjellerstedt
> >>> Sent: den 19 maj 2014 10:15
> >>> To: OE Core (openembedded-core@lists.openembedded.org)
> >>> Subject: [OE-core] Using users/groups from another recipe than the
> >>> one creating them
> >>>
> >>> Which assumption is correct: "a recipe A that depends on another
> >>> recipe B can use users/groups that B creates" or "all recipes must
> >>> create the users/groups they require themselves"?
> >>>
> >>> The problem for us is that we have a lot of recipes that create
> >>> users and groups, and subsequently a number of other related
> >>> recipes that need to use those users and groups.
> >>>
> >>> If the first assumption is correct then the useradd.bbclass needs
> >>> to be corrected so that it adds a dependency from do_install to
> >>> base-passwd:do_populate_sysroot and
> >>> base-passwd:do_populate_sysroot_setscene, because if either of
> >>> those tasks execute they will overwrite /etc/passwd and /etc/group,
> >>> effectively removing any users/groups that were created earlier...
> >>>
> >>> On the other hand, if it is the second assumption that is correct,
> >>> then there should be QA tests in place to make sure all recipes
> >>> create the users/groups they use.
> >>>
> >>> //Peter
> >>
> >> *ping*
> >>
> >> Doesn't anyone know how users and groups are supposed to work?
> >>
> >> //Peter
> >
> > *ping again*
> 
> I never saw the original emails, either of them.
> 
> > Well, this is somewhat discouraging. Three weeks and not a single
> > response. Are we really the only ones that care about users and
> > groups and how they are created? Doesn't anyone know which of my
> > two assumptions above are correct?
> >
> > Personally I would prefer that all recipes create the users and
> > groups they require themselves as this keeps them more self
> > contained. I have no idea how to write a QA test to verify this,
> > but I assume it should be possible...
> 
> The rule is:
> 
> There is a limited number of base users and groups, if your package
> uses a user/group in that base set nothing else is needed.
> 
> If your package needs an additional user/group, then it must either:
> 
> *) Add the user/group itself.  Multiple packages are allowed to add 
>    the same users/groups, with the understanding that the options 
>    -must- be identical!

I have actually been toying with an extension to the 
useradd.bbclass that allows one to configure the user and group 
options in one configuration file, and then just state in the 
recipes which users are needed. That way there is no risk of using 
different options for the same user if it is created from different 
recipes. Haven't finished it though. One thing I have not solved is 
how to handle requiring the configuration file from all layers as 
they may all need to add users (or what to do if multiple layers 
define the same user differently...)

> *) Must -require- (not recommend) a package that adds the group that
>    they require.  It's a good idea in the recipe to comment that the 
>    'require' adds the user/group as well.
> 
> This is pretty common where a single recipe provides multiple packages.
> If a common package creates the user/group, then all of the other 
> packages [that need that user/group] should then require the common 
> package.

Ok, then my first assumption was the correct one. In that case I 
will look into fixing the useradd.bbclass so that the missing 
dependency is added.

> As far as the QA resources go.  There are actually two sides to 
> the QA. The first is in the do_install step, if the users/groups 
> are being used there -- and don't exist -- a failure should be 
> generated.  That exists just by the nature of the system build 
> processes.
> 
> The second is that once a package has been generated (the package
> directory that is), it should be verified that all files are owned 
> by in the base set of users/groups, the user/group has been added, 
> or a dependency on the package that added them exist.  This is 
> difficult to do though.  There is currently no tracking mechanism 
> in the system to say which recipes added which users/groups to the 
> system.  A mechanism similar to the sysroot file population would 
> be required to capture the user and group changes and by which 
> package(s). This of course would also have to work with the 
> sstate-cache mechanism.
> 
> The first step in implementing this would be to capture the 
> user/group changes and record them in a database.  (again, I think 
> similar to the populate sysroot file).  Then during the QA process, 
> you can iterate over the files and find out where each user/group 
> in use comes from.  If the provider is not in the 'required' set of 
> packages generate a QA -warning- (since this is new and unproven 
> code).. We can upgrade it to an error once it's stable.

Hmm, ok. This all sound like something we want. However, my Python 
skills are probably not up to the task, unfortunately.

> --Mark
> 
> > //Peter

//Peter

Re: Using users/groups from another recipe than the one creating them

[Mark Hatle]

On 6/9/14, 11:47 AM, Peter Kjellerstedt wrote:
>> -----Original Message-----
>> From: openembedded-core-bounces@lists.openembedded.org
>> [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf Of
>> Mark Hatle
>> Sent: den 9 juni 2014 16:47
>> To: Peter Kjellerstedt; OE Core (openembedded-
>> core@lists.openembedded.org)
>> Subject: Re: [OE-core] Using users/groups from another recipe than the
>> one creating them
>>
>> On 6/9/14, 8:39 AM, Peter Kjellerstedt wrote:
>>>> -----Original Message-----
>>>> From: openembedded-core-bounces@lists.openembedded.org
>>>> [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf
>>>> Of Peter Kjellerstedt
>>>> Sent: den 23 maj 2014 12:38
>>>> To: OE Core (openembedded-core@lists.openembedded.org)
>>>> Subject: Re: [OE-core] Using users/groups from another recipe than
>>>> the one creating them
>>>>
>>>>> -----Original Message-----
>>>>> From: openembedded-core-bounces@lists.openembedded.org
>>>>> [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf
>>>>> Of Peter Kjellerstedt
>>>>> Sent: den 19 maj 2014 10:15
>>>>> To: OE Core (openembedded-core@lists.openembedded.org)
>>>>> Subject: [OE-core] Using users/groups from another recipe than the
>>>>> one creating them
>>>>>
>>>>> Which assumption is correct: "a recipe A that depends on another
>>>>> recipe B can use users/groups that B creates" or "all recipes must
>>>>> create the users/groups they require themselves"?
>>>>>
>>>>> The problem for us is that we have a lot of recipes that create
>>>>> users and groups, and subsequently a number of other related
>>>>> recipes that need to use those users and groups.
>>>>>
>>>>> If the first assumption is correct then the useradd.bbclass needs
>>>>> to be corrected so that it adds a dependency from do_install to
>>>>> base-passwd:do_populate_sysroot and
>>>>> base-passwd:do_populate_sysroot_setscene, because if either of
>>>>> those tasks execute they will overwrite /etc/passwd and /etc/group,
>>>>> effectively removing any users/groups that were created earlier...
>>>>>
>>>>> On the other hand, if it is the second assumption that is correct,
>>>>> then there should be QA tests in place to make sure all recipes
>>>>> create the users/groups they use.
>>>>>
>>>>> //Peter
>>>>
>>>> *ping*
>>>>
>>>> Doesn't anyone know how users and groups are supposed to work?
>>>>
>>>> //Peter
>>>
>>> *ping again*
>>
>> I never saw the original emails, either of them.
>>
>>> Well, this is somewhat discouraging. Three weeks and not a single
>>> response. Are we really the only ones that care about users and
>>> groups and how they are created? Doesn't anyone know which of my
>>> two assumptions above are correct?
>>>
>>> Personally I would prefer that all recipes create the users and
>>> groups they require themselves as this keeps them more self
>>> contained. I have no idea how to write a QA test to verify this,
>>> but I assume it should be possible...
>>
>> The rule is:
>>
>> There is a limited number of base users and groups, if your package
>> uses a user/group in that base set nothing else is needed.
>>
>> If your package needs an additional user/group, then it must either:
>>
>> *) Add the user/group itself.  Multiple packages are allowed to add
>>     the same users/groups, with the understanding that the options
>>     -must- be identical!
>
> I have actually been toying with an extension to the
> useradd.bbclass that allows one to configure the user and group
> options in one configuration file, and then just state in the
> recipes which users are needed. That way there is no risk of using
> different options for the same user if it is created from different
> recipes. Haven't finished it though. One thing I have not solved is
> how to handle requiring the configuration file from all layers as
> they may all need to add users (or what to do if multiple layers
> define the same user differently...)

If you implement this, you should follow the example of the 
"useradd-staticids.bbclass".  This is something that a user can optionally add 
into their environment to better define how things are configured.

I still contend that these items belong in the recipes and that it's a recipe 
bug if they're not done properly.  We can help automate this, but I don't know 
if that is yet a reasonable general solution.

The staticids class collects together "USERADD_UID_TABLES" values, and then 
rewrites the USERADD_PARAMS/GROUPADD_PARAMS to match whatever the table(s) say. 
  This is the same mechanism that we use for the filesystem table to synchronize 
standard owners/groups/perms in various recipes.

I would still expect the recipe to say I need the following users/groups.. and 
then to pick up the data from the tables if necessary.

>> *) Must -require- (not recommend) a package that adds the group that
>>     they require.  It's a good idea in the recipe to comment that the
>>     'require' adds the user/group as well.
>>
>> This is pretty common where a single recipe provides multiple packages.
>> If a common package creates the user/group, then all of the other
>> packages [that need that user/group] should then require the common
>> package.
>
> Ok, then my first assumption was the correct one. In that case I
> will look into fixing the useradd.bbclass so that the missing
> dependency is added.

useradd isn't the right place for the dependency, it really is a recipe level 
dependency.

>> As far as the QA resources go.  There are actually two sides to
>> the QA. The first is in the do_install step, if the users/groups
>> are being used there -- and don't exist -- a failure should be
>> generated.  That exists just by the nature of the system build
>> processes.
>>
>> The second is that once a package has been generated (the package
>> directory that is), it should be verified that all files are owned
>> by in the base set of users/groups, the user/group has been added,
>> or a dependency on the package that added them exist.  This is
>> difficult to do though.  There is currently no tracking mechanism
>> in the system to say which recipes added which users/groups to the
>> system.  A mechanism similar to the sysroot file population would
>> be required to capture the user and group changes and by which
>> package(s). This of course would also have to work with the
>> sstate-cache mechanism.
>>
>> The first step in implementing this would be to capture the
>> user/group changes and record them in a database.  (again, I think
>> similar to the populate sysroot file).  Then during the QA process,
>> you can iterate over the files and find out where each user/group
>> in use comes from.  If the provider is not in the 'required' set of
>> packages generate a QA -warning- (since this is new and unproven
>> code).. We can upgrade it to an error once it's stable.
>
> Hmm, ok. This all sound like something we want. However, my Python
> skills are probably not up to the task, unfortunately.

You should look into how the populate_sysroot function writes out the list of 
files installed into the sysroot.. and see if you can do something similar for 
the packages.  It's python, but I don't think it's that heavy of a python code.

--Mark

>> --Mark
>>
>>> //Peter
>
> //Peter
>

Re: Using users/groups from another recipe than the one creating them

[Peter Kjellerstedt]

> -----Original Message-----
> From: openembedded-core-bounces@lists.openembedded.org
> [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf Of
> Mark Hatle
> Sent: den 9 juni 2014 19:03
> To: Peter Kjellerstedt; OE Core (openembedded-
> core@lists.openembedded.org)
> Subject: Re: [OE-core] Using users/groups from another recipe than the
> one creating them
> 
> On 6/9/14, 11:47 AM, Peter Kjellerstedt wrote:
> >> -----Original Message-----
> >> From: openembedded-core-bounces@lists.openembedded.org
> >> [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf
> >> Of Mark Hatle
> >> Sent: den 9 juni 2014 16:47
> >> To: Peter Kjellerstedt; OE Core (openembedded-
> >> core@lists.openembedded.org)
> >> Subject: Re: [OE-core] Using users/groups from another recipe than
> >> the one creating them
> >>
> >> On 6/9/14, 8:39 AM, Peter Kjellerstedt wrote:
> >>>> -----Original Message-----
> >>>> From: openembedded-core-bounces@lists.openembedded.org
> >>>> [mailto:openembedded-core-bounces@lists.openembedded.org] On
> >>>> Behalf Of Peter Kjellerstedt
> >>>> Sent: den 23 maj 2014 12:38
> >>>> To: OE Core (openembedded-core@lists.openembedded.org)
> >>>> Subject: Re: [OE-core] Using users/groups from another recipe than
> >>>> the one creating them
> >>>>
> >>>>> -----Original Message-----
> >>>>> From: openembedded-core-bounces@lists.openembedded.org
> >>>>> [mailto:openembedded-core-bounces@lists.openembedded.org] On
> >>>>> Behalf Of Peter Kjellerstedt
> >>>>> Sent: den 19 maj 2014 10:15
> >>>>> To: OE Core (openembedded-core@lists.openembedded.org)
> >>>>> Subject: [OE-core] Using users/groups from another recipe than
> >>>>> the one creating them
> >>>>>
> >>>>> Which assumption is correct: "a recipe A that depends on another
> >>>>> recipe B can use users/groups that B creates" or "all recipes
> >>>>> must create the users/groups they require themselves"?
> >>>>>
> >>>>> The problem for us is that we have a lot of recipes that create
> >>>>> users and groups, and subsequently a number of other related
> >>>>> recipes that need to use those users and groups.
> >>>>>
> >>>>> If the first assumption is correct then the useradd.bbclass needs
> >>>>> to be corrected so that it adds a dependency from do_install to
> >>>>> base-passwd:do_populate_sysroot and
> >>>>> base-passwd:do_populate_sysroot_setscene, because if either of
> >>>>> those tasks execute they will overwrite /etc/passwd and
> >>>>> /etc/group, effectively removing any users/groups that were 
> >>>>> created earlier...
> >>>>>
> >>>>> On the other hand, if it is the second assumption that is correct,
> >>>>> then there should be QA tests in place to make sure all recipes
> >>>>> create the users/groups they use.
> >>>>>
> >>>>> //Peter
> >>>>
> >>>> *ping*
> >>>>
> >>>> Doesn't anyone know how users and groups are supposed to work?
> >>>>
> >>>> //Peter
> >>>
> >>> *ping again*
> >>
> >> I never saw the original emails, either of them.
> >>
> >>> Well, this is somewhat discouraging. Three weeks and not a single
> >>> response. Are we really the only ones that care about users and
> >>> groups and how they are created? Doesn't anyone know which of my
> >>> two assumptions above are correct?
> >>>
> >>> Personally I would prefer that all recipes create the users and
> >>> groups they require themselves as this keeps them more self
> >>> contained. I have no idea how to write a QA test to verify this,
> >>> but I assume it should be possible...
> >>
> >> The rule is:
> >>
> >> There is a limited number of base users and groups, if your package
> >> uses a user/group in that base set nothing else is needed.
> >>
> >> If your package needs an additional user/group, then it must either:
> >>
> >> *) Add the user/group itself.  Multiple packages are allowed to add
> >>     the same users/groups, with the understanding that the options
> >>     -must- be identical!
> >
> > I have actually been toying with an extension to the
> > useradd.bbclass that allows one to configure the user and group
> > options in one configuration file, and then just state in the
> > recipes which users are needed. That way there is no risk of using
> > different options for the same user if it is created from different
> > recipes. Haven't finished it though. One thing I have not solved is
> > how to handle requiring the configuration file from all layers as
> > they may all need to add users (or what to do if multiple layers
> > define the same user differently...)
> 
> If you implement this, you should follow the example of the
> "useradd-staticids.bbclass".  This is something that a user can
> optionally add into their environment to better define how things 
> are configured.
> 
> I still contend that these items belong in the recipes and that it's 
> a recipe bug if they're not done properly.  We can help automate this, 
> but I don't know if that is yet a reasonable general solution.
> 
> The staticids class collects together "USERADD_UID_TABLES" values, 
> and then rewrites the USERADD_PARAMS/GROUPADD_PARAMS to match whatever 
> the table(s) say.  This is the same mechanism that we use for the 
> filesystem table to synchronize standard owners/groups/perms in various 
> recipes.
> 
> I would still expect the recipe to say I need the following
> users/groups.. and then to pick up the data from the tables if necessary.

Thanks for the tips. I will definitely look at what 
useradd-staticids.bbclass does if I decide to continue to work 
on this.

> >> *) Must -require- (not recommend) a package that adds the group that
> >>     they require.  It's a good idea in the recipe to comment that
> >>     the 'require' adds the user/group as well.
> >>
> >> This is pretty common where a single recipe provides multiple
> >> packages.  If a common package creates the user/group, then all 
> >> of the other packages [that need that user/group] should then 
> >> require the common package.
> >
> > Ok, then my first assumption was the correct one. In that case I
> > will look into fixing the useradd.bbclass so that the missing
> > dependency is added.
> 
> useradd isn't the right place for the dependency, it really is a 
> recipe level dependency.

useradd.bbclass adds a dependency on base-passwd for each recipe 
that inherits it. In addition it does this:

USERADDSETSCENEDEPS = "${MLPREFIX}base-passwd:do_populate_sysroot_setscene shadow-native:do_populate_sysroot_setscene ${MLPREFIX}shadow-sysroot:do_populate_sysroot_setscene"
USERADDSETSCENEDEPS_class-cross = ""
USERADDSETSCENEDEPS_class-native = ""
USERADDSETSCENEDEPS_class-nativesdk = ""
do_package_setscene[depends] += "${USERADDSETSCENEDEPS}"
do_populate_sysroot_setscene[depends] += "${USERADDSETSCENEDEPS}"

However, this does not seem to trigger anything. If I first do 
"bitbake base-passwd -c clean && bitbake base-passwd" (which 
will wipe /etc/passwd and /etc/groups in the sysroot) and then 
try to rebuild a recipe that creates a user, e.g. "bitbake dbus", 
then noting happens and the messagebus user is not added to 
/etc/passwd. Consequently, any other recipes that rely on dbus 
to create the messagebus user will then fail if one tries to 
rebuild them.

I tried to add the following to see if any of it did any 
difference, but it didn't:

USERADDDEPS = "${MLPREFIX}base-passwd:do_populate_sysroot ${MLPREFIX}base-passwd:do_populate_sysroot_setscene"
USERADDDEPS_class-cross = ""
USERADDDEPS_class-native = ""
USERADDDEPS_class-nativesdk = ""
do_install[depends] += "${USERADDDEPS}"
do_package[depends] += "${USERADDDEPS}"
do_populate_sysroot[depends] += "${USERADDDEPS}"

My guess is that since the signature of the
base-passwd:do_populate_sysroot_setscene task has not changed it 
will not trigger the dependent tasks to run even if it has changed 
the sysroot...

Since it does not seem possible to trigger the recipes that 
install users to redo it after base-passwd has trashed /etc/passwd, 
maybe the solution is to use the "database" you describe below. 
That way, when base-passwd does its job, it would only affect the 
users it has installed itself and leave the others alone.

> >> As far as the QA resources go.  There are actually two sides to
> >> the QA. The first is in the do_install step, if the users/groups
> >> are being used there -- and don't exist -- a failure should be
> >> generated.  That exists just by the nature of the system build
> >> processes.
> >>
> >> The second is that once a package has been generated (the package
> >> directory that is), it should be verified that all files are owned
> >> by in the base set of users/groups, the user/group has been added,
> >> or a dependency on the package that added them exist.  This is
> >> difficult to do though.  There is currently no tracking mechanism
> >> in the system to say which recipes added which users/groups to the
> >> system.  A mechanism similar to the sysroot file population would
> >> be required to capture the user and group changes and by which
> >> package(s). This of course would also have to work with the
> >> sstate-cache mechanism.
> >>
> >> The first step in implementing this would be to capture the
> >> user/group changes and record them in a database.  (again, I think
> >> similar to the populate sysroot file).  Then during the QA process,
> >> you can iterate over the files and find out where each user/group
> >> in use comes from.  If the provider is not in the 'required' set of
> >> packages generate a QA -warning- (since this is new and unproven
> >> code).. We can upgrade it to an error once it's stable.
> >
> > Hmm, ok. This all sound like something we want. However, my Python
> > skills are probably not up to the task, unfortunately.
> 
> You should look into how the populate_sysroot function writes out 
> the list of files installed into the sysroot.. and see if you can 
> do something similar for the packages.  It's python, but I don't 
> think it's that heavy of a python code.
> 
> --Mark

I will have to save the database idea to after I am back from my 
summer vacation in August. I will not have time to do any work 
on it before then anyway.

//Peter