[RFC PATCH 0/2] RFC: Implement deterministic uid/gid

[RFC PATCH 0/2] RFC: Implement deterministic uid/gid

[Mark Hatle]

The following series implements the deterministic uid/gid setting for a
distribution.  Currently when a filesystem is generated the uid/gid values
are generally set at install time, so the install order determines what
the actual uid/gid values become.  In order to create a deterministic uid/gid
set, that still dynamically constructs the passwd/group file, we add an
option to read a special passwd/group file to allow the system to determine
the values.

It uses the existing parameters, and the values from the special passwd/group
files to reconstruct the parameter set to ensure these items are fully
defined with static values.

The first patch (01/02) is generally applicable.  It fixes a real bug in
the way the user/group adds occur today within the system.

Patch 02/02 implements the new functionality.


The following changes since commit 8505f0fa48cc79d51616b923c6e2c778c4b46a44:

  libmatchbox: use PACKAGECONFIG (2013-12-10 14:13:01 +0000)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib mhatle/uidgid
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=mhatle/uidgid

Mark Hatle (2):
  useradd.bbclass: Fix build time install issues
  useradd.bbclass: Add ability to select a static uid/gid automatically

 meta/classes/useradd.bbclass         | 247 ++++++++++++++++++++++++++++++++++-
 meta/conf/local.conf.sample.extended |  24 ++++
 2 files changed, 265 insertions(+), 6 deletions(-)

-- 
1.8.1.2.545.g2f19ada

[RFC PATCH 1/2] useradd.bbclass: Fix build time install issues

[Mark Hatle]

When the system attempts to populate the sysroot's passwd/group files, it
does so in a single block.  However, with the way it was previously
implemented, the system would always run through the code necessary to
populate the sysroot, even in the case of target packages.  This had
the side effect that a cross-installed filesystem may not match a
target installed filesystem.

The code was slightly reorganized to ensure that the cross/target installed
pre-install script behavior is the same.  It also moves the block that
configures the sysroot parameters to the sysroot specific section of
the code.

Also some minor validation was occuring even on nativesdk packages.
Nativesdk packages should be skipped when processing useradd ops.

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
---
 meta/classes/useradd.bbclass | 21 +++++++++++++++------
 1 file changed, 15 insertions(+), 6 deletions(-)

diff --git a/meta/classes/useradd.bbclass b/meta/classes/useradd.bbclass
index a850e9d..a60ab09 100644
--- a/meta/classes/useradd.bbclass
+++ b/meta/classes/useradd.bbclass
@@ -24,12 +24,11 @@ if test "x$D" != "x"; then
 	# Installing into a sysroot
 	SYSROOT="$D"
 	OPT="--root $D"
+fi
 
-	# Add groups and users defined for all recipe packages
-	GROUPADD_PARAM="${@get_all_cmd_params(d, 'groupadd')}"
-	USERADD_PARAM="${@get_all_cmd_params(d, 'useradd')}"
-	GROUPMEMS_PARAM="${@get_all_cmd_params(d, 'groupmems')}"
-else
+# If we're not doing a special SSTATE/SYSROOT install
+# then set the values, otherwise use the environment
+if test "x$UA_SYSROOT" = "x"; then
 	# Installing onto a target
 	# Add groups and users defined only for this package
 	GROUPADD_PARAM="${GROUPADD_PARAM}"
@@ -97,6 +96,15 @@ useradd_sysroot () {
 	# Explicitly set $D since it isn't set to anything
 	# before do_install
 	D=${STAGING_DIR_TARGET}
+
+	# Add groups and users defined for all recipe packages
+	GROUPADD_PARAM="${@get_all_cmd_params(d, 'groupadd')}"
+	USERADD_PARAM="${@get_all_cmd_params(d, 'useradd')}"
+	GROUPMEMS_PARAM="${@get_all_cmd_params(d, 'groupmems')}"
+
+	# Tell the system to use the environment vars
+	UA_SYSROOT=1
+
 	useradd_preinst
 }
 
@@ -137,7 +145,8 @@ def update_useradd_after_parse(d):
             bb.fatal("%s inherits useradd but doesn't set USERADD_PARAM, GROUPADD_PARAM or GROUPMEMS_PARAM for package %s" % (d.getVar('FILE'), pkg))
 
 python __anonymous() {
-    update_useradd_after_parse(d)
+    if not bb.data.inherits_class('nativesdk', d):
+        update_useradd_after_parse(d)
 }
 
 # Return a single [GROUP|USER]ADD_PARAM formatted string which includes the
-- 
1.8.1.2.545.g2f19ada

[RFC PATCH 2/2] useradd.bbclass: Add ability to select a static uid/gid automatically

[Mark Hatle]

[YOCTO #5436]

Automatic selection of static uid/gid is needed for a dynamically generated
passwd and group file to have a deterministic outcome.

When a package is installed and instructs the system to add a new user or
group, unless it selects a static uid/gid value, the next available uid/gid
will be used.  The order in which packages are installed is dynamically
computed, and may change from one installation to the next.  This results
in a non-deterministic set of uid/gid values.

Enabling USERADD_REWRITE_PARAMS, and providing a preconfigured passwd/group
file will allow the continued dynamic generation of the passwd/group file
on the target, but ensure a deterministic outcome.  (Dynamic generation is
desired so that user and groups that have no corresponding functionality
are not present within the final system image.)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
---
 meta/classes/useradd.bbclass         | 226 +++++++++++++++++++++++++++++++++++
 meta/conf/local.conf.sample.extended |  24 ++++
 2 files changed, 250 insertions(+)

diff --git a/meta/classes/useradd.bbclass b/meta/classes/useradd.bbclass
index a60ab09..7f2c6e8 100644
--- a/meta/classes/useradd.bbclass
+++ b/meta/classes/useradd.bbclass
@@ -144,9 +144,235 @@ def update_useradd_after_parse(d):
         if not d.getVar('USERADD_PARAM_%s' % pkg, True) and not d.getVar('GROUPADD_PARAM_%s' % pkg, True) and not d.getVar('GROUPMEMS_PARAM_%s' % pkg, True):
             bb.fatal("%s inherits useradd but doesn't set USERADD_PARAM, GROUPADD_PARAM or GROUPMEMS_PARAM for package %s" % (d.getVar('FILE'), pkg))
 
+# In order to support a deterministic set of 'dynamic' users/groups,
+# we need a function to reformat the params based on a static file
+def update_useradd_static_config(d):
+    import argparse
+    import re
+
+    class myArgumentParser( argparse.ArgumentParser ):
+        def _print_message(self, message, file=None):
+            bb.warn("%s - %s: %s" % (d.getVar('PN', True), pkg, message))
+
+        # This should never be called...
+        def exit(self, status=0, message=None):
+            message = message or ("%s - %s: useradd.bbclass: Argument parsing exited" % (d.getVar('PN', True), pkg))
+            error(message)
+
+        def error(self, message):
+            raise bb.build.FuncFailed(message)
+
+    # We parse and rewrite the useradd components
+    def rewrite_useradd(params):
+        # The following comes from --help on useradd from shadow
+        parser = myArgumentParser(prog='useradd')
+        parser.add_argument("-b", "--base-dir", metavar="BASE_DIR", help="base directory for the home directory of the new account")
+        parser.add_argument("-c", "--comment", metavar="COMMENT", help="GECOS field of the new account")
+        parser.add_argument("-d", "--home-dir", metavar="HOME_DIR", help="home directory of the new account")
+        parser.add_argument("-D", "--defaults", help="print or change default useradd configuration", action="store_true")
+        parser.add_argument("-e", "--expiredate", metavar="EXPIRE_DATE", help="expiration date of the new account")
+        parser.add_argument("-f", "--inactive", metavar="INACTIVE", help="password inactivity period of the new account")
+        parser.add_argument("-g", "--gid", metavar="GROUP", help="name or ID of the primary group of the new account")
+        parser.add_argument("-G", "--groups", metavar="GROUPS", help="list of supplementary groups of the new account")
+        parser.add_argument("-k", "--skel", metavar="SKEL_DIR", help="use this alternative skeleton directory")
+        parser.add_argument("-K", "--key", metavar="KEY=VALUE", help="override /etc/login.defs defaults")
+        parser.add_argument("-l", "--no-log-init", help="do not add the user to the lastlog and faillog databases", action="store_true")
+        parser.add_argument("-m", "--create-home", help="create the user's home directory", action="store_true")
+        parser.add_argument("-M", "--no-create-home", help="do not create the user's home directory", action="store_true")
+        parser.add_argument("-N", "--no-user-group", help="do not create a group with the same name as the user", action="store_true")
+        parser.add_argument("-o", "--non-unique", help="allow to create users with duplicate (non-unique UID)", action="store_true")
+        parser.add_argument("-p", "--password", metavar="PASSWORD", help="encrypted password of the new account")
+        parser.add_argument("-R", "--root", metavar="CHROOT_DIR", help="directory to chroot into")
+        parser.add_argument("-r", "--system", help="create a system account", action="store_true")
+        parser.add_argument("-s", "--shell", metavar="SHELL", help="login shell of the new account")
+        parser.add_argument("-u", "--uid", metavar="UID", help="user ID of the new account")
+        parser.add_argument("-U", "--user-group", help="create a group with the same name as the user", action="store_true")
+        parser.add_argument("LOGIN", help="Login name of the new user")
+
+        # Return a list of configuration files based on either the default
+        # files/passwd or the contents of USERADD_UID_TABLES
+        # paths are resulved via BBPATH
+        def get_passwd_list(d):
+            str = ""
+            bbpath = d.getVar('BBPATH', True)
+            passwd_tables = d.getVar('USERADD_UID_TABLES', True)
+            if not passwd_tables:
+                passwd_tables = 'files/passwd'
+            for conf_file in passwd_tables.split():
+                str += " %s" % bb.utils.which(bbpath, conf_file)
+            return str
+
+        newparams = []
+        for param in re.split('''[ ]*;[ ]*(?=(?:[^'"]|'[^']*'|"[^"]*")*$)''', params):
+            param=param.strip()
+            try:
+                uaargs = parser.parse_args(re.split('''[ ]*(?=(?:[^'"]|'[^']*'|"[^"]*")*$)''', param))
+            except:
+                raise bb.build.FuncFailed("%s: Unable to parse arguments for USERADD_PARAM_%s: %s" % (d.getVar('PN', True), pkg, param))
+
+            # files/passwd or the contents of USERADD_UID_TABLES
+            # Use the standard passwd layout:
+            #  username:password:user_id:group_id:comment:home_directory:login_shell
+            # (we want to process in reverse order, as 'last found' in the list wins)
+            for conf in get_passwd_list(d).split()[::-1]:
+                if os.path.exists(conf):
+                    f = open(conf, "r")
+                    for line in f:
+                        if line.startswith('#'):
+                            continue
+                        field = line.rstrip().split(":")
+                        if field[0] == uaargs.LOGIN:
+                            if uaargs.uid and (uaargs.uid != field[2]):
+                               bb.warn("Changing username %s's uid from (%s) to (%s), verify configuration files!" % (uaargs.LOGIN, uaargs.uid, field[2]))
+                            uaargs.uid = field[2]
+
+                            # By default the system has creation of the matching groups enabled
+                            # So if explicit user-group creation is on, an implicit group is specified
+                            # or a group -name- is specified, we need to pick the number and create 
+                            # the group using the addgroup...
+                            if uaargs.user_group or not uaargs.gid or not uaargs.gid.isdigit():
+                               # Set the implicit case
+                               if not uaargs.gid:
+                                   uaargs.gid = uaargs.LOGIN
+                               bb.debug(2, "Adding group %s  gid (%s)!" % (uaargs.gid, field[3]))
+                               uaargs.user_group = False
+                               groupadd = d.getVar("GROUPADD_PARAM_%s" % pkg, True)
+                               newgroup = "-g %s %s" % (field[3], uaargs.gid)
+                               if groupadd:
+                                   d.setVar("GROUPADD_PARAM_%s" % pkg, "%s ; %s" % (groupadd, newgroup))
+                               else:
+                                   d.setVar("GROUPADD_PARAM_%s" % pkg, newgroup)
+
+                            if uaargs.gid and uaargs.gid.isdigit() and (uaargs.gid != field[3]):
+                               bb.warn("Changing groupname %s's gid from (%s) to (%s), verify configuration files!" % (uaargs.LOGIN, uaargs.gid, field[3]))
+                            uaargs.gid = field[3]
+
+                            uaargs.comment = [uaargs.comment, "'%s'" % field[4]][field[4] != ""]
+                            uaargs.home_dir = field[5]
+                            uaargs.shell = field[6]
+                            break
+
+            # Should be an error if a specific option is set...
+            if d.getVar('USERADD_ERROR_DYNAMIC', True) == '1' and (not uaargs.uid or not uaargs.gid):
+                bb.error("%s - %s: Username %s does not have a static uid/gid defined." % (d.getVar('PN', True), pkg, uaargs.LOGIN))
+
+            # Reconstruct the args...
+            newparam  = ['', ' --defaults'][uaargs.defaults]
+            newparam += ['', ' --base-dir %s' % uaargs.base_dir][uaargs.base_dir != None]
+            newparam += ['', ' --comment %s' % uaargs.comment][uaargs.comment != None]
+            newparam += ['', ' --home-dir %s' % uaargs.home_dir][uaargs.home_dir != None]
+            newparam += ['', ' --expiredata %s' % uaargs.expiredate][uaargs.expiredate != None]
+            newparam += ['', ' --inactive %s' % uaargs.inactive][uaargs.inactive != None]
+            newparam += ['', ' --gid %s' % uaargs.gid][uaargs.gid != None]
+            newparam += ['', ' --groups %s' % uaargs.groups][uaargs.groups != None]
+            newparam += ['', ' --skel %s' % uaargs.skel][uaargs.skel != None]
+            newparam += ['', ' --key %s' % uaargs.key][uaargs.key != None]
+            newparam += ['', ' --no-log-init'][uaargs.no_log_init]
+            newparam += ['', ' --create-home'][uaargs.create_home]
+            newparam += ['', ' --no-create-home'][uaargs.no_create_home]
+            newparam += ['', ' --no-user-group'][uaargs.no_user_group]
+            newparam += ['', ' --non-unique'][uaargs.non_unique]
+            newparam += ['', ' --password %s' % uaargs.password][uaargs.password != None]
+            newparam += ['', ' --root %s' % uaargs.root][uaargs.root != None]
+            newparam += ['', ' --system'][uaargs.system]
+            newparam += ['', ' --shell %s' % uaargs.shell][uaargs.shell != None]
+            newparam += ['', ' --uid %s' % uaargs.uid][uaargs.uid != None]
+            newparam += ['', ' --user-group'][uaargs.user_group]
+            newparam += ' %s' % uaargs.LOGIN
+
+            newparams.append(newparam)
+
+        return " ;".join(newparams).strip()
+
+    # We parse and rewrite the groupadd components
+    def rewrite_groupadd(params):
+        # The following comes from --help on groupadd from shadow
+        parser = myArgumentParser(prog='groupadd')
+        parser.add_argument("-f", "--force", help="exit successfully if the group already exists, and cancel -g if the GID is already used", action="store_true")
+        parser.add_argument("-g", "--gid", metavar="GID", help="use GID for the new group")
+        parser.add_argument("-K", "--key", metavar="KEY=VALUE", help="override /etc/login.defs defaults")
+        parser.add_argument("-o", "--non-unique", help="allow to create groups with duplicate (non-unique) GID", action="store_true")
+        parser.add_argument("-p", "--password", metavar="PASSWORD", help="use this encrypted password for the new group")
+        parser.add_argument("-R", "--root", metavar="CHROOT_DIR", help="directory to chroot into")
+        parser.add_argument("-r", "--system", help="create a system account", action="store_true")
+        parser.add_argument("GROUP", help="Group name of the new group")
+
+        # Return a list of configuration files based on either the default
+        # files/group or the contents of USERADD_GID_TABLES
+        # paths are resulved via BBPATH
+        def get_group_list(d):
+            str = ""
+            bbpath = d.getVar('BBPATH', True)
+            group_tables = d.getVar('USERADD_GID_TABLES', True)
+            if not group_tables:
+                group_tables = 'files/group'
+            for conf_file in group_tables.split():
+                str += " %s" % bb.utils.which(bbpath, conf_file)
+            return str
+
+        newparams = []
+        for param in re.split('''[ ]*;[ ]*(?=(?:[^'"]|'[^']*'|"[^"]*")*$)''', params):
+            param=param.strip()
+            try:
+                gaargs = parser.parse_args(re.split('''[ ]*(?=(?:[^'"]|'[^']*'|"[^"]*")*$)''', param))
+            except:
+                raise bb.build.FuncFailed("%s: Unable to parse arguments for GROUPADD_PARAM_%s: %s" % (d.getVar('PN', True), pkg, param))
+
+            # Need to iterate over layers and open the right file(s)
+            # Use the standard group layout:
+            #  groupname:password:group_id:group_members
+            for conf in get_group_list(d).split()[::-1]:
+                if os.path.exists(conf):
+                    f = open(conf, "r")
+                    for line in f:
+                        if line.startswith('#'):
+                            continue
+                        field = line.rstrip().split(":")
+                        if field[0] == gaargs.GROUP:
+                            if gaargs.gid and (gaargs.gid != field[2]):
+                                bb.warn("Changing groupname %s's gid from (%s) to (%s), verify configuration files!" % (gaargs.GROUP, gaargs.gid, field[2]))
+                            gaargs.gid = field[2]
+                            break
+
+            if d.getVar('USERADD_ERROR_DYNAMIC', True) == '1' and not gaargs.gid:
+                bb.error("%s - %s: Groupname %s does not have a static gid defined." % (d.getVar('PN', True), pkg, gaargs.GROUP))
+
+            # Reconstruct the args...
+            newparam  = ['', ' --force'][gaargs.force]
+            newparam += ['', ' --gid %s' % gaargs.gid][gaargs.gid != None]
+            newparam += ['', ' --key %s' % gaargs.key][gaargs.key != None]
+            newparam += ['', ' --non-unique'][gaargs.non_unique]
+            newparam += ['', ' --password %s' % gaargs.password][gaargs.password != None]
+            newparam += ['', ' --root %s' % gaargs.root][gaargs.root != None]
+            newparam += ['', ' --system'][gaargs.system]
+            newparam += ' %s' % gaargs.GROUP
+
+            newparams.append(newparam)
+
+        return " ;".join(newparams).strip()
+
+    # Load and process the users and groups, rewriting the adduser/addgroup params
+    useradd_packages = d.getVar('USERADD_PACKAGES', True)
+
+    for pkg in useradd_packages.split():
+        # Groupmems doesn't have anything we might want to change, so simply validating
+        # is a bit of a waste -- only process useradd/groupadd
+        useradd_param = d.getVar('USERADD_PARAM_%s' % pkg, True)
+        if useradd_param:
+            d.setVar('USERADD_PARAM_%s' % pkg, rewrite_useradd(useradd_param))
+
+        groupadd_param = d.getVar('GROUPADD_PARAM_%s' % pkg, True)
+        if groupadd_param:
+            d.setVar('GROUPADD_PARAM_%s' % pkg, rewrite_groupadd(groupadd_param))
+
+
+
 python __anonymous() {
     if not bb.data.inherits_class('nativesdk', d):
         update_useradd_after_parse(d)
+
+        if d.getVar('USERADD_REWRITE_PARAMS', True) == '1':
+            update_useradd_static_config(d)
 }
 
 # Return a single [GROUP|USER]ADD_PARAM formatted string which includes the
diff --git a/meta/conf/local.conf.sample.extended b/meta/conf/local.conf.sample.extended
index 00716f9..ee8c78f 100644
--- a/meta/conf/local.conf.sample.extended
+++ b/meta/conf/local.conf.sample.extended
@@ -238,6 +238,30 @@
 #usermod -s /bin/sh tester; \
 #"
 
+# Various packages dynamically add users and groups to the system at package
+# install time.  For programs that do not care what the uid/gid is of the
+# resulting users/groups, the order of the install will determine the final
+# uid/gid.  This can lead to non-deterministic uid/gid values from one build
+# to another.  Use the following settings to specify that all user/group adds
+# should be created based on a static passwd/group file.
+#
+# By default the system looks in the BBPATH for files/passwd and files/group
+# the default can be overriden by spefying USERADD_UID/GID_TABLES.
+#
+# Note, if you change the value of the USERADD_REWRITE_PARAMS after starting
+# to build.  The TMPDIR may have incompatible uid/gids in it.  You must 'rm' 
+# the TMPDIR to avoid this issue.
+#
+#USERADD_REWRITE_PARAMS = "1"
+#USERADD_UID_TABLES = "files/passwd"
+#USERADD_GID_TABLES = "files/group"
+#
+# In order to prevent generating a system where a dynamicly assigned uid/gid
+# can exist, you can enable the following setting.  This will force the 
+# system to error out if the user/group name is not defined in the 
+# files/passwd or files/group (or specified replacements.)
+#USERADD_ERROR_DYNAMIC = "1"
+
 # Enabling FORTRAN
 # Note this is not officially supported and is just illustrated here to
 # show an example of how it can be done
-- 
1.8.1.2.545.g2f19ada

Re: [RFC PATCH 0/2] RFC: Implement deterministic uid/gid

[Mark Hatle]

I have updated the git://git.yoctoproject.org/poky-contrib mhatle/uidgid to the 
latest oe-core master.

I haven't seen any comments on this RFC yet.  Does anyone have any opinion 
either way on the code referenced here?

I'm confident patch 01/02 should be added to master.

The patch 02/02 works properly in all of my testing and does implement support 
for 'deterministic uid/gid' with dynamic passwd/group file construction during 
package install.  The code is only activated if USERADD_REWRITE_PARAMS is set to 
'1', so it's low risk -- but it does add a fairly large chunk of code to be 
maintained over time.

--Mark

On 12/10/13, 12:31 PM, Mark Hatle wrote:
> The following series implements the deterministic uid/gid setting for a
> distribution.  Currently when a filesystem is generated the uid/gid values
> are generally set at install time, so the install order determines what
> the actual uid/gid values become.  In order to create a deterministic uid/gid
> set, that still dynamically constructs the passwd/group file, we add an
> option to read a special passwd/group file to allow the system to determine
> the values.
>
> It uses the existing parameters, and the values from the special passwd/group
> files to reconstruct the parameter set to ensure these items are fully
> defined with static values.
>
> The first patch (01/02) is generally applicable.  It fixes a real bug in
> the way the user/group adds occur today within the system.
>
> Patch 02/02 implements the new functionality.
>
>
> The following changes since commit 8505f0fa48cc79d51616b923c6e2c778c4b46a44:
>
>    libmatchbox: use PACKAGECONFIG (2013-12-10 14:13:01 +0000)
>
> are available in the git repository at:
>
>    git://git.yoctoproject.org/poky-contrib mhatle/uidgid
>    http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=mhatle/uidgid
>
> Mark Hatle (2):
>    useradd.bbclass: Fix build time install issues
>    useradd.bbclass: Add ability to select a static uid/gid automatically
>
>   meta/classes/useradd.bbclass         | 247 ++++++++++++++++++++++++++++++++++-
>   meta/conf/local.conf.sample.extended |  24 ++++
>   2 files changed, 265 insertions(+), 6 deletions(-)
>

Re: [RFC PATCH 0/2] RFC: Implement deterministic uid/gid

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 3268 bytes --]

On Thu, Jan 09, 2014 at 01:49:28PM -0600, Mark Hatle wrote:
> I have updated the git://git.yoctoproject.org/poky-contrib mhatle/uidgid to the 
> latest oe-core master.
> 
> I haven't seen any comments on this RFC yet.  Does anyone have any opinion 
> either way on the code referenced here?

I wanted to do more tests and find some smaller reproducer, but I'm
interested in this series.

What I'm seeing in our builds is that when you compare
files-in-image.txt report from independent (not reusing sstate-cache)
builds with exactly the same metadata, group owners aren't
deterministic.

Random files in image getting owned by "messagebus" group etc., these
issues went away when useradd class creating these ad-hoc groups had
parameter for fixed gid or when I've added these groups directly to
base-passwd group.master.

Is this the same problem you're addressing in this patchset?

> I'm confident patch 01/02 should be added to master.
> 
> The patch 02/02 works properly in all of my testing and does implement support 
> for 'deterministic uid/gid' with dynamic passwd/group file construction during 
> package install.  The code is only activated if USERADD_REWRITE_PARAMS is set to 
> '1', so it's low risk -- but it does add a fairly large chunk of code to be 
> maintained over time.
> 
> --Mark
> 
> On 12/10/13, 12:31 PM, Mark Hatle wrote:
> > The following series implements the deterministic uid/gid setting for a
> > distribution.  Currently when a filesystem is generated the uid/gid values
> > are generally set at install time, so the install order determines what
> > the actual uid/gid values become.  In order to create a deterministic uid/gid
> > set, that still dynamically constructs the passwd/group file, we add an
> > option to read a special passwd/group file to allow the system to determine
> > the values.
> >
> > It uses the existing parameters, and the values from the special passwd/group
> > files to reconstruct the parameter set to ensure these items are fully
> > defined with static values.
> >
> > The first patch (01/02) is generally applicable.  It fixes a real bug in
> > the way the user/group adds occur today within the system.
> >
> > Patch 02/02 implements the new functionality.
> >
> >
> > The following changes since commit 8505f0fa48cc79d51616b923c6e2c778c4b46a44:
> >
> >    libmatchbox: use PACKAGECONFIG (2013-12-10 14:13:01 +0000)
> >
> > are available in the git repository at:
> >
> >    git://git.yoctoproject.org/poky-contrib mhatle/uidgid
> >    http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=mhatle/uidgid
> >
> > Mark Hatle (2):
> >    useradd.bbclass: Fix build time install issues
> >    useradd.bbclass: Add ability to select a static uid/gid automatically
> >
> >   meta/classes/useradd.bbclass         | 247 ++++++++++++++++++++++++++++++++++-
> >   meta/conf/local.conf.sample.extended |  24 ++++
> >   2 files changed, 265 insertions(+), 6 deletions(-)
> >
> 
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

Re: [RFC PATCH 0/2] RFC: Implement deterministic uid/gid

[Mark Hatle]

On 1/9/14, 5:19 PM, Martin Jansa wrote:
> On Thu, Jan 09, 2014 at 01:49:28PM -0600, Mark Hatle wrote:
>> I have updated the git://git.yoctoproject.org/poky-contrib mhatle/uidgid to the
>> latest oe-core master.
>>
>> I haven't seen any comments on this RFC yet.  Does anyone have any opinion
>> either way on the code referenced here?
>
> I wanted to do more tests and find some smaller reproducer, but I'm
> interested in this series.
>
> What I'm seeing in our builds is that when you compare
> files-in-image.txt report from independent (not reusing sstate-cache)
> builds with exactly the same metadata, group owners aren't
> deterministic.

Yes, this is partially to address that.  The issue is that the package install 
order itself is not deterministic.  The only thing that is, is the resulting set 
of files, modes, owner and group -names-.

Since the order the scripts are run is determines the effective uid/gid, the 
non-deterministic ordering causes these issues.

This patch set deals with that by allowing the user to specify -the- one and 
true deterministic set which is then reused.

> Random files in image getting owned by "messagebus" group etc., these
> issues went away when useradd class creating these ad-hoc groups had
> parameter for fixed gid or when I've added these groups directly to
> base-passwd group.master.
>
> Is this the same problem you're addressing in this patchset?

I have never seen a problem where specific files uname/gname are wrong.  It 
might be a related problem, but that shouldn't be happening in any case.

The -names- should always be right compared to the contents of the passwd/group 
files located within the constructed image.

I know there is an open bug where there were a few cases (in the sysroot) where 
things used a uname/gname from the default passwd/group file, but it had not yet 
been written.  So it fell back to the host's passwd/group file which had 
different numbers.  This is certainly incorrect behavior, and I think it's fixed 
by either inheriting the adduser class in those cases (even if they don't add 
users) or requiring the base-passwd in those specific package depends... but 
I've been looking for a reproducer to work though that.

--Mark

>> I'm confident patch 01/02 should be added to master.
>>
>> The patch 02/02 works properly in all of my testing and does implement support
>> for 'deterministic uid/gid' with dynamic passwd/group file construction during
>> package install.  The code is only activated if USERADD_REWRITE_PARAMS is set to
>> '1', so it's low risk -- but it does add a fairly large chunk of code to be
>> maintained over time.
>>
>> --Mark
>>
>> On 12/10/13, 12:31 PM, Mark Hatle wrote:
>>> The following series implements the deterministic uid/gid setting for a
>>> distribution.  Currently when a filesystem is generated the uid/gid values
>>> are generally set at install time, so the install order determines what
>>> the actual uid/gid values become.  In order to create a deterministic uid/gid
>>> set, that still dynamically constructs the passwd/group file, we add an
>>> option to read a special passwd/group file to allow the system to determine
>>> the values.
>>>
>>> It uses the existing parameters, and the values from the special passwd/group
>>> files to reconstruct the parameter set to ensure these items are fully
>>> defined with static values.
>>>
>>> The first patch (01/02) is generally applicable.  It fixes a real bug in
>>> the way the user/group adds occur today within the system.
>>>
>>> Patch 02/02 implements the new functionality.
>>>
>>>
>>> The following changes since commit 8505f0fa48cc79d51616b923c6e2c778c4b46a44:
>>>
>>>     libmatchbox: use PACKAGECONFIG (2013-12-10 14:13:01 +0000)
>>>
>>> are available in the git repository at:
>>>
>>>     git://git.yoctoproject.org/poky-contrib mhatle/uidgid
>>>     http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=mhatle/uidgid
>>>
>>> Mark Hatle (2):
>>>     useradd.bbclass: Fix build time install issues
>>>     useradd.bbclass: Add ability to select a static uid/gid automatically
>>>
>>>    meta/classes/useradd.bbclass         | 247 ++++++++++++++++++++++++++++++++++-
>>>    meta/conf/local.conf.sample.extended |  24 ++++
>>>    2 files changed, 265 insertions(+), 6 deletions(-)
>>>
>>
>> _______________________________________________
>> Openembedded-core mailing list
>> Openembedded-core@lists.openembedded.org
>> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>

Re: [RFC PATCH 0/2] RFC: Implement deterministic uid/gid

[Mark Hatle]

On 1/9/14, 1:49 PM, Mark Hatle wrote:
> I have updated the git://git.yoctoproject.org/poky-contrib mhatle/uidgid to the
> latest oe-core master.
>
> I haven't seen any comments on this RFC yet.  Does anyone have any opinion
> either way on the code referenced here?
>
> I'm confident patch 01/02 should be added to master.
>
> The patch 02/02 works properly in all of my testing and does implement support
> for 'deterministic uid/gid' with dynamic passwd/group file construction during
> package install.  The code is only activated if USERADD_REWRITE_PARAMS is set to
> '1', so it's low risk -- but it does add a fairly large chunk of code to be
> maintained over time.

I'm replying to myself on behalf of an external reviewer who is not on this 
mailing list.  They identified one item that they would like to see:

> Looking at how to override the passwd fields, it seems like it's an
> 'all-or-nothing' scenario for each user?
> E.g., if you specify a passwd line for a user, all fields has to be set
> and will be overridden?
> Look at lines like 227 and 248-252 in the patched file (I looked at
> http://git.yoctoproject.org/cgit.cgi/poky-contrib/tree/meta/classes/useradd.bbclass?h=mhatle/uidgid&id=21a7d772b9aee15d16ee8e29cb00a22e5fe21aac
>
> ).
> We've discussed the possibility to have this kind of scenario where we
> only want to e.g. override the default shell for the user (that is, not
> override uid, comment, etc). Is it maybe possible to e.g. add some kind
> of special marker to fields that should be discarded in the overridden
> file (like username:#:#:#:#:#:/bin/tcsh). That way we can automatically
> track upstream changes of the user but still override the shell.

It should be pretty easy for me to generate a third commit that implements this 
type of functional.  Does anyone have any opinions on a proper marker to 
indicate re-use the dynamic value?  (I'd like to stay away from a blank, as 
there are valid reasons to set fields to blank.  They're suggesting in the above 
a single '#' statement -- does this seem reasonable?

--Mark

> --Mark
>
> On 12/10/13, 12:31 PM, Mark Hatle wrote:
>> The following series implements the deterministic uid/gid setting for a
>> distribution.  Currently when a filesystem is generated the uid/gid values
>> are generally set at install time, so the install order determines what
>> the actual uid/gid values become.  In order to create a deterministic uid/gid
>> set, that still dynamically constructs the passwd/group file, we add an
>> option to read a special passwd/group file to allow the system to determine
>> the values.
>>
>> It uses the existing parameters, and the values from the special passwd/group
>> files to reconstruct the parameter set to ensure these items are fully
>> defined with static values.
>>
>> The first patch (01/02) is generally applicable.  It fixes a real bug in
>> the way the user/group adds occur today within the system.
>>
>> Patch 02/02 implements the new functionality.
>>
>>
>> The following changes since commit 8505f0fa48cc79d51616b923c6e2c778c4b46a44:
>>
>>     libmatchbox: use PACKAGECONFIG (2013-12-10 14:13:01 +0000)
>>
>> are available in the git repository at:
>>
>>     git://git.yoctoproject.org/poky-contrib mhatle/uidgid
>>     http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=mhatle/uidgid
>>
>> Mark Hatle (2):
>>     useradd.bbclass: Fix build time install issues
>>     useradd.bbclass: Add ability to select a static uid/gid automatically
>>
>>    meta/classes/useradd.bbclass         | 247 ++++++++++++++++++++++++++++++++++-
>>    meta/conf/local.conf.sample.extended |  24 ++++
>>    2 files changed, 265 insertions(+), 6 deletions(-)
>>
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>