* [v2 PATCH 0/2] Implement deterministic uid/gid
@ 2014-02-04 23:39 Mark Hatle
2014-02-04 23:39 ` [v2 PATCH 1/2] useradd.bbclass: Fix build time install issues Mark Hatle
2014-02-04 23:39 ` [v2 PATCH 2/2] useradd.bbclass: Add ability to select a static uid/gid automatically Mark Hatle
0 siblings, 2 replies; 6+ messages in thread
From: Mark Hatle @ 2014-02-04 23:39 UTC (permalink / raw)
To: openembedded-core
V2:
Rebase to latest master...
Rework the code a bit based on comments from a few people. Specifically
add a mode where passwd/group file entries are NOT overridden (blank info).
Clearly comment that the 'password' field is ignored, as is the group's
member fields.
Ensure that the 'enforcing' mode, doesn't trigger build failures, but simply
excludes the recipe from the build list. If the package is needed an error
indicating the problem will be generated. Makes for a cleaner build, and a
more targeted passwd/group file.
This was tested by doing the following:
(not enabling any of the code), build core-image-sato
copy the passwd/group file from tmp-eglibc/sysroots/<machine>/etc/ to meta/files/.
Clear the build directory
Enable the code adding the following to the conf/local.conf:
USERADD_REWRITE_PARAMS = '1'
Build, compare the rootfs /etc/passwd and /etc/group to the version in meta/files.
Verify the uid, gid and other information match. (Note xuser will have a slight
difference in the 'shell' field, but this is does to the difference between the
configuration of the sysroot and the target filesystem.)
Clear the build directory again
Enable the code adding the following to conf/local.conf:
USERADD_ERROR_DYNAMIC = '1'
Repeat the validation steps.
Clear the build directory again
Modify the meta/files/passwd and remove the items in the comment, home_dir and
shell fields. i.e.:
root::0:0:root:/home/root:/bin/sh
becomes
root::0:0:::
Repeat the build, verify the fields are all correct in the final image.
V1:
The following series implements the deterministic uid/gid setting for a
distribution. Currently when a filesystem is generated the uid/gid values
are generally set at install time, so the install order determines what
the actual uid/gid values become. In order to create a deterministic uid/gid
set, that still dynamically constructs the passwd/group file, we add an
option to read a special passwd/group file to allow the system to determine
the values.
It uses the existing parameters, and the values from the special passwd/group
files to reconstruct the parameter set to ensure these items are fully
defined with static values.
The first patch (01/02) is generally applicable. It fixes a real bug in
the way the user/group adds occur today within the system.
Patch 02/02 implements the new functionality.
The following changes since commit 8461283a648d7c5affd51971ebd9b35a8a4c625f:
sstate: Improve funciton checksums (2014-02-04 22:49:58 +0000)
are available in the git repository at:
git://git.yoctoproject.org/poky-contrib mhatle/uidgid
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=mhatle/uidgid
Mark Hatle (2):
useradd.bbclass: Fix build time install issues
useradd.bbclass: Add ability to select a static uid/gid automatically
meta/classes/useradd.bbclass | 279 ++++++++++++++++++++++++++++++++++-
meta/conf/local.conf.sample.extended | 24 +++
2 files changed, 297 insertions(+), 6 deletions(-)
--
1.8.5.3
^ permalink raw reply [flat|nested] 6+ messages in thread
* [v2 PATCH 1/2] useradd.bbclass: Fix build time install issues
2014-02-04 23:39 [v2 PATCH 0/2] Implement deterministic uid/gid Mark Hatle
@ 2014-02-04 23:39 ` Mark Hatle
2014-02-04 23:39 ` [v2 PATCH 2/2] useradd.bbclass: Add ability to select a static uid/gid automatically Mark Hatle
1 sibling, 0 replies; 6+ messages in thread
From: Mark Hatle @ 2014-02-04 23:39 UTC (permalink / raw)
To: openembedded-core
When the system attempts to populate the sysroot's passwd/group files, it
does so in a single block. However, with the way it was previously
implemented, the system would always run through the code necessary to
populate the sysroot, even in the case of target packages. This had
the side effect that a cross-installed filesystem may not match a
target installed filesystem.
The code was slightly reorganized to ensure that the cross/target installed
pre-install script behavior is the same. It also moves the block that
configures the sysroot parameters to the sysroot specific section of
the code.
Also some minor validation was occuring even on nativesdk packages.
Nativesdk packages should be skipped when processing useradd ops.
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
---
meta/classes/useradd.bbclass | 21 +++++++++++++++------
1 file changed, 15 insertions(+), 6 deletions(-)
diff --git a/meta/classes/useradd.bbclass b/meta/classes/useradd.bbclass
index a2ad648..ad6f61a 100644
--- a/meta/classes/useradd.bbclass
+++ b/meta/classes/useradd.bbclass
@@ -24,12 +24,11 @@ if test "x$D" != "x"; then
# Installing into a sysroot
SYSROOT="$D"
OPT="--root $D"
+fi
- # Add groups and users defined for all recipe packages
- GROUPADD_PARAM="${@get_all_cmd_params(d, 'groupadd')}"
- USERADD_PARAM="${@get_all_cmd_params(d, 'useradd')}"
- GROUPMEMS_PARAM="${@get_all_cmd_params(d, 'groupmems')}"
-else
+# If we're not doing a special SSTATE/SYSROOT install
+# then set the values, otherwise use the environment
+if test "x$UA_SYSROOT" = "x"; then
# Installing onto a target
# Add groups and users defined only for this package
GROUPADD_PARAM="${GROUPADD_PARAM}"
@@ -97,6 +96,15 @@ useradd_sysroot () {
# Explicitly set $D since it isn't set to anything
# before do_install
D=${STAGING_DIR_TARGET}
+
+ # Add groups and users defined for all recipe packages
+ GROUPADD_PARAM="${@get_all_cmd_params(d, 'groupadd')}"
+ USERADD_PARAM="${@get_all_cmd_params(d, 'useradd')}"
+ GROUPMEMS_PARAM="${@get_all_cmd_params(d, 'groupmems')}"
+
+ # Tell the system to use the environment vars
+ UA_SYSROOT=1
+
useradd_preinst
}
@@ -137,7 +145,8 @@ def update_useradd_after_parse(d):
bb.fatal("%s inherits useradd but doesn't set USERADD_PARAM, GROUPADD_PARAM or GROUPMEMS_PARAM for package %s" % (d.getVar('FILE'), pkg))
python __anonymous() {
- update_useradd_after_parse(d)
+ if not bb.data.inherits_class('nativesdk', d):
+ update_useradd_after_parse(d)
}
# Return a single [GROUP|USER]ADD_PARAM formatted string which includes the
--
1.8.5.3
^ permalink raw reply related [flat|nested] 6+ messages in thread
* [v2 PATCH 2/2] useradd.bbclass: Add ability to select a static uid/gid automatically
2014-02-04 23:39 [v2 PATCH 0/2] Implement deterministic uid/gid Mark Hatle
2014-02-04 23:39 ` [v2 PATCH 1/2] useradd.bbclass: Fix build time install issues Mark Hatle
@ 2014-02-04 23:39 ` Mark Hatle
2014-02-05 18:01 ` Saul Wold
2014-02-06 22:17 ` Richard Purdie
1 sibling, 2 replies; 6+ messages in thread
From: Mark Hatle @ 2014-02-04 23:39 UTC (permalink / raw)
To: openembedded-core
[YOCTO #5436]
Automatic selection of static uid/gid is needed for a dynamically generated
passwd and group file to have a deterministic outcome.
When a package is installed and instructs the system to add a new user or
group, unless it selects a static uid/gid value, the next available uid/gid
will be used. The order in which packages are installed is dynamically
computed, and may change from one installation to the next. This results
in a non-deterministic set of uid/gid values.
Enabling USERADD_REWRITE_PARAMS, and providing a preconfigured passwd/group
file will allow the continued dynamic generation of the passwd/group file
on the target, but ensure a deterministic outcome. (Dynamic generation is
desired so that user and groups that have no corresponding functionality
are not present within the final system image.)
The rewrite params function will override each of the fields in the
useradd and groupadd calls with the values specified. Note, the password
field is ignored as is the member groups field in the group file. If the
field is empty, the value will not be overridden. (Note, there is no way
to 'blank' a field, as this would only generally affect the 'comment' field
and there really is no reason to blank it.)
Enabling USERADD_ERROR_DYNAMIC will cause packages without static uid/gid
to generate an error and be skipped for the purpose of building. This is
used to prevent non-deterministic behavior.
USERADD_UID_TABLES and USERADD_GID_TABLES may be used to specify the name
of the passwd and group files. By default they are assumed to be
'files/passwd' and 'files/group'. Layers are searched in BBPATH order.
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
---
meta/classes/useradd.bbclass | 258 +++++++++++++++++++++++++++++++++++
meta/conf/local.conf.sample.extended | 24 ++++
2 files changed, 282 insertions(+)
diff --git a/meta/classes/useradd.bbclass b/meta/classes/useradd.bbclass
index ad6f61a..ff14b27 100644
--- a/meta/classes/useradd.bbclass
+++ b/meta/classes/useradd.bbclass
@@ -144,9 +144,267 @@ def update_useradd_after_parse(d):
if not d.getVar('USERADD_PARAM_%s' % pkg, True) and not d.getVar('GROUPADD_PARAM_%s' % pkg, True) and not d.getVar('GROUPMEMS_PARAM_%s' % pkg, True):
bb.fatal("%s inherits useradd but doesn't set USERADD_PARAM, GROUPADD_PARAM or GROUPMEMS_PARAM for package %s" % (d.getVar('FILE'), pkg))
+# In order to support a deterministic set of 'dynamic' users/groups,
+# we need a function to reformat the params based on a static file
+def update_useradd_static_config(d):
+ import argparse
+ import re
+
+ class myArgumentParser( argparse.ArgumentParser ):
+ def _print_message(self, message, file=None):
+ bb.warn("%s - %s: %s" % (d.getVar('PN', True), pkg, message))
+
+ # This should never be called...
+ def exit(self, status=0, message=None):
+ message = message or ("%s - %s: useradd.bbclass: Argument parsing exited" % (d.getVar('PN', True), pkg))
+ error(message)
+
+ def error(self, message):
+ raise bb.build.FuncFailed(message)
+
+ # We parse and rewrite the useradd components
+ def rewrite_useradd(params):
+ # The following comes from --help on useradd from shadow
+ parser = myArgumentParser(prog='useradd')
+ parser.add_argument("-b", "--base-dir", metavar="BASE_DIR", help="base directory for the home directory of the new account")
+ parser.add_argument("-c", "--comment", metavar="COMMENT", help="GECOS field of the new account")
+ parser.add_argument("-d", "--home-dir", metavar="HOME_DIR", help="home directory of the new account")
+ parser.add_argument("-D", "--defaults", help="print or change default useradd configuration", action="store_true")
+ parser.add_argument("-e", "--expiredate", metavar="EXPIRE_DATE", help="expiration date of the new account")
+ parser.add_argument("-f", "--inactive", metavar="INACTIVE", help="password inactivity period of the new account")
+ parser.add_argument("-g", "--gid", metavar="GROUP", help="name or ID of the primary group of the new account")
+ parser.add_argument("-G", "--groups", metavar="GROUPS", help="list of supplementary groups of the new account")
+ parser.add_argument("-k", "--skel", metavar="SKEL_DIR", help="use this alternative skeleton directory")
+ parser.add_argument("-K", "--key", metavar="KEY=VALUE", help="override /etc/login.defs defaults")
+ parser.add_argument("-l", "--no-log-init", help="do not add the user to the lastlog and faillog databases", action="store_true")
+ parser.add_argument("-m", "--create-home", help="create the user's home directory", action="store_true")
+ parser.add_argument("-M", "--no-create-home", help="do not create the user's home directory", action="store_true")
+ parser.add_argument("-N", "--no-user-group", help="do not create a group with the same name as the user", action="store_true")
+ parser.add_argument("-o", "--non-unique", help="allow to create users with duplicate (non-unique UID)", action="store_true")
+ parser.add_argument("-p", "--password", metavar="PASSWORD", help="encrypted password of the new account")
+ parser.add_argument("-R", "--root", metavar="CHROOT_DIR", help="directory to chroot into")
+ parser.add_argument("-r", "--system", help="create a system account", action="store_true")
+ parser.add_argument("-s", "--shell", metavar="SHELL", help="login shell of the new account")
+ parser.add_argument("-u", "--uid", metavar="UID", help="user ID of the new account")
+ parser.add_argument("-U", "--user-group", help="create a group with the same name as the user", action="store_true")
+ parser.add_argument("LOGIN", help="Login name of the new user")
+
+ # Return a list of configuration files based on either the default
+ # files/passwd or the contents of USERADD_UID_TABLES
+ # paths are resulved via BBPATH
+ def get_passwd_list(d):
+ str = ""
+ bbpath = d.getVar('BBPATH', True)
+ passwd_tables = d.getVar('USERADD_UID_TABLES', True)
+ if not passwd_tables:
+ passwd_tables = 'files/passwd'
+ for conf_file in passwd_tables.split():
+ str += " %s" % bb.utils.which(bbpath, conf_file)
+ return str
+
+ newparams = []
+ for param in re.split('''[ \t]*;[ \t]*(?=(?:[^'"]|'[^']*'|"[^"]*")*$)''', params):
+ param=param.strip()
+ try:
+ uaargs = parser.parse_args(re.split('''[ \t]*(?=(?:[^'"]|'[^']*'|"[^"]*")*$)''', param))
+ except:
+ raise bb.build.FuncFailed("%s: Unable to parse arguments for USERADD_PARAM_%s: '%s'" % (d.getVar('PN', True), pkg, param))
+
+ # files/passwd or the contents of USERADD_UID_TABLES
+ # Use the standard passwd layout:
+ # username:password:user_id:group_id:comment:home_directory:login_shell
+ # (we want to process in reverse order, as 'last found' in the list wins)
+ #
+ # If a field is left blank, the original value will be used. The 'username'
+ # field is required.
+ #
+ # Note: we ignore the password field, as including even the hashed password
+ # in the useradd command may introduce a security hole. It's assumed that
+ # all new users get the default ('*' which prevents login) until the user is
+ # specifically configured by the system admin.
+ for conf in get_passwd_list(d).split()[::-1]:
+ if os.path.exists(conf):
+ f = open(conf, "r")
+ for line in f:
+ if line.startswith('#'):
+ continue
+ field = line.rstrip().split(":")
+ if field[0] == uaargs.LOGIN:
+ if uaargs.uid and field[2] and (uaargs.uid != field[2]):
+ bb.warn("%s: Changing username %s's uid from (%s) to (%s), verify configuration files!" % (d.getVar('PN', True), uaargs.LOGIN, uaargs.uid, field[2]))
+ uaargs.uid = [field[2], uaargs.uid][not field[2]]
+
+ # Determine the possible groupname
+ # Unless the group name (or gid) is specified, we assume that the LOGIN is the groupname
+ #
+ # By default the system has creation of the matching groups enabled
+ # So if the implicit username-group creation is on, then the implicit groupname (LOGIN)
+ # is used, and we disable the user_group option.
+ #
+ uaargs.groupname = [uaargs.gid, uaargs.LOGIN][not uaargs.gid or uaargs.user_group]
+ uaargs.user_group = False
+
+ uaargs.gid = [uaargs.gid, uaargs.groupname][not uaargs.gid]
+ uaargs.gid = [field[3], uaargs.gid][not field[3]]
+
+ if uaargs.groupname == uaargs.gid:
+ # Nothing to do...
+ pass
+ elif (uaargs.groupname and uaargs.groupname.isdigit()) and (uaargs.gid and uaargs.gid.isdigit()) and (uaargs.groupname != uaargs.gid):
+ # We want to add a group, but we don't know it's name... so we can't add the group...
+ # We have to assume the group has previously been added or we'll fail on the adduser...
+ # Note: specifying the actual gid is very rare in OE, usually the group name is specified.
+ bb.warn("%s: Changing gid for login %s from (%s) to (%s), verify configuration files!" % (d.getVar('PN', True), uaargs.LOGIN, uaargs.groupname, uaargs.gid))
+ elif uaargs.groupname and (uaargs.gid and uaargs.gid.isdigit()):
+ bb.debug(1, "Adding group %s gid (%s)!" % (uaargs.groupname, uaargs.gid))
+ groupadd = d.getVar("GROUPADD_PARAM_%s" % pkg, True)
+ newgroup = "-g %s %s" % (uaargs.gid, uaargs.groupname)
+ if groupadd:
+ d.setVar("GROUPADD_PARAM_%s" % pkg, "%s ; %s" % (groupadd, newgroup))
+ else:
+ d.setVar("GROUPADD_PARAM_%s" % pkg, newgroup)
+
+ uaargs.comment = ["'%s'" % field[4], uaargs.comment][not field[4]]
+ uaargs.home_dir = [field[5], uaargs.home_dir][not field[5]]
+ uaargs.shell = [field[6], uaargs.shell][not field[6]]
+ break
+
+ # Should be an error if a specific option is set...
+ if d.getVar('USERADD_ERROR_DYNAMIC', True) == '1' and (not uaargs.uid or not uaargs.gid):
+ raise bb.build.FuncFailed("%s - %s: Username %s does not have a static uid/gid defined." % (d.getVar('PN', True), pkg, uaargs.LOGIN))
+
+ # Reconstruct the args...
+ newparam = ['', ' --defaults'][uaargs.defaults]
+ newparam += ['', ' --base-dir %s' % uaargs.base_dir][uaargs.base_dir != None]
+ newparam += ['', ' --comment %s' % uaargs.comment][uaargs.comment != None]
+ newparam += ['', ' --home-dir %s' % uaargs.home_dir][uaargs.home_dir != None]
+ newparam += ['', ' --expiredata %s' % uaargs.expiredate][uaargs.expiredate != None]
+ newparam += ['', ' --inactive %s' % uaargs.inactive][uaargs.inactive != None]
+ newparam += ['', ' --gid %s' % uaargs.gid][uaargs.gid != None]
+ newparam += ['', ' --groups %s' % uaargs.groups][uaargs.groups != None]
+ newparam += ['', ' --skel %s' % uaargs.skel][uaargs.skel != None]
+ newparam += ['', ' --key %s' % uaargs.key][uaargs.key != None]
+ newparam += ['', ' --no-log-init'][uaargs.no_log_init]
+ newparam += ['', ' --create-home'][uaargs.create_home]
+ newparam += ['', ' --no-create-home'][uaargs.no_create_home]
+ newparam += ['', ' --no-user-group'][uaargs.no_user_group]
+ newparam += ['', ' --non-unique'][uaargs.non_unique]
+ newparam += ['', ' --password %s' % uaargs.password][uaargs.password != None]
+ newparam += ['', ' --root %s' % uaargs.root][uaargs.root != None]
+ newparam += ['', ' --system'][uaargs.system]
+ newparam += ['', ' --shell %s' % uaargs.shell][uaargs.shell != None]
+ newparam += ['', ' --uid %s' % uaargs.uid][uaargs.uid != None]
+ newparam += ['', ' --user-group'][uaargs.user_group]
+ newparam += ' %s' % uaargs.LOGIN
+
+ newparams.append(newparam)
+
+ return " ;".join(newparams).strip()
+
+ # We parse and rewrite the groupadd components
+ def rewrite_groupadd(params):
+ # The following comes from --help on groupadd from shadow
+ parser = myArgumentParser(prog='groupadd')
+ parser.add_argument("-f", "--force", help="exit successfully if the group already exists, and cancel -g if the GID is already used", action="store_true")
+ parser.add_argument("-g", "--gid", metavar="GID", help="use GID for the new group")
+ parser.add_argument("-K", "--key", metavar="KEY=VALUE", help="override /etc/login.defs defaults")
+ parser.add_argument("-o", "--non-unique", help="allow to create groups with duplicate (non-unique) GID", action="store_true")
+ parser.add_argument("-p", "--password", metavar="PASSWORD", help="use this encrypted password for the new group")
+ parser.add_argument("-R", "--root", metavar="CHROOT_DIR", help="directory to chroot into")
+ parser.add_argument("-r", "--system", help="create a system account", action="store_true")
+ parser.add_argument("GROUP", help="Group name of the new group")
+
+ # Return a list of configuration files based on either the default
+ # files/group or the contents of USERADD_GID_TABLES
+ # paths are resulved via BBPATH
+ def get_group_list(d):
+ str = ""
+ bbpath = d.getVar('BBPATH', True)
+ group_tables = d.getVar('USERADD_GID_TABLES', True)
+ if not group_tables:
+ group_tables = 'files/group'
+ for conf_file in group_tables.split():
+ str += " %s" % bb.utils.which(bbpath, conf_file)
+ return str
+
+ newparams = []
+ for param in re.split('''[ \t]*;[ \t]*(?=(?:[^'"]|'[^']*'|"[^"]*")*$)''', params):
+ param=param.strip()
+ try:
+ # If we're processing multiple lines, we could have left over values here...
+ gaargs = parser.parse_args(re.split('''[ \t]*(?=(?:[^'"]|'[^']*'|"[^"]*")*$)''', param))
+ except:
+ raise bb.build.FuncFailed("%s: Unable to parse arguments for GROUPADD_PARAM_%s: '%s'" % (d.getVar('PN', True), pkg, param))
+
+ # Need to iterate over layers and open the right file(s)
+ # Use the standard group layout:
+ # groupname:password:group_id:group_members
+ #
+ # If a field is left blank, the original value will be used. The 'groupname' field
+ # is required.
+ #
+ # Note: similar to the passwd file, the 'password' filed is ignored
+ # Note: group_members is ignored, group members must be configured with the GROUPMEMS_PARAM
+ for conf in get_group_list(d).split()[::-1]:
+ if os.path.exists(conf):
+ f = open(conf, "r")
+ for line in f:
+ if line.startswith('#'):
+ continue
+ field = line.rstrip().split(":")
+ if field[0] == gaargs.GROUP and field[2]:
+ if gaargs.gid and (gaargs.gid != field[2]):
+ bb.warn("%s: Changing groupname %s's gid from (%s) to (%s), verify configuration files!" % (d.getVar('PN', True), gaargs.GROUP, gaargs.gid, field[2]))
+ gaargs.gid = field[2]
+ break
+
+ if d.getVar('USERADD_ERROR_DYNAMIC', True) == '1' and not gaargs.gid:
+ raise bb.build.FuncFailed("%s - %s: Groupname %s does not have a static gid defined." % (d.getVar('PN', True), pkg, gaargs.GROUP))
+
+ # Reconstruct the args...
+ newparam = ['', ' --force'][gaargs.force]
+ newparam += ['', ' --gid %s' % gaargs.gid][gaargs.gid != None]
+ newparam += ['', ' --key %s' % gaargs.key][gaargs.key != None]
+ newparam += ['', ' --non-unique'][gaargs.non_unique]
+ newparam += ['', ' --password %s' % gaargs.password][gaargs.password != None]
+ newparam += ['', ' --root %s' % gaargs.root][gaargs.root != None]
+ newparam += ['', ' --system'][gaargs.system]
+ newparam += ' %s' % gaargs.GROUP
+
+ newparams.append(newparam)
+
+ return " ;".join(newparams).strip()
+
+ # Load and process the users and groups, rewriting the adduser/addgroup params
+ useradd_packages = d.getVar('USERADD_PACKAGES', True)
+
+ for pkg in useradd_packages.split():
+ # Groupmems doesn't have anything we might want to change, so simply validating
+ # is a bit of a waste -- only process useradd/groupadd
+ useradd_param = d.getVar('USERADD_PARAM_%s' % pkg, True)
+ if useradd_param:
+ #bb.warn("Before: 'USERADD_PARAM_%s' - '%s'" % (pkg, useradd_param))
+ d.setVar('USERADD_PARAM_%s' % pkg, rewrite_useradd(useradd_param))
+ #bb.warn("After: 'USERADD_PARAM_%s' - '%s'" % (pkg, d.getVar('USERADD_PARAM_%s' % pkg, True)))
+
+ groupadd_param = d.getVar('GROUPADD_PARAM_%s' % pkg, True)
+ if groupadd_param:
+ #bb.warn("Before: 'GROUPADD_PARAM_%s' - '%s'" % (pkg, groupadd_param))
+ d.setVar('GROUPADD_PARAM_%s' % pkg, rewrite_groupadd(groupadd_param))
+ #bb.warn("After: 'GROUPADD_PARAM_%s' - '%s'" % (pkg, d.getVar('GROUPADD_PARAM_%s' % pkg, True)))
+
+
+
python __anonymous() {
if not bb.data.inherits_class('nativesdk', d):
update_useradd_after_parse(d)
+
+ if d.getVar('USERADD_REWRITE_PARAMS', True) == '1':
+ try:
+ update_useradd_static_config(d)
+ except bb.build.FuncFailed as f:
+ bb.debug(1, "Skipping recipe %s: %s" % (d.getVar('PN', True), f))
+ raise bb.parse.SkipPackage(f)
}
# Return a single [GROUP|USER]ADD_PARAM formatted string which includes the
diff --git a/meta/conf/local.conf.sample.extended b/meta/conf/local.conf.sample.extended
index c7c4f40..b73a0a7 100644
--- a/meta/conf/local.conf.sample.extended
+++ b/meta/conf/local.conf.sample.extended
@@ -253,6 +253,30 @@
#usermod -s /bin/sh tester; \
#"
+# Various packages dynamically add users and groups to the system at package
+# install time. For programs that do not care what the uid/gid is of the
+# resulting users/groups, the order of the install will determine the final
+# uid/gid. This can lead to non-deterministic uid/gid values from one build
+# to another. Use the following settings to specify that all user/group adds
+# should be created based on a static passwd/group file.
+#
+# By default the system looks in the BBPATH for files/passwd and files/group
+# the default can be overriden by spefying USERADD_UID/GID_TABLES.
+#
+# Note, if you change the value of the USERADD_REWRITE_PARAMS after starting
+# to build. The TMPDIR may have incompatible uid/gids in it. You must 'rm'
+# the TMPDIR to avoid this issue.
+#
+#USERADD_REWRITE_PARAMS = "1"
+#USERADD_UID_TABLES = "files/passwd"
+#USERADD_GID_TABLES = "files/group"
+#
+# In order to prevent generating a system where a dynamicly assigned uid/gid
+# can exist, you can enable the following setting. This will force the
+# system to error out if the user/group name is not defined in the
+# files/passwd or files/group (or specified replacements.)
+#USERADD_ERROR_DYNAMIC = "1"
+
# Enabling FORTRAN
# Note this is not officially supported and is just illustrated here to
# show an example of how it can be done
--
1.8.5.3
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [v2 PATCH 2/2] useradd.bbclass: Add ability to select a static uid/gid automatically
2014-02-04 23:39 ` [v2 PATCH 2/2] useradd.bbclass: Add ability to select a static uid/gid automatically Mark Hatle
@ 2014-02-05 18:01 ` Saul Wold
2014-02-05 18:14 ` Mark Hatle
2014-02-06 22:17 ` Richard Purdie
1 sibling, 1 reply; 6+ messages in thread
From: Saul Wold @ 2014-02-05 18:01 UTC (permalink / raw)
To: Mark Hatle, openembedded-core
On 02/04/2014 03:39 PM, Mark Hatle wrote:
> [YOCTO #5436]
>
> Automatic selection of static uid/gid is needed for a dynamically generated
> passwd and group file to have a deterministic outcome.
>
> When a package is installed and instructs the system to add a new user or
> group, unless it selects a static uid/gid value, the next available uid/gid
> will be used. The order in which packages are installed is dynamically
> computed, and may change from one installation to the next. This results
> in a non-deterministic set of uid/gid values.
>
> Enabling USERADD_REWRITE_PARAMS, and providing a preconfigured passwd/group
> file will allow the continued dynamic generation of the passwd/group file
> on the target, but ensure a deterministic outcome. (Dynamic generation is
> desired so that user and groups that have no corresponding functionality
> are not present within the final system image.)
>
> The rewrite params function will override each of the fields in the
> useradd and groupadd calls with the values specified. Note, the password
> field is ignored as is the member groups field in the group file. If the
> field is empty, the value will not be overridden. (Note, there is no way
> to 'blank' a field, as this would only generally affect the 'comment' field
> and there really is no reason to blank it.)
>
> Enabling USERADD_ERROR_DYNAMIC will cause packages without static uid/gid
> to generate an error and be skipped for the purpose of building. This is
> used to prevent non-deterministic behavior.
>
> USERADD_UID_TABLES and USERADD_GID_TABLES may be used to specify the name
> of the passwd and group files. By default they are assumed to be
> 'files/passwd' and 'files/group'. Layers are searched in BBPATH order.
>
> Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
> ---
> meta/classes/useradd.bbclass | 258 +++++++++++++++++++++++++++++++++++
> meta/conf/local.conf.sample.extended | 24 ++++
> 2 files changed, 282 insertions(+)
>
This need to be split to 2 patches one for oe-core and the
local.conf.sample.extended for poky.
Sau!
> diff --git a/meta/classes/useradd.bbclass b/meta/classes/useradd.bbclass
> index ad6f61a..ff14b27 100644
> --- a/meta/classes/useradd.bbclass
> +++ b/meta/classes/useradd.bbclass
> @@ -144,9 +144,267 @@ def update_useradd_after_parse(d):
> if not d.getVar('USERADD_PARAM_%s' % pkg, True) and not d.getVar('GROUPADD_PARAM_%s' % pkg, True) and not d.getVar('GROUPMEMS_PARAM_%s' % pkg, True):
> bb.fatal("%s inherits useradd but doesn't set USERADD_PARAM, GROUPADD_PARAM or GROUPMEMS_PARAM for package %s" % (d.getVar('FILE'), pkg))
>
> +# In order to support a deterministic set of 'dynamic' users/groups,
> +# we need a function to reformat the params based on a static file
> +def update_useradd_static_config(d):
> + import argparse
> + import re
> +
> + class myArgumentParser( argparse.ArgumentParser ):
> + def _print_message(self, message, file=None):
> + bb.warn("%s - %s: %s" % (d.getVar('PN', True), pkg, message))
> +
> + # This should never be called...
> + def exit(self, status=0, message=None):
> + message = message or ("%s - %s: useradd.bbclass: Argument parsing exited" % (d.getVar('PN', True), pkg))
> + error(message)
> +
> + def error(self, message):
> + raise bb.build.FuncFailed(message)
> +
> + # We parse and rewrite the useradd components
> + def rewrite_useradd(params):
> + # The following comes from --help on useradd from shadow
> + parser = myArgumentParser(prog='useradd')
> + parser.add_argument("-b", "--base-dir", metavar="BASE_DIR", help="base directory for the home directory of the new account")
> + parser.add_argument("-c", "--comment", metavar="COMMENT", help="GECOS field of the new account")
> + parser.add_argument("-d", "--home-dir", metavar="HOME_DIR", help="home directory of the new account")
> + parser.add_argument("-D", "--defaults", help="print or change default useradd configuration", action="store_true")
> + parser.add_argument("-e", "--expiredate", metavar="EXPIRE_DATE", help="expiration date of the new account")
> + parser.add_argument("-f", "--inactive", metavar="INACTIVE", help="password inactivity period of the new account")
> + parser.add_argument("-g", "--gid", metavar="GROUP", help="name or ID of the primary group of the new account")
> + parser.add_argument("-G", "--groups", metavar="GROUPS", help="list of supplementary groups of the new account")
> + parser.add_argument("-k", "--skel", metavar="SKEL_DIR", help="use this alternative skeleton directory")
> + parser.add_argument("-K", "--key", metavar="KEY=VALUE", help="override /etc/login.defs defaults")
> + parser.add_argument("-l", "--no-log-init", help="do not add the user to the lastlog and faillog databases", action="store_true")
> + parser.add_argument("-m", "--create-home", help="create the user's home directory", action="store_true")
> + parser.add_argument("-M", "--no-create-home", help="do not create the user's home directory", action="store_true")
> + parser.add_argument("-N", "--no-user-group", help="do not create a group with the same name as the user", action="store_true")
> + parser.add_argument("-o", "--non-unique", help="allow to create users with duplicate (non-unique UID)", action="store_true")
> + parser.add_argument("-p", "--password", metavar="PASSWORD", help="encrypted password of the new account")
> + parser.add_argument("-R", "--root", metavar="CHROOT_DIR", help="directory to chroot into")
> + parser.add_argument("-r", "--system", help="create a system account", action="store_true")
> + parser.add_argument("-s", "--shell", metavar="SHELL", help="login shell of the new account")
> + parser.add_argument("-u", "--uid", metavar="UID", help="user ID of the new account")
> + parser.add_argument("-U", "--user-group", help="create a group with the same name as the user", action="store_true")
> + parser.add_argument("LOGIN", help="Login name of the new user")
> +
> + # Return a list of configuration files based on either the default
> + # files/passwd or the contents of USERADD_UID_TABLES
> + # paths are resulved via BBPATH
> + def get_passwd_list(d):
> + str = ""
> + bbpath = d.getVar('BBPATH', True)
> + passwd_tables = d.getVar('USERADD_UID_TABLES', True)
> + if not passwd_tables:
> + passwd_tables = 'files/passwd'
> + for conf_file in passwd_tables.split():
> + str += " %s" % bb.utils.which(bbpath, conf_file)
> + return str
> +
> + newparams = []
> + for param in re.split('''[ \t]*;[ \t]*(?=(?:[^'"]|'[^']*'|"[^"]*")*$)''', params):
> + param=param.strip()
> + try:
> + uaargs = parser.parse_args(re.split('''[ \t]*(?=(?:[^'"]|'[^']*'|"[^"]*")*$)''', param))
> + except:
> + raise bb.build.FuncFailed("%s: Unable to parse arguments for USERADD_PARAM_%s: '%s'" % (d.getVar('PN', True), pkg, param))
> +
> + # files/passwd or the contents of USERADD_UID_TABLES
> + # Use the standard passwd layout:
> + # username:password:user_id:group_id:comment:home_directory:login_shell
> + # (we want to process in reverse order, as 'last found' in the list wins)
> + #
> + # If a field is left blank, the original value will be used. The 'username'
> + # field is required.
> + #
> + # Note: we ignore the password field, as including even the hashed password
> + # in the useradd command may introduce a security hole. It's assumed that
> + # all new users get the default ('*' which prevents login) until the user is
> + # specifically configured by the system admin.
> + for conf in get_passwd_list(d).split()[::-1]:
> + if os.path.exists(conf):
> + f = open(conf, "r")
> + for line in f:
> + if line.startswith('#'):
> + continue
> + field = line.rstrip().split(":")
> + if field[0] == uaargs.LOGIN:
> + if uaargs.uid and field[2] and (uaargs.uid != field[2]):
> + bb.warn("%s: Changing username %s's uid from (%s) to (%s), verify configuration files!" % (d.getVar('PN', True), uaargs.LOGIN, uaargs.uid, field[2]))
> + uaargs.uid = [field[2], uaargs.uid][not field[2]]
> +
> + # Determine the possible groupname
> + # Unless the group name (or gid) is specified, we assume that the LOGIN is the groupname
> + #
> + # By default the system has creation of the matching groups enabled
> + # So if the implicit username-group creation is on, then the implicit groupname (LOGIN)
> + # is used, and we disable the user_group option.
> + #
> + uaargs.groupname = [uaargs.gid, uaargs.LOGIN][not uaargs.gid or uaargs.user_group]
> + uaargs.user_group = False
> +
> + uaargs.gid = [uaargs.gid, uaargs.groupname][not uaargs.gid]
> + uaargs.gid = [field[3], uaargs.gid][not field[3]]
> +
> + if uaargs.groupname == uaargs.gid:
> + # Nothing to do...
> + pass
> + elif (uaargs.groupname and uaargs.groupname.isdigit()) and (uaargs.gid and uaargs.gid.isdigit()) and (uaargs.groupname != uaargs.gid):
> + # We want to add a group, but we don't know it's name... so we can't add the group...
> + # We have to assume the group has previously been added or we'll fail on the adduser...
> + # Note: specifying the actual gid is very rare in OE, usually the group name is specified.
> + bb.warn("%s: Changing gid for login %s from (%s) to (%s), verify configuration files!" % (d.getVar('PN', True), uaargs.LOGIN, uaargs.groupname, uaargs.gid))
> + elif uaargs.groupname and (uaargs.gid and uaargs.gid.isdigit()):
> + bb.debug(1, "Adding group %s gid (%s)!" % (uaargs.groupname, uaargs.gid))
> + groupadd = d.getVar("GROUPADD_PARAM_%s" % pkg, True)
> + newgroup = "-g %s %s" % (uaargs.gid, uaargs.groupname)
> + if groupadd:
> + d.setVar("GROUPADD_PARAM_%s" % pkg, "%s ; %s" % (groupadd, newgroup))
> + else:
> + d.setVar("GROUPADD_PARAM_%s" % pkg, newgroup)
> +
> + uaargs.comment = ["'%s'" % field[4], uaargs.comment][not field[4]]
> + uaargs.home_dir = [field[5], uaargs.home_dir][not field[5]]
> + uaargs.shell = [field[6], uaargs.shell][not field[6]]
> + break
> +
> + # Should be an error if a specific option is set...
> + if d.getVar('USERADD_ERROR_DYNAMIC', True) == '1' and (not uaargs.uid or not uaargs.gid):
> + raise bb.build.FuncFailed("%s - %s: Username %s does not have a static uid/gid defined." % (d.getVar('PN', True), pkg, uaargs.LOGIN))
> +
> + # Reconstruct the args...
> + newparam = ['', ' --defaults'][uaargs.defaults]
> + newparam += ['', ' --base-dir %s' % uaargs.base_dir][uaargs.base_dir != None]
> + newparam += ['', ' --comment %s' % uaargs.comment][uaargs.comment != None]
> + newparam += ['', ' --home-dir %s' % uaargs.home_dir][uaargs.home_dir != None]
> + newparam += ['', ' --expiredata %s' % uaargs.expiredate][uaargs.expiredate != None]
> + newparam += ['', ' --inactive %s' % uaargs.inactive][uaargs.inactive != None]
> + newparam += ['', ' --gid %s' % uaargs.gid][uaargs.gid != None]
> + newparam += ['', ' --groups %s' % uaargs.groups][uaargs.groups != None]
> + newparam += ['', ' --skel %s' % uaargs.skel][uaargs.skel != None]
> + newparam += ['', ' --key %s' % uaargs.key][uaargs.key != None]
> + newparam += ['', ' --no-log-init'][uaargs.no_log_init]
> + newparam += ['', ' --create-home'][uaargs.create_home]
> + newparam += ['', ' --no-create-home'][uaargs.no_create_home]
> + newparam += ['', ' --no-user-group'][uaargs.no_user_group]
> + newparam += ['', ' --non-unique'][uaargs.non_unique]
> + newparam += ['', ' --password %s' % uaargs.password][uaargs.password != None]
> + newparam += ['', ' --root %s' % uaargs.root][uaargs.root != None]
> + newparam += ['', ' --system'][uaargs.system]
> + newparam += ['', ' --shell %s' % uaargs.shell][uaargs.shell != None]
> + newparam += ['', ' --uid %s' % uaargs.uid][uaargs.uid != None]
> + newparam += ['', ' --user-group'][uaargs.user_group]
> + newparam += ' %s' % uaargs.LOGIN
> +
> + newparams.append(newparam)
> +
> + return " ;".join(newparams).strip()
> +
> + # We parse and rewrite the groupadd components
> + def rewrite_groupadd(params):
> + # The following comes from --help on groupadd from shadow
> + parser = myArgumentParser(prog='groupadd')
> + parser.add_argument("-f", "--force", help="exit successfully if the group already exists, and cancel -g if the GID is already used", action="store_true")
> + parser.add_argument("-g", "--gid", metavar="GID", help="use GID for the new group")
> + parser.add_argument("-K", "--key", metavar="KEY=VALUE", help="override /etc/login.defs defaults")
> + parser.add_argument("-o", "--non-unique", help="allow to create groups with duplicate (non-unique) GID", action="store_true")
> + parser.add_argument("-p", "--password", metavar="PASSWORD", help="use this encrypted password for the new group")
> + parser.add_argument("-R", "--root", metavar="CHROOT_DIR", help="directory to chroot into")
> + parser.add_argument("-r", "--system", help="create a system account", action="store_true")
> + parser.add_argument("GROUP", help="Group name of the new group")
> +
> + # Return a list of configuration files based on either the default
> + # files/group or the contents of USERADD_GID_TABLES
> + # paths are resulved via BBPATH
> + def get_group_list(d):
> + str = ""
> + bbpath = d.getVar('BBPATH', True)
> + group_tables = d.getVar('USERADD_GID_TABLES', True)
> + if not group_tables:
> + group_tables = 'files/group'
> + for conf_file in group_tables.split():
> + str += " %s" % bb.utils.which(bbpath, conf_file)
> + return str
> +
> + newparams = []
> + for param in re.split('''[ \t]*;[ \t]*(?=(?:[^'"]|'[^']*'|"[^"]*")*$)''', params):
> + param=param.strip()
> + try:
> + # If we're processing multiple lines, we could have left over values here...
> + gaargs = parser.parse_args(re.split('''[ \t]*(?=(?:[^'"]|'[^']*'|"[^"]*")*$)''', param))
> + except:
> + raise bb.build.FuncFailed("%s: Unable to parse arguments for GROUPADD_PARAM_%s: '%s'" % (d.getVar('PN', True), pkg, param))
> +
> + # Need to iterate over layers and open the right file(s)
> + # Use the standard group layout:
> + # groupname:password:group_id:group_members
> + #
> + # If a field is left blank, the original value will be used. The 'groupname' field
> + # is required.
> + #
> + # Note: similar to the passwd file, the 'password' filed is ignored
> + # Note: group_members is ignored, group members must be configured with the GROUPMEMS_PARAM
> + for conf in get_group_list(d).split()[::-1]:
> + if os.path.exists(conf):
> + f = open(conf, "r")
> + for line in f:
> + if line.startswith('#'):
> + continue
> + field = line.rstrip().split(":")
> + if field[0] == gaargs.GROUP and field[2]:
> + if gaargs.gid and (gaargs.gid != field[2]):
> + bb.warn("%s: Changing groupname %s's gid from (%s) to (%s), verify configuration files!" % (d.getVar('PN', True), gaargs.GROUP, gaargs.gid, field[2]))
> + gaargs.gid = field[2]
> + break
> +
> + if d.getVar('USERADD_ERROR_DYNAMIC', True) == '1' and not gaargs.gid:
> + raise bb.build.FuncFailed("%s - %s: Groupname %s does not have a static gid defined." % (d.getVar('PN', True), pkg, gaargs.GROUP))
> +
> + # Reconstruct the args...
> + newparam = ['', ' --force'][gaargs.force]
> + newparam += ['', ' --gid %s' % gaargs.gid][gaargs.gid != None]
> + newparam += ['', ' --key %s' % gaargs.key][gaargs.key != None]
> + newparam += ['', ' --non-unique'][gaargs.non_unique]
> + newparam += ['', ' --password %s' % gaargs.password][gaargs.password != None]
> + newparam += ['', ' --root %s' % gaargs.root][gaargs.root != None]
> + newparam += ['', ' --system'][gaargs.system]
> + newparam += ' %s' % gaargs.GROUP
> +
> + newparams.append(newparam)
> +
> + return " ;".join(newparams).strip()
> +
> + # Load and process the users and groups, rewriting the adduser/addgroup params
> + useradd_packages = d.getVar('USERADD_PACKAGES', True)
> +
> + for pkg in useradd_packages.split():
> + # Groupmems doesn't have anything we might want to change, so simply validating
> + # is a bit of a waste -- only process useradd/groupadd
> + useradd_param = d.getVar('USERADD_PARAM_%s' % pkg, True)
> + if useradd_param:
> + #bb.warn("Before: 'USERADD_PARAM_%s' - '%s'" % (pkg, useradd_param))
> + d.setVar('USERADD_PARAM_%s' % pkg, rewrite_useradd(useradd_param))
> + #bb.warn("After: 'USERADD_PARAM_%s' - '%s'" % (pkg, d.getVar('USERADD_PARAM_%s' % pkg, True)))
> +
> + groupadd_param = d.getVar('GROUPADD_PARAM_%s' % pkg, True)
> + if groupadd_param:
> + #bb.warn("Before: 'GROUPADD_PARAM_%s' - '%s'" % (pkg, groupadd_param))
> + d.setVar('GROUPADD_PARAM_%s' % pkg, rewrite_groupadd(groupadd_param))
> + #bb.warn("After: 'GROUPADD_PARAM_%s' - '%s'" % (pkg, d.getVar('GROUPADD_PARAM_%s' % pkg, True)))
> +
> +
> +
> python __anonymous() {
> if not bb.data.inherits_class('nativesdk', d):
> update_useradd_after_parse(d)
> +
> + if d.getVar('USERADD_REWRITE_PARAMS', True) == '1':
> + try:
> + update_useradd_static_config(d)
> + except bb.build.FuncFailed as f:
> + bb.debug(1, "Skipping recipe %s: %s" % (d.getVar('PN', True), f))
> + raise bb.parse.SkipPackage(f)
> }
>
> # Return a single [GROUP|USER]ADD_PARAM formatted string which includes the
> diff --git a/meta/conf/local.conf.sample.extended b/meta/conf/local.conf.sample.extended
> index c7c4f40..b73a0a7 100644
> --- a/meta/conf/local.conf.sample.extended
> +++ b/meta/conf/local.conf.sample.extended
> @@ -253,6 +253,30 @@
> #usermod -s /bin/sh tester; \
> #"
>
> +# Various packages dynamically add users and groups to the system at package
> +# install time. For programs that do not care what the uid/gid is of the
> +# resulting users/groups, the order of the install will determine the final
> +# uid/gid. This can lead to non-deterministic uid/gid values from one build
> +# to another. Use the following settings to specify that all user/group adds
> +# should be created based on a static passwd/group file.
> +#
> +# By default the system looks in the BBPATH for files/passwd and files/group
> +# the default can be overriden by spefying USERADD_UID/GID_TABLES.
> +#
> +# Note, if you change the value of the USERADD_REWRITE_PARAMS after starting
> +# to build. The TMPDIR may have incompatible uid/gids in it. You must 'rm'
> +# the TMPDIR to avoid this issue.
> +#
> +#USERADD_REWRITE_PARAMS = "1"
> +#USERADD_UID_TABLES = "files/passwd"
> +#USERADD_GID_TABLES = "files/group"
> +#
> +# In order to prevent generating a system where a dynamicly assigned uid/gid
> +# can exist, you can enable the following setting. This will force the
> +# system to error out if the user/group name is not defined in the
> +# files/passwd or files/group (or specified replacements.)
> +#USERADD_ERROR_DYNAMIC = "1"
> +
> # Enabling FORTRAN
> # Note this is not officially supported and is just illustrated here to
> # show an example of how it can be done
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [v2 PATCH 2/2] useradd.bbclass: Add ability to select a static uid/gid automatically
2014-02-05 18:01 ` Saul Wold
@ 2014-02-05 18:14 ` Mark Hatle
0 siblings, 0 replies; 6+ messages in thread
From: Mark Hatle @ 2014-02-05 18:14 UTC (permalink / raw)
To: Saul Wold, openembedded-core
On 2/5/14, 12:01 PM, Saul Wold wrote:
> On 02/04/2014 03:39 PM, Mark Hatle wrote:
>> [YOCTO #5436]
>>
>> Automatic selection of static uid/gid is needed for a dynamically generated
>> passwd and group file to have a deterministic outcome.
>>
>> When a package is installed and instructs the system to add a new user or
>> group, unless it selects a static uid/gid value, the next available uid/gid
>> will be used. The order in which packages are installed is dynamically
>> computed, and may change from one installation to the next. This results
>> in a non-deterministic set of uid/gid values.
>>
>> Enabling USERADD_REWRITE_PARAMS, and providing a preconfigured passwd/group
>> file will allow the continued dynamic generation of the passwd/group file
>> on the target, but ensure a deterministic outcome. (Dynamic generation is
>> desired so that user and groups that have no corresponding functionality
>> are not present within the final system image.)
>>
>> The rewrite params function will override each of the fields in the
>> useradd and groupadd calls with the values specified. Note, the password
>> field is ignored as is the member groups field in the group file. If the
>> field is empty, the value will not be overridden. (Note, there is no way
>> to 'blank' a field, as this would only generally affect the 'comment' field
>> and there really is no reason to blank it.)
>>
>> Enabling USERADD_ERROR_DYNAMIC will cause packages without static uid/gid
>> to generate an error and be skipped for the purpose of building. This is
>> used to prevent non-deterministic behavior.
>>
>> USERADD_UID_TABLES and USERADD_GID_TABLES may be used to specify the name
>> of the passwd and group files. By default they are assumed to be
>> 'files/passwd' and 'files/group'. Layers are searched in BBPATH order.
>>
>> Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
>> ---
>> meta/classes/useradd.bbclass | 258 +++++++++++++++++++++++++++++++++++
>> meta/conf/local.conf.sample.extended | 24 ++++
>> 2 files changed, 282 insertions(+)
>>
>
> This need to be split to 2 patches one for oe-core and the
> local.conf.sample.extended for poky.
>
This patch is specific to oe-core, and was NOT generated to apply to poky.
This includes the changes to the local.conf.sample.extended (located in
meta/conf on oe-core).
http://git.openembedded.org/openembedded-core/tree/meta/conf/local.conf.sample.extended
Once this goes into oe-core, I can work on a patch for meta-yocto if desired,
likely the hunk will just apply there as well.
--Mark
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [v2 PATCH 2/2] useradd.bbclass: Add ability to select a static uid/gid automatically
2014-02-04 23:39 ` [v2 PATCH 2/2] useradd.bbclass: Add ability to select a static uid/gid automatically Mark Hatle
2014-02-05 18:01 ` Saul Wold
@ 2014-02-06 22:17 ` Richard Purdie
1 sibling, 0 replies; 6+ messages in thread
From: Richard Purdie @ 2014-02-06 22:17 UTC (permalink / raw)
To: Mark Hatle; +Cc: openembedded-core
On Tue, 2014-02-04 at 17:39 -0600, Mark Hatle wrote:
> [YOCTO #5436]
>
> Automatic selection of static uid/gid is needed for a dynamically generated
> passwd and group file to have a deterministic outcome.
>
> When a package is installed and instructs the system to add a new user or
> group, unless it selects a static uid/gid value, the next available uid/gid
> will be used. The order in which packages are installed is dynamically
> computed, and may change from one installation to the next. This results
> in a non-deterministic set of uid/gid values.
>
> Enabling USERADD_REWRITE_PARAMS, and providing a preconfigured passwd/group
> file will allow the continued dynamic generation of the passwd/group file
> on the target, but ensure a deterministic outcome. (Dynamic generation is
> desired so that user and groups that have no corresponding functionality
> are not present within the final system image.)
>
> The rewrite params function will override each of the fields in the
> useradd and groupadd calls with the values specified. Note, the password
> field is ignored as is the member groups field in the group file. If the
> field is empty, the value will not be overridden. (Note, there is no way
> to 'blank' a field, as this would only generally affect the 'comment' field
> and there really is no reason to blank it.)
>
> Enabling USERADD_ERROR_DYNAMIC will cause packages without static uid/gid
> to generate an error and be skipped for the purpose of building. This is
> used to prevent non-deterministic behavior.
>
> USERADD_UID_TABLES and USERADD_GID_TABLES may be used to specify the name
> of the passwd and group files. By default they are assumed to be
> 'files/passwd' and 'files/group'. Layers are searched in BBPATH order.
>
> Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
> ---
> meta/classes/useradd.bbclass | 258 +++++++++++++++++++++++++++++++++++
> meta/conf/local.conf.sample.extended | 24 ++++
> 2 files changed, 282 insertions(+)
I know this has sat in limbo for a while, apologies for that. This patch
does look fairly clean and standalone, to the point I think this should
become a separate class in its own right.
We can do this with something like:
useradd.bbclass:
+inherit ${USERADDEXTENSIONS}
and then set
USERADDEXTENSIONS = "useradd-staticids"
to enable the code. I think the rest of it may work as is as a file
useradd-staticids.bbclass as long as the inherit is at the end of
useradd.bbclass so the anonumous python stacks the correct way around.
Bitbake can cope with a variable that expands to empty.
I haven't gone over the python code in detail and if its a standalone
class, I'm feeling less need to do so.
Cheers,
Richard
> diff --git a/meta/classes/useradd.bbclass b/meta/classes/useradd.bbclass
> index ad6f61a..ff14b27 100644
> --- a/meta/classes/useradd.bbclass
> +++ b/meta/classes/useradd.bbclass
> @@ -144,9 +144,267 @@ def update_useradd_after_parse(d):
> if not d.getVar('USERADD_PARAM_%s' % pkg, True) and not d.getVar('GROUPADD_PARAM_%s' % pkg, True) and not d.getVar('GROUPMEMS_PARAM_%s' % pkg, True):
> bb.fatal("%s inherits useradd but doesn't set USERADD_PARAM, GROUPADD_PARAM or GROUPMEMS_PARAM for package %s" % (d.getVar('FILE'), pkg))
>
> +# In order to support a deterministic set of 'dynamic' users/groups,
> +# we need a function to reformat the params based on a static file
> +def update_useradd_static_config(d):
> + import argparse
> + import re
> +
> + class myArgumentParser( argparse.ArgumentParser ):
> + def _print_message(self, message, file=None):
> + bb.warn("%s - %s: %s" % (d.getVar('PN', True), pkg, message))
> +
> + # This should never be called...
> + def exit(self, status=0, message=None):
> + message = message or ("%s - %s: useradd.bbclass: Argument parsing exited" % (d.getVar('PN', True), pkg))
> + error(message)
> +
> + def error(self, message):
> + raise bb.build.FuncFailed(message)
> +
> + # We parse and rewrite the useradd components
> + def rewrite_useradd(params):
> + # The following comes from --help on useradd from shadow
> + parser = myArgumentParser(prog='useradd')
> + parser.add_argument("-b", "--base-dir", metavar="BASE_DIR", help="base directory for the home directory of the new account")
> + parser.add_argument("-c", "--comment", metavar="COMMENT", help="GECOS field of the new account")
> + parser.add_argument("-d", "--home-dir", metavar="HOME_DIR", help="home directory of the new account")
> + parser.add_argument("-D", "--defaults", help="print or change default useradd configuration", action="store_true")
> + parser.add_argument("-e", "--expiredate", metavar="EXPIRE_DATE", help="expiration date of the new account")
> + parser.add_argument("-f", "--inactive", metavar="INACTIVE", help="password inactivity period of the new account")
> + parser.add_argument("-g", "--gid", metavar="GROUP", help="name or ID of the primary group of the new account")
> + parser.add_argument("-G", "--groups", metavar="GROUPS", help="list of supplementary groups of the new account")
> + parser.add_argument("-k", "--skel", metavar="SKEL_DIR", help="use this alternative skeleton directory")
> + parser.add_argument("-K", "--key", metavar="KEY=VALUE", help="override /etc/login.defs defaults")
> + parser.add_argument("-l", "--no-log-init", help="do not add the user to the lastlog and faillog databases", action="store_true")
> + parser.add_argument("-m", "--create-home", help="create the user's home directory", action="store_true")
> + parser.add_argument("-M", "--no-create-home", help="do not create the user's home directory", action="store_true")
> + parser.add_argument("-N", "--no-user-group", help="do not create a group with the same name as the user", action="store_true")
> + parser.add_argument("-o", "--non-unique", help="allow to create users with duplicate (non-unique UID)", action="store_true")
> + parser.add_argument("-p", "--password", metavar="PASSWORD", help="encrypted password of the new account")
> + parser.add_argument("-R", "--root", metavar="CHROOT_DIR", help="directory to chroot into")
> + parser.add_argument("-r", "--system", help="create a system account", action="store_true")
> + parser.add_argument("-s", "--shell", metavar="SHELL", help="login shell of the new account")
> + parser.add_argument("-u", "--uid", metavar="UID", help="user ID of the new account")
> + parser.add_argument("-U", "--user-group", help="create a group with the same name as the user", action="store_true")
> + parser.add_argument("LOGIN", help="Login name of the new user")
> +
> + # Return a list of configuration files based on either the default
> + # files/passwd or the contents of USERADD_UID_TABLES
> + # paths are resulved via BBPATH
> + def get_passwd_list(d):
> + str = ""
> + bbpath = d.getVar('BBPATH', True)
> + passwd_tables = d.getVar('USERADD_UID_TABLES', True)
> + if not passwd_tables:
> + passwd_tables = 'files/passwd'
> + for conf_file in passwd_tables.split():
> + str += " %s" % bb.utils.which(bbpath, conf_file)
> + return str
> +
> + newparams = []
> + for param in re.split('''[ \t]*;[ \t]*(?=(?:[^'"]|'[^']*'|"[^"]*")*$)''', params):
> + param=param.strip()
> + try:
> + uaargs = parser.parse_args(re.split('''[ \t]*(?=(?:[^'"]|'[^']*'|"[^"]*")*$)''', param))
> + except:
> + raise bb.build.FuncFailed("%s: Unable to parse arguments for USERADD_PARAM_%s: '%s'" % (d.getVar('PN', True), pkg, param))
> +
> + # files/passwd or the contents of USERADD_UID_TABLES
> + # Use the standard passwd layout:
> + # username:password:user_id:group_id:comment:home_directory:login_shell
> + # (we want to process in reverse order, as 'last found' in the list wins)
> + #
> + # If a field is left blank, the original value will be used. The 'username'
> + # field is required.
> + #
> + # Note: we ignore the password field, as including even the hashed password
> + # in the useradd command may introduce a security hole. It's assumed that
> + # all new users get the default ('*' which prevents login) until the user is
> + # specifically configured by the system admin.
> + for conf in get_passwd_list(d).split()[::-1]:
> + if os.path.exists(conf):
> + f = open(conf, "r")
> + for line in f:
> + if line.startswith('#'):
> + continue
> + field = line.rstrip().split(":")
> + if field[0] == uaargs.LOGIN:
> + if uaargs.uid and field[2] and (uaargs.uid != field[2]):
> + bb.warn("%s: Changing username %s's uid from (%s) to (%s), verify configuration files!" % (d.getVar('PN', True), uaargs.LOGIN, uaargs.uid, field[2]))
> + uaargs.uid = [field[2], uaargs.uid][not field[2]]
> +
> + # Determine the possible groupname
> + # Unless the group name (or gid) is specified, we assume that the LOGIN is the groupname
> + #
> + # By default the system has creation of the matching groups enabled
> + # So if the implicit username-group creation is on, then the implicit groupname (LOGIN)
> + # is used, and we disable the user_group option.
> + #
> + uaargs.groupname = [uaargs.gid, uaargs.LOGIN][not uaargs.gid or uaargs.user_group]
> + uaargs.user_group = False
> +
> + uaargs.gid = [uaargs.gid, uaargs.groupname][not uaargs.gid]
> + uaargs.gid = [field[3], uaargs.gid][not field[3]]
> +
> + if uaargs.groupname == uaargs.gid:
> + # Nothing to do...
> + pass
> + elif (uaargs.groupname and uaargs.groupname.isdigit()) and (uaargs.gid and uaargs.gid.isdigit()) and (uaargs.groupname != uaargs.gid):
> + # We want to add a group, but we don't know it's name... so we can't add the group...
> + # We have to assume the group has previously been added or we'll fail on the adduser...
> + # Note: specifying the actual gid is very rare in OE, usually the group name is specified.
> + bb.warn("%s: Changing gid for login %s from (%s) to (%s), verify configuration files!" % (d.getVar('PN', True), uaargs.LOGIN, uaargs.groupname, uaargs.gid))
> + elif uaargs.groupname and (uaargs.gid and uaargs.gid.isdigit()):
> + bb.debug(1, "Adding group %s gid (%s)!" % (uaargs.groupname, uaargs.gid))
> + groupadd = d.getVar("GROUPADD_PARAM_%s" % pkg, True)
> + newgroup = "-g %s %s" % (uaargs.gid, uaargs.groupname)
> + if groupadd:
> + d.setVar("GROUPADD_PARAM_%s" % pkg, "%s ; %s" % (groupadd, newgroup))
> + else:
> + d.setVar("GROUPADD_PARAM_%s" % pkg, newgroup)
> +
> + uaargs.comment = ["'%s'" % field[4], uaargs.comment][not field[4]]
> + uaargs.home_dir = [field[5], uaargs.home_dir][not field[5]]
> + uaargs.shell = [field[6], uaargs.shell][not field[6]]
> + break
> +
> + # Should be an error if a specific option is set...
> + if d.getVar('USERADD_ERROR_DYNAMIC', True) == '1' and (not uaargs.uid or not uaargs.gid):
> + raise bb.build.FuncFailed("%s - %s: Username %s does not have a static uid/gid defined." % (d.getVar('PN', True), pkg, uaargs.LOGIN))
> +
> + # Reconstruct the args...
> + newparam = ['', ' --defaults'][uaargs.defaults]
> + newparam += ['', ' --base-dir %s' % uaargs.base_dir][uaargs.base_dir != None]
> + newparam += ['', ' --comment %s' % uaargs.comment][uaargs.comment != None]
> + newparam += ['', ' --home-dir %s' % uaargs.home_dir][uaargs.home_dir != None]
> + newparam += ['', ' --expiredata %s' % uaargs.expiredate][uaargs.expiredate != None]
> + newparam += ['', ' --inactive %s' % uaargs.inactive][uaargs.inactive != None]
> + newparam += ['', ' --gid %s' % uaargs.gid][uaargs.gid != None]
> + newparam += ['', ' --groups %s' % uaargs.groups][uaargs.groups != None]
> + newparam += ['', ' --skel %s' % uaargs.skel][uaargs.skel != None]
> + newparam += ['', ' --key %s' % uaargs.key][uaargs.key != None]
> + newparam += ['', ' --no-log-init'][uaargs.no_log_init]
> + newparam += ['', ' --create-home'][uaargs.create_home]
> + newparam += ['', ' --no-create-home'][uaargs.no_create_home]
> + newparam += ['', ' --no-user-group'][uaargs.no_user_group]
> + newparam += ['', ' --non-unique'][uaargs.non_unique]
> + newparam += ['', ' --password %s' % uaargs.password][uaargs.password != None]
> + newparam += ['', ' --root %s' % uaargs.root][uaargs.root != None]
> + newparam += ['', ' --system'][uaargs.system]
> + newparam += ['', ' --shell %s' % uaargs.shell][uaargs.shell != None]
> + newparam += ['', ' --uid %s' % uaargs.uid][uaargs.uid != None]
> + newparam += ['', ' --user-group'][uaargs.user_group]
> + newparam += ' %s' % uaargs.LOGIN
> +
> + newparams.append(newparam)
> +
> + return " ;".join(newparams).strip()
> +
> + # We parse and rewrite the groupadd components
> + def rewrite_groupadd(params):
> + # The following comes from --help on groupadd from shadow
> + parser = myArgumentParser(prog='groupadd')
> + parser.add_argument("-f", "--force", help="exit successfully if the group already exists, and cancel -g if the GID is already used", action="store_true")
> + parser.add_argument("-g", "--gid", metavar="GID", help="use GID for the new group")
> + parser.add_argument("-K", "--key", metavar="KEY=VALUE", help="override /etc/login.defs defaults")
> + parser.add_argument("-o", "--non-unique", help="allow to create groups with duplicate (non-unique) GID", action="store_true")
> + parser.add_argument("-p", "--password", metavar="PASSWORD", help="use this encrypted password for the new group")
> + parser.add_argument("-R", "--root", metavar="CHROOT_DIR", help="directory to chroot into")
> + parser.add_argument("-r", "--system", help="create a system account", action="store_true")
> + parser.add_argument("GROUP", help="Group name of the new group")
> +
> + # Return a list of configuration files based on either the default
> + # files/group or the contents of USERADD_GID_TABLES
> + # paths are resulved via BBPATH
> + def get_group_list(d):
> + str = ""
> + bbpath = d.getVar('BBPATH', True)
> + group_tables = d.getVar('USERADD_GID_TABLES', True)
> + if not group_tables:
> + group_tables = 'files/group'
> + for conf_file in group_tables.split():
> + str += " %s" % bb.utils.which(bbpath, conf_file)
> + return str
> +
> + newparams = []
> + for param in re.split('''[ \t]*;[ \t]*(?=(?:[^'"]|'[^']*'|"[^"]*")*$)''', params):
> + param=param.strip()
> + try:
> + # If we're processing multiple lines, we could have left over values here...
> + gaargs = parser.parse_args(re.split('''[ \t]*(?=(?:[^'"]|'[^']*'|"[^"]*")*$)''', param))
> + except:
> + raise bb.build.FuncFailed("%s: Unable to parse arguments for GROUPADD_PARAM_%s: '%s'" % (d.getVar('PN', True), pkg, param))
> +
> + # Need to iterate over layers and open the right file(s)
> + # Use the standard group layout:
> + # groupname:password:group_id:group_members
> + #
> + # If a field is left blank, the original value will be used. The 'groupname' field
> + # is required.
> + #
> + # Note: similar to the passwd file, the 'password' filed is ignored
> + # Note: group_members is ignored, group members must be configured with the GROUPMEMS_PARAM
> + for conf in get_group_list(d).split()[::-1]:
> + if os.path.exists(conf):
> + f = open(conf, "r")
> + for line in f:
> + if line.startswith('#'):
> + continue
> + field = line.rstrip().split(":")
> + if field[0] == gaargs.GROUP and field[2]:
> + if gaargs.gid and (gaargs.gid != field[2]):
> + bb.warn("%s: Changing groupname %s's gid from (%s) to (%s), verify configuration files!" % (d.getVar('PN', True), gaargs.GROUP, gaargs.gid, field[2]))
> + gaargs.gid = field[2]
> + break
> +
> + if d.getVar('USERADD_ERROR_DYNAMIC', True) == '1' and not gaargs.gid:
> + raise bb.build.FuncFailed("%s - %s: Groupname %s does not have a static gid defined." % (d.getVar('PN', True), pkg, gaargs.GROUP))
> +
> + # Reconstruct the args...
> + newparam = ['', ' --force'][gaargs.force]
> + newparam += ['', ' --gid %s' % gaargs.gid][gaargs.gid != None]
> + newparam += ['', ' --key %s' % gaargs.key][gaargs.key != None]
> + newparam += ['', ' --non-unique'][gaargs.non_unique]
> + newparam += ['', ' --password %s' % gaargs.password][gaargs.password != None]
> + newparam += ['', ' --root %s' % gaargs.root][gaargs.root != None]
> + newparam += ['', ' --system'][gaargs.system]
> + newparam += ' %s' % gaargs.GROUP
> +
> + newparams.append(newparam)
> +
> + return " ;".join(newparams).strip()
> +
> + # Load and process the users and groups, rewriting the adduser/addgroup params
> + useradd_packages = d.getVar('USERADD_PACKAGES', True)
> +
> + for pkg in useradd_packages.split():
> + # Groupmems doesn't have anything we might want to change, so simply validating
> + # is a bit of a waste -- only process useradd/groupadd
> + useradd_param = d.getVar('USERADD_PARAM_%s' % pkg, True)
> + if useradd_param:
> + #bb.warn("Before: 'USERADD_PARAM_%s' - '%s'" % (pkg, useradd_param))
> + d.setVar('USERADD_PARAM_%s' % pkg, rewrite_useradd(useradd_param))
> + #bb.warn("After: 'USERADD_PARAM_%s' - '%s'" % (pkg, d.getVar('USERADD_PARAM_%s' % pkg, True)))
> +
> + groupadd_param = d.getVar('GROUPADD_PARAM_%s' % pkg, True)
> + if groupadd_param:
> + #bb.warn("Before: 'GROUPADD_PARAM_%s' - '%s'" % (pkg, groupadd_param))
> + d.setVar('GROUPADD_PARAM_%s' % pkg, rewrite_groupadd(groupadd_param))
> + #bb.warn("After: 'GROUPADD_PARAM_%s' - '%s'" % (pkg, d.getVar('GROUPADD_PARAM_%s' % pkg, True)))
> +
> +
> +
> python __anonymous() {
> if not bb.data.inherits_class('nativesdk', d):
> update_useradd_after_parse(d)
> +
> + if d.getVar('USERADD_REWRITE_PARAMS', True) == '1':
> + try:
> + update_useradd_static_config(d)
> + except bb.build.FuncFailed as f:
> + bb.debug(1, "Skipping recipe %s: %s" % (d.getVar('PN', True), f))
> + raise bb.parse.SkipPackage(f)
> }
>
> # Return a single [GROUP|USER]ADD_PARAM formatted string which includes the
> diff --git a/meta/conf/local.conf.sample.extended b/meta/conf/local.conf.sample.extended
> index c7c4f40..b73a0a7 100644
> --- a/meta/conf/local.conf.sample.extended
> +++ b/meta/conf/local.conf.sample.extended
> @@ -253,6 +253,30 @@
> #usermod -s /bin/sh tester; \
> #"
>
> +# Various packages dynamically add users and groups to the system at package
> +# install time. For programs that do not care what the uid/gid is of the
> +# resulting users/groups, the order of the install will determine the final
> +# uid/gid. This can lead to non-deterministic uid/gid values from one build
> +# to another. Use the following settings to specify that all user/group adds
> +# should be created based on a static passwd/group file.
> +#
> +# By default the system looks in the BBPATH for files/passwd and files/group
> +# the default can be overriden by spefying USERADD_UID/GID_TABLES.
> +#
> +# Note, if you change the value of the USERADD_REWRITE_PARAMS after starting
> +# to build. The TMPDIR may have incompatible uid/gids in it. You must 'rm'
> +# the TMPDIR to avoid this issue.
> +#
> +#USERADD_REWRITE_PARAMS = "1"
> +#USERADD_UID_TABLES = "files/passwd"
> +#USERADD_GID_TABLES = "files/group"
> +#
> +# In order to prevent generating a system where a dynamicly assigned uid/gid
> +# can exist, you can enable the following setting. This will force the
> +# system to error out if the user/group name is not defined in the
> +# files/passwd or files/group (or specified replacements.)
> +#USERADD_ERROR_DYNAMIC = "1"
> +
> # Enabling FORTRAN
> # Note this is not officially supported and is just illustrated here to
> # show an example of how it can be done
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2014-02-06 22:17 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2014-02-04 23:39 [v2 PATCH 0/2] Implement deterministic uid/gid Mark Hatle
2014-02-04 23:39 ` [v2 PATCH 1/2] useradd.bbclass: Fix build time install issues Mark Hatle
2014-02-04 23:39 ` [v2 PATCH 2/2] useradd.bbclass: Add ability to select a static uid/gid automatically Mark Hatle
2014-02-05 18:01 ` Saul Wold
2014-02-05 18:14 ` Mark Hatle
2014-02-06 22:17 ` Richard Purdie
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox