* [PATCH 0/1] remove startup questions @ 2015-10-02 3:10 brian avery 2015-10-02 3:10 ` [PATCH 1/1] toaster: improve ease of toaster startup brian avery 0 siblings, 1 reply; 4+ messages in thread From: brian avery @ 2015-10-02 3:10 UTC (permalink / raw) To: toaster This removes the startup questions from toaster and uses the following dir structure. cwd/toaster.sqlite cwd/_toaster_clones cwd/build You can override TOASTER_CONF to point at the toasterconf.json in meta. There are still a fair number of directory assumptions that need to be worked out for 2.1 but this should let us start up easier and automate tests better. Since we are putting the builddir here, we should really allow users to set the SSTATE_DIR and DL_DIR variables. I will try to add that as part of the change needed for 7880 since it has to set a number of new variables. -b The following changes since commit 3143235ff04998bad4a2af3e8fe0361664f65aa2: toaster: exclude recipes with empty names (2015-10-01 14:14:10 -0700) are available in the git repository at: git://git.yoctoproject.org/poky-contrib bavery/toaster/startup_simplification_8217 http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=bavery/toaster/startup_simplification_8217 brian avery (1): toaster: improve ease of toaster startup bitbake/bin/toaster | 34 ++++++-- .../management/commands/checksettings.py | 98 +++++----------------- 2 files changed, 49 insertions(+), 83 deletions(-) -- 1.9.1 ^ permalink raw reply [flat|nested] 4+ messages in thread
* [PATCH 1/1] toaster: improve ease of toaster startup 2015-10-02 3:10 [PATCH 0/1] remove startup questions brian avery @ 2015-10-02 3:10 ` brian avery 2015-10-02 13:52 ` Ed Bartosh 0 siblings, 1 reply; 4+ messages in thread From: brian avery @ 2015-10-02 3:10 UTC (permalink / raw) To: toaster * presumes that toaster will be centred on dir where you started toaster * will build in cwd/build * will checkout new layers to cwd/_toaster_clones * will make database in cwd/toaster.sqlite [YOCTO #8217] Signed-off-by: brian avery <avery.brian@gmail.com> --- bitbake/bin/toaster | 34 ++++++-- .../management/commands/checksettings.py | 98 +++++----------------- 2 files changed, 49 insertions(+), 83 deletions(-) diff --git a/bitbake/bin/toaster b/bitbake/bin/toaster index ac27826..ce04a60 100755 --- a/bitbake/bin/toaster +++ b/bitbake/bin/toaster @@ -54,11 +54,10 @@ webserverStartAll() fi retval=0 - if [ "$TOASTER_MANAGED" '=' '1' ]; then - python $BBBASEDIR/lib/toaster/manage.py syncdb || retval=1 - else - python $BBBASEDIR/lib/toaster/manage.py syncdb --noinput || retval=1 - fi + # you can always add a superuser later via + # python bitbake/lib/toaster/manage.py python manage.py createsuperuser --username=<ME> + python $BBBASEDIR/lib/toaster/manage.py syncdb --noinput || retval=1 + python $BBBASEDIR/lib/toaster/manage.py migrate orm || retval=2 if [ $retval -eq 1 ]; then echo "Failed db sync, stopping system start" 1>&2 @@ -163,7 +162,30 @@ RUNNING=0 NOTOASTERUI=0 WEBSERVER=1 TOASTER_BRBE="" -WEB_PORT="8000" +if [ "$WEB_PORT" == "" ]; then + WEB_PORT="8000" +fi +# this is the configuraton file we are using for toaster +# note default is assuming yocto. Override this if you are +# running in a pure OE environment and use the toasterconf.json +# in meta/conf/toasterconf.json +# note: for future there are a number of relative path assumptions +# in the local layers that currently prevent using an arbitrary +# toasterconf.json +if [ "$TOASTER_CONF" == "" ]; then + BIN_DIR=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd ) + export TOASTER_CONF="$BIN_DIR/../../meta-yocto/conf/toasterconf.json" +fi +# this defines the dir toaster will use for +# 1) clones of layers (in _toaster_clones ) +# 2) the build dir (in build) +# 3) the sqlite db if that is being used. +# 4) pid's we need to clean up on exit/shutdown +# note: for future. in order to make this an arbitrary directory, we need to +# make sure that the toaster.sqlite file doesn't default to `pwd` like it currently does. +export TOASTER_DIR=`pwd` + + NOBROWSER=0 for param in $*; do diff --git a/bitbake/lib/toaster/bldcontrol/management/commands/checksettings.py b/bitbake/lib/toaster/bldcontrol/management/commands/checksettings.py index b2c573c..2ba61b5 100644 --- a/bitbake/lib/toaster/bldcontrol/management/commands/checksettings.py +++ b/bitbake/lib/toaster/bldcontrol/management/commands/checksettings.py @@ -65,17 +65,9 @@ class Command(NoArgsCommand): return DN(self._find_first_path_for_file(DN(self.guesspath), "bblayers.conf", 4)) def _verify_build_environment(self): - # refuse to start if we have no build environments - while BuildEnvironment.objects.count() == 0: - print(" !! No build environments found. Toaster needs at least one build environment in order to be able to run builds.\n" + - "You can manually define build environments in the database table bldcontrol_buildenvironment.\n" + - "Or Toaster can define a simple localhost-based build environment for you.") - - i = raw_input(" -- Do you want to create a basic localhost build environment ? (Y/n) "); - if not len(i) or i.startswith("y") or i.startswith("Y"): - BuildEnvironment.objects.create(pk = 1, betype = 0) - else: - raise Exception("Toaster cannot start without build environments. Aborting.") + # provide a local build env. This will be extended later to include non local + if BuildEnvironment.objects.count() == 0: + BuildEnvironment.objects.create(pk = 1, betype = 0) # we make sure we have builddir and sourcedir for all defined build envionments @@ -83,16 +75,9 @@ class Command(NoArgsCommand): be.needs_import = False def _verify_be(): is_changed = False - print("\nVerifying the build environment. If the local build environment is not properly configured, you will be asked to configure it.") def _update_sourcedir(): - suggesteddir = self._get_suggested_sourcedir(be) - if len(suggesteddir) > 0: - be.sourcedir = raw_input("This is the directory Toaster uses to check out the source code of the layers you will build. Toaster will create new clones of the layers, so existing content in the chosen directory will not be changed.\nToaster suggests you use \"%s\" as your layers checkout directory. If you select this directory, a layer like \"meta-intel\" will end up in \"%s/meta-intel\".\nPress Enter to select \"%s\" or type the full path to a different directory. If you provide your own directory, it must be a parent of the cloned directory for the sources you are using to run Toaster: " % (suggesteddir, suggesteddir, suggesteddir)) - else: - be.sourcedir = raw_input("Toaster needs to know in which directory it should check out the source code of the layers you will build. The directory should be a parent of the cloned directory for the sources you are using to run Toaster. Toaster will create new clones of the layers, so existing content in the chosen directory will not be changed.\nType the full path to the directory (for example: \"%s\": " % os.environ.get('HOME', '/tmp/')) - if len(be.sourcedir) == 0 and len(suggesteddir) > 0: - be.sourcedir = suggesteddir + be.sourcedir = os.environ.get('TOASTER_DIR') return True if len(be.sourcedir) == 0: @@ -103,23 +88,13 @@ class Command(NoArgsCommand): print "\n -- Validation: The layers checkout directory must be set to an absolute path." is_changed = _update_sourcedir() - if not be.sourcedir in DN(__file__): - print "\n -- Validation: The layers checkout directory must be a parent of the current checkout." - is_changed = _update_sourcedir() - if is_changed: if be.betype == BuildEnvironment.TYPE_LOCAL: be.needs_import = True return True def _update_builddir(): - suggesteddir = self._get_suggested_builddir(be) - if len(suggesteddir) > 0: - be.builddir = raw_input("Toaster needs to know where your build directory is located.\nThe build directory is where all the artifacts created by your builds will be stored. Toaster suggests \"%s\".\nPress Enter to select \"%s\" or type the full path to a different directory: " % (suggesteddir, suggesteddir)) - else: - be.builddir = raw_input("Toaster needs to know where is your build directory.\nThe build directory is where all the artifacts created by your builds will be stored. Type the full path to the directory (for example: \" %s/build\")" % os.environ.get('HOME','/tmp/')) - if len(be.builddir) == 0 and len(suggesteddir) > 0: - be.builddir = suggesteddir + be.builddir = os.environ.get('TOASTER_DIR')+"/build" return True if len(be.builddir) == 0: @@ -138,53 +113,22 @@ class Command(NoArgsCommand): if be.needs_import: - print "\nToaster can use a SINGLE predefined configuration file to set up default project settings and layer information sources.\n" - - # find configuration files - config_files = [] - for dirname in self._recursive_list_directories(be.sourcedir,2): - if os.path.exists(os.path.join(dirname, ".templateconf")): - import subprocess - proc = subprocess.Popen('bash -c ". '+os.path.join(dirname, ".templateconf")+'; echo \"\$TEMPLATECONF\""', shell=True, stdout=subprocess.PIPE) - conffilepath, stderroroutput = proc.communicate() - proc.wait() - if proc.returncode != 0: - raise Exception("Failed to source TEMPLATECONF: %s" % stderroroutput) - - conffilepath = os.path.join(conffilepath.strip(), "toasterconf.json") - candidatefilepath = os.path.join(dirname, conffilepath) - if "toaster_cloned" in candidatefilepath: - continue - if os.path.exists(candidatefilepath): - config_files.append(candidatefilepath) - - if len(config_files) > 0: - print "Toaster will list now the configuration files that it found. Select the number to use the desired configuration file." - for cf in config_files: - print " [%d] - %s" % (config_files.index(cf) + 1, cf) - print "\n [0] - Exit without importing any file" - try: - i = raw_input("\nEnter your option: ") - if len(i) and (int(i) - 1 >= 0 and int(i) - 1 < len(config_files)): - print "\nImporting file: %s" % config_files[int(i)-1] - from loadconf import Command as LoadConfigCommand - - LoadConfigCommand()._import_layer_config(config_files[int(i)-1]) - # we run lsupdates after config update - print "\nLayer configuration imported. Updating information from the layer sources, please wait.\nYou can re-update any time later by running bitbake/lib/toaster/manage.py lsupdates" - from django.core.management import call_command - call_command("lsupdates") - - # we don't look for any other config files - return is_changed - except Exception as e: - print "Failure while trying to import the toaster config file: %s" % e - traceback.print_exc(e) - else: - print "\nToaster could not find a configuration file. You need to configure Toaster manually using the web interface, or create a configuration file and use\n bitbake/lib/toaster/managepy.py loadconf [filename]\n command to load it. You can use https://wiki.yoctoproject.org/wiki/File:Toasterconf.json.txt.patch as a starting point." - - - + try: + config_file=os.environ.get('TOASTER_CONF') + print "\nImporting file: %s" % config_file + from loadconf import Command as LoadConfigCommand + + LoadConfigCommand()._import_layer_config(config_file) + # we run lsupdates after config update + print "\nLayer configuration imported. Updating information from the layer sources, please wait.\nYou can re-update any time later by running bitbake/lib/toaster/manage.py lsupdates" + from django.core.management import call_command + call_command("lsupdates") + + # we don't look for any other config files + return is_changed + except Exception as e: + print "Failure while trying to import the toaster config file %s: %s" % (config_file,e) + traceback.print_exc(e) return is_changed -- 1.9.1 ^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [PATCH 1/1] toaster: improve ease of toaster startup 2015-10-02 3:10 ` [PATCH 1/1] toaster: improve ease of toaster startup brian avery @ 2015-10-02 13:52 ` Ed Bartosh 2015-10-02 15:12 ` Brian Avery 0 siblings, 1 reply; 4+ messages in thread From: Ed Bartosh @ 2015-10-02 13:52 UTC (permalink / raw) To: brian avery; +Cc: toaster Hi Brian, First of all I'd suggest splitting your patch to several commits. There are a lot of trailing spaces in the code. please, remove them. Running pylint on python code and fixing its findings would be good too. My further comments are below. On Thu, Oct 01, 2015 at 08:10:58PM -0700, brian avery wrote: > * presumes that toaster will be centred on dir where you started toaster > > * will build in cwd/build > > * will checkout new layers to cwd/_toaster_clones > > * will make database in cwd/toaster.sqlite > > [YOCTO #8217] > > Signed-off-by: brian avery <avery.brian@gmail.com> > --- > bitbake/bin/toaster | 34 ++++++-- > .../management/commands/checksettings.py | 98 +++++----------------- > 2 files changed, 49 insertions(+), 83 deletions(-) > > diff --git a/bitbake/bin/toaster b/bitbake/bin/toaster > index ac27826..ce04a60 100755 > --- a/bitbake/bin/toaster > +++ b/bitbake/bin/toaster > @@ -54,11 +54,10 @@ webserverStartAll() > fi > > retval=0 > - if [ "$TOASTER_MANAGED" '=' '1' ]; then > - python $BBBASEDIR/lib/toaster/manage.py syncdb || retval=1 > - else > - python $BBBASEDIR/lib/toaster/manage.py syncdb --noinput || retval=1 > - fi > + # you can always add a superuser later via > + # python bitbake/lib/toaster/manage.py python manage.py createsuperuser --username=<ME> > + python $BBBASEDIR/lib/toaster/manage.py syncdb --noinput || retval=1 > + > python $BBBASEDIR/lib/toaster/manage.py migrate orm || retval=2 > if [ $retval -eq 1 ]; then > echo "Failed db sync, stopping system start" 1>&2 > @@ -163,7 +162,30 @@ RUNNING=0 > NOTOASTERUI=0 > WEBSERVER=1 > TOASTER_BRBE="" > -WEB_PORT="8000" > +if [ "$WEB_PORT" == "" ]; then > + WEB_PORT="8000" > +fi > +# this is the configuraton file we are using for toaster > +# note default is assuming yocto. Override this if you are > +# running in a pure OE environment and use the toasterconf.json > +# in meta/conf/toasterconf.json > +# note: for future there are a number of relative path assumptions > +# in the local layers that currently prevent using an arbitrary > +# toasterconf.json > +if [ "$TOASTER_CONF" == "" ]; then > + BIN_DIR=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd ) This would be more clear I think: BIN_DIR=$(dirname "${BASH_SOURCE[0]}") cd $BIN_DIR > + export TOASTER_CONF="$BIN_DIR/../../meta-yocto/conf/toasterconf.json" we should probably check if this file exists > +fi > +# this defines the dir toaster will use for > +# 1) clones of layers (in _toaster_clones ) > +# 2) the build dir (in build) > +# 3) the sqlite db if that is being used. > +# 4) pid's we need to clean up on exit/shutdown > +# note: for future. in order to make this an arbitrary directory, we need to > +# make sure that the toaster.sqlite file doesn't default to `pwd` like it currently does. > +export TOASTER_DIR=`pwd` > + > + > NOBROWSER=0 > > for param in $*; do > diff --git a/bitbake/lib/toaster/bldcontrol/management/commands/checksettings.py b/bitbake/lib/toaster/bldcontrol/management/commands/checksettings.py > index b2c573c..2ba61b5 100644 > --- a/bitbake/lib/toaster/bldcontrol/management/commands/checksettings.py > +++ b/bitbake/lib/toaster/bldcontrol/management/commands/checksettings.py > @@ -65,17 +65,9 @@ class Command(NoArgsCommand): > return DN(self._find_first_path_for_file(DN(self.guesspath), "bblayers.conf", 4)) > > def _verify_build_environment(self): > - # refuse to start if we have no build environments > - while BuildEnvironment.objects.count() == 0: > - print(" !! No build environments found. Toaster needs at least one build environment in order to be able to run builds.\n" + > - "You can manually define build environments in the database table bldcontrol_buildenvironment.\n" + > - "Or Toaster can define a simple localhost-based build environment for you.") > - > - i = raw_input(" -- Do you want to create a basic localhost build environment ? (Y/n) "); > - if not len(i) or i.startswith("y") or i.startswith("Y"): > - BuildEnvironment.objects.create(pk = 1, betype = 0) > - else: > - raise Exception("Toaster cannot start without build environments. Aborting.") > + # provide a local build env. This will be extended later to include non local > + if BuildEnvironment.objects.count() == 0: > + BuildEnvironment.objects.create(pk = 1, betype = 0) BuildEnvironment.objects.create(betype=BuildEnvironment.TYPE_LOCAL) > > > # we make sure we have builddir and sourcedir for all defined build envionments > @@ -83,16 +75,9 @@ class Command(NoArgsCommand): > be.needs_import = False > def _verify_be(): > is_changed = False > - print("\nVerifying the build environment. If the local build environment is not properly configured, you will be asked to configure it.") > > def _update_sourcedir(): > - suggesteddir = self._get_suggested_sourcedir(be) > - if len(suggesteddir) > 0: > - be.sourcedir = raw_input("This is the directory Toaster uses to check out the source code of the layers you will build. Toaster will create new clones of the layers, so existing content in the chosen directory will not be changed.\nToaster suggests you use \"%s\" as your layers checkout directory. If you select this directory, a layer like \"meta-intel\" will end up in \"%s/meta-intel\".\nPress Enter to select \"%s\" or type the full path to a different directory. If you provide your own directory, it must be a parent of the cloned directory for the sources you are using to run Toaster: " % (suggesteddir, suggesteddir, suggesteddir)) > - else: > - be.sourcedir = raw_input("Toaster needs to know in which directory it should check out the source code of the layers you will build. The directory should be a parent of the cloned directory for the sources you are using to run Toaster. Toaster will create new clones of the layers, so existing content in the chosen directory will not be changed.\nType the full path to the directory (for example: \"%s\": " % os.environ.get('HOME', '/tmp/')) > - if len(be.sourcedir) == 0 and len(suggesteddir) > 0: > - be.sourcedir = suggesteddir > + be.sourcedir = os.environ.get('TOASTER_DIR') > return True > > if len(be.sourcedir) == 0: > @@ -103,23 +88,13 @@ class Command(NoArgsCommand): > print "\n -- Validation: The layers checkout directory must be set to an absolute path." > is_changed = _update_sourcedir() > > - if not be.sourcedir in DN(__file__): > - print "\n -- Validation: The layers checkout directory must be a parent of the current checkout." > - is_changed = _update_sourcedir() > - > if is_changed: > if be.betype == BuildEnvironment.TYPE_LOCAL: > be.needs_import = True > return True > > def _update_builddir(): > - suggesteddir = self._get_suggested_builddir(be) > - if len(suggesteddir) > 0: > - be.builddir = raw_input("Toaster needs to know where your build directory is located.\nThe build directory is where all the artifacts created by your builds will be stored. Toaster suggests \"%s\".\nPress Enter to select \"%s\" or type the full path to a different directory: " % (suggesteddir, suggesteddir)) > - else: > - be.builddir = raw_input("Toaster needs to know where is your build directory.\nThe build directory is where all the artifacts created by your builds will be stored. Type the full path to the directory (for example: \" %s/build\")" % os.environ.get('HOME','/tmp/')) > - if len(be.builddir) == 0 and len(suggesteddir) > 0: > - be.builddir = suggesteddir > + be.builddir = os.environ.get('TOASTER_DIR')+"/build" > return True > > if len(be.builddir) == 0: > @@ -138,53 +113,22 @@ class Command(NoArgsCommand): > > > if be.needs_import: > - print "\nToaster can use a SINGLE predefined configuration file to set up default project settings and layer information sources.\n" > - > - # find configuration files > - config_files = [] > - for dirname in self._recursive_list_directories(be.sourcedir,2): > - if os.path.exists(os.path.join(dirname, ".templateconf")): > - import subprocess > - proc = subprocess.Popen('bash -c ". '+os.path.join(dirname, ".templateconf")+'; echo \"\$TEMPLATECONF\""', shell=True, stdout=subprocess.PIPE) > - conffilepath, stderroroutput = proc.communicate() > - proc.wait() > - if proc.returncode != 0: > - raise Exception("Failed to source TEMPLATECONF: %s" % stderroroutput) > - > - conffilepath = os.path.join(conffilepath.strip(), "toasterconf.json") > - candidatefilepath = os.path.join(dirname, conffilepath) > - if "toaster_cloned" in candidatefilepath: > - continue > - if os.path.exists(candidatefilepath): > - config_files.append(candidatefilepath) > - > - if len(config_files) > 0: > - print "Toaster will list now the configuration files that it found. Select the number to use the desired configuration file." > - for cf in config_files: > - print " [%d] - %s" % (config_files.index(cf) + 1, cf) > - print "\n [0] - Exit without importing any file" > - try: > - i = raw_input("\nEnter your option: ") > - if len(i) and (int(i) - 1 >= 0 and int(i) - 1 < len(config_files)): > - print "\nImporting file: %s" % config_files[int(i)-1] > - from loadconf import Command as LoadConfigCommand > - > - LoadConfigCommand()._import_layer_config(config_files[int(i)-1]) > - # we run lsupdates after config update > - print "\nLayer configuration imported. Updating information from the layer sources, please wait.\nYou can re-update any time later by running bitbake/lib/toaster/manage.py lsupdates" > - from django.core.management import call_command > - call_command("lsupdates") > - > - # we don't look for any other config files > - return is_changed > - except Exception as e: > - print "Failure while trying to import the toaster config file: %s" % e > - traceback.print_exc(e) > - else: > - print "\nToaster could not find a configuration file. You need to configure Toaster manually using the web interface, or create a configuration file and use\n bitbake/lib/toaster/managepy.py loadconf [filename]\n command to load it. You can use https://wiki.yoctoproject.org/wiki/File:Toasterconf.json.txt.patch as a starting point." > - > - > - > + try: > + config_file=os.environ.get('TOASTER_CONF') > + print "\nImporting file: %s" % config_file > + from loadconf import Command as LoadConfigCommand > + > + LoadConfigCommand()._import_layer_config(config_file) > + # we run lsupdates after config update > + print "\nLayer configuration imported. Updating information from the layer sources, please wait.\nYou can re-update any time later by running bitbake/lib/toaster/manage.py lsupdates" > + from django.core.management import call_command > + call_command("lsupdates") > + > + # we don't look for any other config files > + return is_changed > + except Exception as e: > + print "Failure while trying to import the toaster config file %s: %s" % (config_file,e) > + traceback.print_exc(e) > > return is_changed > -- Regards, Ed ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [PATCH 1/1] toaster: improve ease of toaster startup 2015-10-02 13:52 ` Ed Bartosh @ 2015-10-02 15:12 ` Brian Avery 0 siblings, 0 replies; 4+ messages in thread From: Brian Avery @ 2015-10-02 15:12 UTC (permalink / raw) To: Ed Bartosh; +Cc: toaster Hi, split/spaces/pylint. will do and will send v2. BIN_DIR: sure, it's a confusing line as written :0 checking for the file is problematic since we don't have a fallback. If it's not there we start without a conf file and people just have to add layers by hand. I can check and make sure it behaves that way if the file is missing or the env var is empty. >BuildEnvironment.objects.create(betype=BuildEnvironment.TYPE_LOCAL) better! thanks for looking it over :) -b ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2015-10-02 15:12 UTC | newest] Thread overview: 4+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2015-10-02 3:10 [PATCH 0/1] remove startup questions brian avery 2015-10-02 3:10 ` [PATCH 1/1] toaster: improve ease of toaster startup brian avery 2015-10-02 13:52 ` Ed Bartosh 2015-10-02 15:12 ` Brian Avery
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.