README.md | 317 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 317 insertions(+)
create mode 100644 README.md
diff --git a/README.md b/README.md
new file mode 100644
index 0000000..b50061d
--- /dev/null
+++ b/README.md
@@ -0,0 +1,317 @@
+## Directories
+
+- `git/trash`: Directory for storing successfully finished and not
+ published builds.
+
+- `git/mirror`: *Optional*, directory for storing repo mirrors. If
+ repos are present in this directory, `git config` will do a local
+ clone on the repo instead of fetching from remote.
+
+## Json File Containing Repository Data
+
+In order to build a target, user needs to create a json file to hold
+the data of the repos that are required by the target (e.g. poky,
+bitbake, meta-openembbeded, etc.). All available repo data can be
+found in the `repo-defaults` entry in the `config.json` file. User can
+then copy the desired repo data and create a new json file consisting
+of only the repo data. An example of this would be:
+
+```json
+{
+ "poky" : {
+ "url" : "git://git.yoctoproject.org/poky",
+ "branch" : "master",
+ "revision" : "HEAD",
+ "checkout-dirname" : ".",
+ "no-layer-add" : true,
+ "call-init" : true
+ },
+ "bitbake" : {
+ "url" : "git://git.openembedded.org/bitbake",
+ "branch" : "master",
+ "revision" : "HEAD",
+ "no-layer-add" : true
+ }
+}
+```
+
+In the subsequent sections, this file will be referred to as **`repos.json`**.
+
+## Entry Point Scripts and Usage
+
+### `scripts/prepare-shared-repos`
+
+This script will fetch repos specified by `repos.json` and cache them
+in a custom directory. The cached repos can then be reused for future
+builds.
+
+Args:
+
+1. `repojson`: The path to `repos.json` containing repository data
+ explained in the previous section.
+
+2. `sharedsrcdir`: The directory where the repos will be cached.
+
+3. `-p`, `--publish-dir`: *Optional*, where to publish artefacts
+ to. Fetched repos would be archived as tarball and stored in this
+ directory. This procedure will not happen if `publish-dir` is not
+ specified.
+
+### `scripts/shared-repo-unpack`
+
+This script first fetches repos specified by `repos.json` *as well as*
+specified in the `NEEDREPOS` entry of the target in `config.json` to
+the autobuilder working directory given by user, then calls another
+script `scripts/layer-config`.
+
+`scripts/layer-config` will source `oe-init-build-env` file and
+execute `bitbake-layers add-layer` to add necessary layers (the
+`call-init` entry in `repos.json` for each repository will determine
+if `oe-init-build-env` is sourced, the `no-layer-add` entry will
+determine if `bitbake add-layer` should be run on this repo). Note
+that if `NEEDREPOS` is not set for the specific build target,
+`NEEDREPOS` in target `defaults` is checked and added. refer to
+`getconfigvar()` function in `scripts/utils.py` for details.
+
+Args:
+
+1. `repojson`: Same as in `scripts/prepare-shared-repos`.
+
+2. `abworkdir`: The autobuilder working directory which will resemble
+ the look of `poky` or `openembbeded-core` repo with build
+ directory, `oe-init-build-env`, etc.
+
+3. `target`: The target name defined under the `overrides` entry in
+ `config.json`.
+
+4. `-c`, `--cache-dir`: *Optional*, path to the directory caching the
+ repos fetch by `scripts/prepare-shared-repos`. If this argument is
+ not set, fetched repos will be stored in the `repos` directory
+ inside `abworkdir`.
+
+5. `-p`, `--publish-dir`: *Optional*, same as in
+ `scripts/prepare-shared-repos`.
+
+### `scripts/run-config`
+
+This script is the command responsible for sourcing the
+`oe-init-build-env` file, preparing for and executing `bitbake` to
+start the build, and running any sanity checks and extra commands
+specified by the target.
+
+First, build history related tasks are done. Build history related
+entries in `config.json` are checked. Specifically, it checks if the
+`BUILDHISTORY` entry is set to `true` for the target, and if the
+`reponame:branchname` argument passed into `run-config` is present in
+the `BUILD_HISTORY_DIRECTPUSH` or the `BUILD_HISTORY_FORKPUSH` entry
+in `config.json`. Then `scripts/buildhistory-init` script is executed
+to fetch the build history repo specified in the `BUILD_HISTORY_REPO`
+entry in `config.json`. However, the default build history repo given
+in `config.json` requires permission to clone.
+
+Then for each `step` of the target, the following tasks are run in
+sequence:
+
+1. `bitbake-layers addlayer` is run for all the layers present in the
+ `ADDLAYER` entry of the target in `config.json`. Note that these
+ layers are different from the ones added in the previous step by
+ `scripts/layer-config`.
+
+2. `scripts/setup-config` generates an `auto.conf` under
+ `abworkdir/build/conf` and writes to it the extra configs specified
+ by the `extravars` entry of the target in `config.json`. It also
+ generates a `sdk-extra.conf` file and writes to it the items
+ specified by the `SDKEXTRAS` entry of the target in `config.json`.
+
+3. `bitbake` command is run on the `BBTARGET` specified in the current
+ step with `-k` option.
+
+4. `SANITYTARGETS` of the step is run.
+
+5. `EXTRACMDS` and `EXTRAPLAINCMDS` of the step is run.
+
+6. Remove all layers added in reverse order using `bitbake-layers
+ remove-layer`.
+
+After the build is finished, publish artefacts (copy generated images
+and sdks to the user-defined publish directory. Refer to
+`scripts/publish-artefacts`), collect build results (copy
+`build_directory/tmp/log/oeqa/testresults.json` to the user-defined
+results directory and also compare the current build with previous
+ones. `buildhistory` is being run, refer to `scripts/collect-results`)
+and send error report if and error occurred and the `SENDERRORS` entry
+for the target is set in `config.json` (`send-error-report` is being
+run, refer to `scripts/upload-error-reports`).
+
+Lastly, everything is cleaned up. If there is any error during the
+build or the build is published (i.e. the `--publish-dir` argument is
+specified), the build directory is renamed by appending `-renamed`
+after the original build directory name and the script
+exits. Otherwise, `janitor/clobberdir` script is run to try to move
+the build directory to the trash directory set in `config.json` by the
+`TRASH_DIR` entry. `janitor/clobberdir` first checks if trash
+directory is present and is valid, then it checks if the trash
+directory is on the same file system with the build directory. If it
+is, move the build directory to trash directory, else delete it.
+
+Args:
+
+1. `target`: Same as in `scripts/shared-repo-unpack`
+
+2. `builddir`: The build directory created by the sourcing of
+ `oe-init-build-env`, **This directory is not the autobuilder
+ working `abworkdir`**, but rather the `build` directory inside it,
+ i.e. it is something like `abworkdir/build`.
+
+3. `branchname`: The branch the build is running on. However, **This
+ does not set the branch the build is running**, it is only used for
+ build history. To change the branch of the build, modify the
+ `branch` entry in `repos.json`.
+
+4. `reponame`: The repo the build is running on. Again, **This does
+ not set the repo the build is running**, it is only used for build
+ history.
+
+5. `-s`, `--sstateprefix`: *Defaults to empty*, the directory prefix
+ to publish sstate into.
+
+6. `-b`, `--buildappsrcrev`: *Defaults to empty*, a build appliance
+ SRCREV to use.
+
+7. `-p`, `--publish-dir`: *Optional*, sets the directory
+ `scripts/publish-artefacts` script will be copying to.
+
+8. `-r`, `--results-dir`: *Optional*, sets the directory
+ `scripts/collect-results` script will be copying to.
+
+9. `-u`, `--build-url`: *Optional*, sets the url to link back to this
+ build from the error report server.
+
+10. `--build-type`: *Defaults to `quick`*, can be either `quick` or
+ `full`. For `quick` type, toolchain tests are skipped.
+
+11. `-t`, `--test`: *Defaults to `false`*, if set to `true`, only
+ `scripts/setup-config` is executed and the commands are dry-run.
+
+12. `-q`, `--quietlogging`: *Defaults to `false`*, prevents the
+ flushing of `bitbake` stdout output.
+
+## Steps to build a target
+
+Suppose we are now in the `scripts` directory. We want to build the
+target `poky-tiny` and work in the directory `$HOME/workdir` and want
+to cache all the repos in the directory `$HOME/cache` for future use.
+
+1. Modify the `BASE_HOMEDIR` in `config.json` to a valid directory for the build to happen. Here we use `$HOME`
+
+```json
+{
+ "BASE_HOMEDIR" : "/home/usr"
+ ...
+}
+```
+
+2. Create required directory `git/trash`. Here we create it under `$HOME`:
+
+```bash
+$ mkdir -p $HOME/git/trash
+```
+
+3. Check the `NEEDREPOS` entry for corresponding build targets in
+ `config.json` and create the `repos.json` file according to the
+ [Json File Containing Repository
+ Data](##{Json-File-Containing-Repository-Data}) section above. In
+ this case `poky` and `bitbake` are needed, so the `repos.json` file
+ should look like:
+
+```json
+{
+ "poky" : {
+ "url" : "git://git.yoctoproject.org/poky",
+ "branch" : "",
+ "revision" : "HEAD",
+ "checkout-dirname" : ".",
+ "no-layer-add" : true,
+ "call-init" : true
+ },
+ "bitbake" : {
+ "url" : "git://git.openembedded.org/bitbake",
+ "branch" : "master",
+ "revision" : "HEAD",
+ "no-layer-add" : true
+ }
+}
+```
+
+4. Run `scripts/prepare-shared-repos` and point to the `repos.json`
+ file as well as the cache directory. In this case we want:
+
+```bash
+$ ./prepare-shared-repos $HOME/repos.json $HOME/cache
+```
+
+5. Run `scripts/shared-repo-unpack` and point to the `repos.json` file
+ as well as the autobuilder's working directory, `-c` or the
+ `--cache-dir` argument can be set since we have already fetched the
+ repos in the previous step:
+
+```bash
+$ ./shared-repo-unpack -c $HOME/cache $HOME/repos.json $HOME/workdir poky-tiny
+```
+
+Now the `$HOME/workdir` should be created and `oe-init-build-env` should be
+sourced, which creates a `build` directory inside `$HOME/workdir`.
+
+6. Run `scripts/run-config` with the target to build, the build
+ directory (**Not the autobuilder working directory**), the branch
+ and the repo we are building on ([**The last two arguments does not
+ affect build**](##`scripts/run-config`)).
+
+```bash
+$ ./run-config poky-tiny $HOME/workdir/build master poky
+```
+
+7. If the build finished without error and is not published, the build
+ directory should be rsynced to the `git/trash` directory. If the
+ build failed, the `build` directory will be renamed to
+ `build-renamed` and stay inside the working directory.
+
+## Notes
+
+1. Instead of asking user for the `config.json` file, all the entry
+ point scripts referred to the file through function
+ `util.loadconfig()`, which first finds out the absolute path of the
+ current executing script (i.e. `scripts/prepare-shared-repos`) and
+ then append `../config.json` to it. This is probably not a good
+ idea.
+
+2. All the copy of the repos are done using `rsync`, `git clone
+ --reference or git clone --shared` may be better.
+
+3. Using `rsync` also caused `prepare-shared-repos` and
+ `shared-repo-unpack` script to do `git clone` and overwrite repo
+ content every time they are being called, even when the repo
+ content is exactly the same. We should probably only be doing `git
+ clone` for new repos and do `git checkout` or `git pull` for
+ existing repos.
+
+4. After a failed or published `scripts/run-config`, the build
+ directory is simply appended with `-renamed` to avoid distracting
+ other builds, this is probably not a good idea (Maybe append random
+ string or build ID type of string after each build directory?).
+
+5. The path specified for `git clone` in function `fetchgitrepo()` in
+ `scripts/utils.py` append `/` directly and causes the path to have
+ two `/` if 'abworkdir' is passed in with a `/` at the end
+ (e.g. `/home/workdir/` instead of `/home/workdir`), this does not
+ have any effect on `git clone` but maybe an extra parsing/formating
+ of the path is better.
+
+6. Feeding an incorrect target name into `scripts/shared-repo-unpack`
+ and `scripts/run-config` will not result in an error. The scripts
+ will instead use configurations in the `defaults` target and
+ `scripts/run-config` will simply skip the bitbake command and exit
+ normally with an exit code 0.
+
+7. `checkvnc` script tries to call `-kill` only on the first display,
+ there should be more flexibility on this.
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#47784): https://lists.yoctoproject.org/g/yocto/message/47784
Mute This Topic: https://lists.yoctoproject.org/mt/68855528/1024635
Group Owner: yocto+owner@lists.yoctoproject.org
Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [akuster@mvista.com]
-=-=-=-=-=-=-=-=-=-=-=-