Re: [PATCH v5 1/1] scripts: Add clean-hashserver-database script

["Alexandre Marques" <c137.marques@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 3696 bytes --]

On Fri, Mar 14, 2025 at 03:14 AM, Alexandre Marques wrote:

> 
> Auxiliary script to clean the hashserver database based on the files
> available in the sstate directory.
> 
> It makes used of the new "hashclient gc-mark-stream" command to mark all
> sstate
> relevant hashes as "alive" and removes everything else from the
> database.
> 
> Usage example:
> ```
> ./scripts/clean-hashserver-database \
> --sstate-dir ~/build/sstate-cache \
> --hashclient ./bitbake/bin/bitabke-hashclient \
> --hashserver-address "ws://localhost:8688/ws" \
> --mark "alive" \
> --threshold-age 60 \
> --clean-db
> ```
> 
> Signed-off-by: Alexander Marques <c137.marques@gmail.com>
> ---
> scripts/clean-hashserver-database | 76 +++++++++++++++++++++++++++++++
> 1 file changed, 76 insertions(+)
> create mode 100755 scripts/clean-hashserver-database
> 
> diff --git a/scripts/clean-hashserver-database
> b/scripts/clean-hashserver-database
> new file mode 100755
> index 0000000000..46b99e0af4
> --- /dev/null
> +++ b/scripts/clean-hashserver-database
> @@ -0,0 +1,76 @@
> +#!/bin/bash
> +set -euo pipefail
> +
> +SSTATE_DIR=""
> +BB_HASHCLIENT=""
> +BB_HASHSERVER=""
> +
> +ALIVE_DB_MARK="alive"
> +CLEAN_DB="false"
> +THRESHOLD_AGE="3600"
> +
> +function help() {
> + cat <<HELP_TEXT
> +Usage: $0 --sstate-dir path --hashclient path --hashserver-address
> address [--mark value] [--clean-db]

Missing reference to `--threshold-age` in the usage example.
> 
> +
> +Auxiliary script remove unused or no longer relevant entries from the
> hashequivalence database, based
> +on the files available on the sstate directory.
> +
> + -h | --help) Show this help message and exit
> + -a | --hashserver-adress) bitbake-hashserver address
> + -c | --hashclient) Path to bitbake-hashclient
> + -m | --mark) Marker string to mark database entries
> + -s | --sstate-dir) Path to the sstate dir
> + -t | --threshold-age) Remove unused entries older than SECONDS old
> (default: 3600)
> + --clean-db) Remove all unmarked and unused entries from the database
> +HELP_TEXT
> +}
> +
> +function argument_parser() {
> + while [ $# -gt 0 ]; do
> + case "$1" in
> + -h | --help) help; exit 0 ;;
> + -a | --hashserver-address) BB_HASHSERVER="$2"; shift ;;
> + -c | --hashclient) BB_HASHCLIENT="$2"; shift ;;
> + -m | --mark) ALIVE_DB_MARK="$2"; shift ;;
> + -s | --sstate-dir) SSTATE_DIR="$2"; shift ;;
> + -t | --threshold-age) THRESHOLD_AGE="$2"; shift ;;
> + --clean-db) CLEAN_DB="true";;
> + *)
> + echo "Argument '$1' is not supported" >&2
> + help >&2
> + exit 1
> + ;;
> + esac
> + shift
> + done
> +
> + function validate_mandatory_argument() {
> + local var_value="$1"
> + local error_message="$2"
> +
> + if [ -z "$var_value" ]; then
> + echo "$error_message"
> + help >&2
> + exit 1
> + fi
> + }
> +
> + validate_mandatory_argument "$SSTATE_DIR" "Please provide the path to
> the sstate dir."
> + validate_mandatory_argument "$BB_HASHCLIENT" "Please provide the path to
> bitbake-hashclient."
> + validate_mandatory_argument "$BB_HASHSERVER" "Please provide the address
> of bitbake-hashserver."
> +}
> +
> +# -- main code --
> +argument_parser $@
> +
> +# Mark all db sstate hashes
> +find "$SSTATE_DIR" -name "*.tar.zst" | \
> +sed 's/.*:\([^_]*\)_.*/unihash \1/' | \
> +$BB_HASHCLIENT --address "$BB_HASHSERVER" gc-mark-stream "$ALIVE_DB_MARK"
> 
> +
> +# Remove unmarked and unused entries
> +if [ "$CLEAN_DB" = "true" ]; then
> + $BB_HASHCLIENT --address "$BB_HASHSERVER" gc-sweep "$ALIVE_DB_MARK"
> + $BB_HASHCLIENT --address "$BB_HASHSERVER" clean-unused "$THRESHOLD_AGE"
> +fi
> --
> 2.34.1

[-- Attachment #2: Type: text/html, Size: 3974 bytes --]