[PATCH] tools: mkenvimage: Add pure shell implementation of mkenvimage

[PATCH] tools: mkenvimage: Add pure shell implementation of mkenvimage

[Marek Vasut]

Add implementation of mkenvimage written purely in bourne shell.

This is not a replacement for mkenvimage tool, but rather a simple
implementation which can be used in environments where mkenvimage
itself cannot be deployed due to various constraints, like hardware
manufacturing plants, but where bourne shell and basic tool are
already available.

The external dependencies which are not shell built-ins are gzip
and grep.

All mkenvimage parameters are implemented and compatible with the
C implementation of mkenvimage.

Signed-off-by: Marek Vasut <marex@denx.de>
---
Cc: Joe Hershberger <joe.hershberger@ni.com>
Cc: Tom Rini <trini@konsulko.com>
---
 tools/mkenvimage.sh | 126 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 126 insertions(+)
 create mode 100755 tools/mkenvimage.sh

diff --git a/tools/mkenvimage.sh b/tools/mkenvimage.sh
new file mode 100755
index 00000000000..f83fac92e5d
--- /dev/null
+++ b/tools/mkenvimage.sh
@@ -0,0 +1,126 @@
+#!/bin/sh
+
+print_help() {
+	echo "mkenvimage [-h] [-V] [-r] [-b] [-p <byte>] -s <environment partition size> -o <output> <input file>"
+	echo ""
+	echo "This tool takes a key=value input file (same as would a \`printenv' show) and generates the corresponding environment image, ready to be flashed."
+	echo ""
+	echo "        The input file is in format:"
+	echo "                key1=value1"
+	echo "                key2=value2"
+	echo "                ..."
+	echo "        Empty lines are skipped, and lines with a # in the first"
+	echo "        column are treated as comments (also skipped)."
+	echo "        -r : the environment has multiple copies in flash"
+	echo "        -b : the target is big endian (default is little endian)"
+	echo "        -p <byte> : fill the image with <byte> bytes instead of 0xff bytes"
+	echo "        -V : print version information and exit"
+	echo ""
+	echo 'If the input file is "-", data is read from standard input'
+}
+
+INFILE=""
+OUTFILE=""
+
+REDUNDANT=0
+BIGENDIAN=0
+BYTEFILL=$(printf '\377')
+SIZE=0
+
+while getopts "hrbp:s:o:V" opt ; do
+	case "$opt" in
+	r)
+		REDUNDANT=1
+		;;
+	b)
+		# Swap CRC endianness
+		BIGENDIAN=1
+		;;
+	p)
+		BYTEFILL="$(printf "%b" "\0$(printf "%o" $((OPTARG)))")"
+		;;
+	s)
+		SIZE="$((OPTARG))"
+		;;
+	o)
+		OUTFILE="${OPTARG}"
+		;;
+	V)
+		echo "mkenvimage version SHELL"
+		exit 0
+		;;
+	h|*)
+		print_help
+		exit 0
+		;;
+	esac
+done
+
+shift $((OPTIND - 1))
+if test -n "${1}" ; then
+	INFILE=${1}
+fi
+
+if test -z "${INFILE}" ; then
+	echo "Input file not specified"
+	print_help
+	exit 1
+fi
+
+if test -z "${OUTFILE}" ; then
+	echo "Output file not specified"
+	print_help
+	exit 1
+fi
+
+if test "${SIZE}" -eq 0 ; then
+	echo "Size not specified"
+	print_help
+	exit 1
+fi
+
+crctmpfile=$(mktemp)
+outtmpfile=$(mktemp)
+
+(
+# Read input environment file, make sure there is a trailing newline,
+# sort the result and remove empty and commented out lines.
+( cat "${INFILE}" ; echo ) | sort -u "${INFILE}" | grep -v '^#' | grep -v '^[ \t]*$' | tr '\n' '\0'
+# Insert one more trailing zero at the end of environment.
+printf '\0'
+# Fill the rest of the environment with BYTEFILL bytes.
+tr '\0' "${BYTEFILL}" < /dev/zero
+# Clip the environment block to correct size. Header is 4 bytes CRC32
+# for regular environment and 5 bytes CRC32 and flags for redundant
+# environment.
+) | dd count="$((SIZE - 4 - REDUNDANT))" bs=1 of="${outtmpfile}" 2>/dev/null
+
+gzip -ck1 "${outtmpfile}" | tail -c8 | head -c4 > "${crctmpfile}"
+
+# Reverse crc32 for big-endian systems
+if test "$BIGENDIAN" -eq 1 ; then
+	# Note that 'rev' is from util-linux and may not be available,
+	# 'xxd' is from vim and may not be available, and 'tac -rs .'
+	# does not handle unterminated strings and newlines well, so
+	# use plain 'dd' to reverse the 4 bytes reliably and using
+	# minimum dependencies:
+	crcbetmpfile=$(mktemp)
+	(
+		dd if="${crctmpfile}" bs=1 count=1 skip=3 ;
+		dd if="${crctmpfile}" bs=1 count=1 skip=2 ;
+		dd if="${crctmpfile}" bs=1 count=1 skip=1 ;
+		dd if="${crctmpfile}" bs=1 count=1 skip=0
+	) >> "${crcbetmpfile}" 2>/dev/null
+	mv "${crcbetmpfile}" "${crctmpfile}"
+fi
+
+# Add 5th byte set to 0x01 to indicate redundant environment image.
+if test "$REDUNDANT" -eq 1 ; then
+	printf '\001' >> "${crctmpfile}"
+fi
+
+# Concatenate the CRC32 and environment block into the final output file.
+cat "${crctmpfile}" "${outtmpfile}" > "${OUTFILE}"
+
+# Clean up the temporary files
+rm -f "${crctmpfile}" "${outtmpfile}"
-- 
2.47.2

Re: [PATCH] tools: mkenvimage: Add pure shell implementation of mkenvimage

[Simon Glass]

Hi Marek,

On Wed, 5 Feb 2025 at 07:37, Marek Vasut <marex@denx.de> wrote:
>
> Add implementation of mkenvimage written purely in bourne shell.
>
> This is not a replacement for mkenvimage tool, but rather a simple
> implementation which can be used in environments where mkenvimage
> itself cannot be deployed due to various constraints, like hardware
> manufacturing plants, but where bourne shell and basic tool are
> already available.
>
> The external dependencies which are not shell built-ins are gzip
> and grep.
>
> All mkenvimage parameters are implemented and compatible with the
> C implementation of mkenvimage.
>
> Signed-off-by: Marek Vasut <marex@denx.de>
> ---
> Cc: Joe Hershberger <joe.hershberger@ni.com>
> Cc: Tom Rini <trini@konsulko.com>
> ---
>  tools/mkenvimage.sh | 126 ++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 126 insertions(+)
>  create mode 100755 tools/mkenvimage.sh

Would it be worth adding a simple test for this?

Regards,
SImon

> +               REDUNDANT=1
> +               ;;
> +       b)
> +               # Swap CRC endianness
> +               BIGENDIAN=1
> +               ;;
> +       p)
> +               BYTEFILL="$(printf "%b" "\0$(printf "%o" $((OPTARG)))")"
> +               ;;
> +       s)
> +               SIZE="$((OPTARG))"
> +               ;;
> +       o)
> +               OUTFILE="${OPTARG}"
> +               ;;
> +       V)
> +               echo "mkenvimage version SHELL"
> +               exit 0
> +               ;;
> +       h|*)
> +               print_help
> +               exit 0
> +               ;;
> +       esac
> +done
> +
> +shift $((OPTIND - 1))
> +if test -n "${1}" ; then
> +       INFILE=${1}
> +fi
> +
> +if test -z "${INFILE}" ; then
> +       echo "Input file not specified"
> +       print_help
> +       exit 1
> +fi
> +
> +if test -z "${OUTFILE}" ; then
> +       echo "Output file not specified"
> +       print_help
> +       exit 1
> +fi
> +
> +if test "${SIZE}" -eq 0 ; then
> +       echo "Size not specified"
> +       print_help
> +       exit 1
> +fi
> +
> +crctmpfile=$(mktemp)
> +outtmpfile=$(mktemp)
> +
> +(
> +# Read input environment file, make sure there is a trailing newline,
> +# sort the result and remove empty and commented out lines.
> +( cat "${INFILE}" ; echo ) | sort -u "${INFILE}" | grep -v '^#' | grep -v '^[ \t]*$' | tr '\n' '\0'
> +# Insert one more trailing zero at the end of environment.
> +printf '\0'
> +# Fill the rest of the environment with BYTEFILL bytes.
> +tr '\0' "${BYTEFILL}" < /dev/zero
> +# Clip the environment block to correct size. Header is 4 bytes CRC32
> +# for regular environment and 5 bytes CRC32 and flags for redundant
> +# environment.
> +) | dd count="$((SIZE - 4 - REDUNDANT))" bs=1 of="${outtmpfile}" 2>/dev/null
> +
> +gzip -ck1 "${outtmpfile}" | tail -c8 | head -c4 > "${crctmpfile}"
> +
> +# Reverse crc32 for big-endian systems
> +if test "$BIGENDIAN" -eq 1 ; then
> +       # Note that 'rev' is from util-linux and may not be available,
> +       # 'xxd' is from vim and may not be available, and 'tac -rs .'
> +       # does not handle unterminated strings and newlines well, so
> +       # use plain 'dd' to reverse the 4 bytes reliably and using
> +       # minimum dependencies:
> +       crcbetmpfile=$(mktemp)
> +       (
> +               dd if="${crctmpfile}" bs=1 count=1 skip=3 ;
> +               dd if="${crctmpfile}" bs=1 count=1 skip=2 ;
> +               dd if="${crctmpfile}" bs=1 count=1 skip=1 ;
> +               dd if="${crctmpfile}" bs=1 count=1 skip=0
> +       ) >> "${crcbetmpfile}" 2>/dev/null
> +       mv "${crcbetmpfile}" "${crctmpfile}"
> +fi
> +
> +# Add 5th byte set to 0x01 to indicate redundant environment image.
> +if test "$REDUNDANT" -eq 1 ; then
> +       printf '\001' >> "${crctmpfile}"
> +fi
> +
> +# Concatenate the CRC32 and environment block into the final output file.
> +cat "${crctmpfile}" "${outtmpfile}" > "${OUTFILE}"
> +
> +# Clean up the temporary files
> +rm -f "${crctmpfile}" "${outtmpfile}"
> --
> 2.47.2
>

Re: [PATCH] tools: mkenvimage: Add pure shell implementation of mkenvimage

[Marek Vasut]

On 2/6/25 1:38 PM, Simon Glass wrote:
> Hi Marek,
> 
> On Wed, 5 Feb 2025 at 07:37, Marek Vasut <marex@denx.de> wrote:
>>
>> Add implementation of mkenvimage written purely in bourne shell.
>>
>> This is not a replacement for mkenvimage tool, but rather a simple
>> implementation which can be used in environments where mkenvimage
>> itself cannot be deployed due to various constraints, like hardware
>> manufacturing plants, but where bourne shell and basic tool are
>> already available.
>>
>> The external dependencies which are not shell built-ins are gzip
>> and grep.
>>
>> All mkenvimage parameters are implemented and compatible with the
>> C implementation of mkenvimage.
>>
>> Signed-off-by: Marek Vasut <marex@denx.de>
>> ---
>> Cc: Joe Hershberger <joe.hershberger@ni.com>
>> Cc: Tom Rini <trini@konsulko.com>
>> ---
>>   tools/mkenvimage.sh | 126 ++++++++++++++++++++++++++++++++++++++++++++
>>   1 file changed, 126 insertions(+)
>>   create mode 100755 tools/mkenvimage.sh
> 
> Would it be worth adding a simple test for this?
Sure, is there an existing test for similar case I can look at ?

Re: [PATCH] tools: mkenvimage: Add pure shell implementation of mkenvimage

[Simon Glass]

Hi Marek,

On Thu, 6 Feb 2025 at 13:52, Marek Vasut <marex@denx.de> wrote:
>
> On 2/6/25 1:38 PM, Simon Glass wrote:
> > Hi Marek,
> >
> > On Wed, 5 Feb 2025 at 07:37, Marek Vasut <marex@denx.de> wrote:
> >>
> >> Add implementation of mkenvimage written purely in bourne shell.
> >>
> >> This is not a replacement for mkenvimage tool, but rather a simple
> >> implementation which can be used in environments where mkenvimage
> >> itself cannot be deployed due to various constraints, like hardware
> >> manufacturing plants, but where bourne shell and basic tool are
> >> already available.
> >>
> >> The external dependencies which are not shell built-ins are gzip
> >> and grep.
> >>
> >> All mkenvimage parameters are implemented and compatible with the
> >> C implementation of mkenvimage.
> >>
> >> Signed-off-by: Marek Vasut <marex@denx.de>
> >> ---
> >> Cc: Joe Hershberger <joe.hershberger@ni.com>
> >> Cc: Tom Rini <trini@konsulko.com>
> >> ---
> >>   tools/mkenvimage.sh | 126 ++++++++++++++++++++++++++++++++++++++++++++
> >>   1 file changed, 126 insertions(+)
> >>   create mode 100755 tools/mkenvimage.sh
> >
> > Would it be worth adding a simple test for this?
> Sure, is there an existing test for similar case I can look at ?

Well test_env_text is something I did for the env2string.awk script,
so perhaps something like that?

Regards,
Simon

Re: [PATCH] tools: mkenvimage: Add pure shell implementation of mkenvimage

[Alexander Dahl]

Hi Marek,

had a short look on your script yesterday.  Looks clean and well
designed, nice job.

Am Thu, Feb 06, 2025 at 09:38:58PM +0100 schrieb Marek Vasut:
> On 2/6/25 1:38 PM, Simon Glass wrote:
> > Hi Marek,
> > 
> > On Wed, 5 Feb 2025 at 07:37, Marek Vasut <marex@denx.de> wrote:
> > > 
> > > Add implementation of mkenvimage written purely in bourne shell.
> > > 
> > > This is not a replacement for mkenvimage tool, but rather a simple
> > > implementation which can be used in environments where mkenvimage
> > > itself cannot be deployed due to various constraints, like hardware
> > > manufacturing plants, but where bourne shell and basic tool are
> > > already available.
> > > 
> > > The external dependencies which are not shell built-ins are gzip
> > > and grep.
> > > 
> > > All mkenvimage parameters are implemented and compatible with the
> > > C implementation of mkenvimage.
> > > 
> > > Signed-off-by: Marek Vasut <marex@denx.de>
> > > ---
> > > Cc: Joe Hershberger <joe.hershberger@ni.com>
> > > Cc: Tom Rini <trini@konsulko.com>
> > > ---
> > >   tools/mkenvimage.sh | 126 ++++++++++++++++++++++++++++++++++++++++++++
> > >   1 file changed, 126 insertions(+)
> > >   create mode 100755 tools/mkenvimage.sh
> > 
> > Would it be worth adding a simple test for this?
> Sure, is there an existing test for similar case I can look at ?

I thought about testing yesterday, but had no good idea on top, but
now it's discussed anyways …  How about throwing the same input
at the c and the sh version and assert the output is the same?  For
more sophisticated tests it should be tested with both variants.

Greets
Alex

Re: [PATCH] tools: mkenvimage: Add pure shell implementation of mkenvimage

[Rasmus Villemoes]

On Wed, Feb 05 2025, Marek Vasut <marex@denx.de> wrote:

> Add implementation of mkenvimage written purely in bourne shell.
>
> This is not a replacement for mkenvimage tool, but rather a simple
> implementation which can be used in environments where mkenvimage
> itself cannot be deployed due to various constraints, like hardware
> manufacturing plants, but where bourne shell and basic tool are
> already available.
>
> The external dependencies which are not shell built-ins are gzip
> and grep.
>

Eh, and cat, sort, dd, tr, mktemp, head, tail, mv and rm ?

> All mkenvimage parameters are implemented and compatible with the
> C implementation of mkenvimage.
>
> Signed-off-by: Marek Vasut <marex@denx.de>
> ---
> Cc: Joe Hershberger <joe.hershberger@ni.com>
> Cc: Tom Rini <trini@konsulko.com>
> ---
>  tools/mkenvimage.sh | 126 ++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 126 insertions(+)
>  create mode 100755 tools/mkenvimage.sh
>
> diff --git a/tools/mkenvimage.sh b/tools/mkenvimage.sh
> new file mode 100755
> index 00000000000..f83fac92e5d
> --- /dev/null
> +++ b/tools/mkenvimage.sh
> @@ -0,0 +1,126 @@
> +#!/bin/sh
> +
> +print_help() {
> +	echo "mkenvimage [-h] [-V] [-r] [-b] [-p <byte>] -s <environment partition size> -o <output> <input file>"

Nit: you included a .sh extension in the filename, probably this output
should reflect that.

> +
> +(
> +# Read input environment file, make sure there is a trailing newline,
> +# sort the result and remove empty and commented out lines.
> +( cat "${INFILE}" ; echo ) | sort -u "${INFILE}" | grep -v '^#' | grep -v '^[ \t]*$' | tr '\n' '\0'

Doesn't sort ignore stdin when given file(s) via argv? Also, the newline
added by that echo would probably by removed by the grep, and you
manually insert a \0 right after, so probably the first cat;echo should
just be removed.

I'm also wondering why you even do that sort; AFAIK, mkenvimage doesn't
do that. And it would make a difference if the input fx had two
definitions for the same variable

foo=Y
foo=X

because when loaded on target, the last occurring value takes
precedence.

Are you handling escaping of newlines, in order to allow values to
contain newline characters?

Rasmus

Re: [PATCH] tools: mkenvimage: Add pure shell implementation of mkenvimage

[Marek Vasut]

On 2/7/25 11:31 AM, Rasmus Villemoes wrote:
> On Wed, Feb 05 2025, Marek Vasut <marex@denx.de> wrote:
> 
>> Add implementation of mkenvimage written purely in bourne shell.
>>
>> This is not a replacement for mkenvimage tool, but rather a simple
>> implementation which can be used in environments where mkenvimage
>> itself cannot be deployed due to various constraints, like hardware
>> manufacturing plants, but where bourne shell and basic tool are
>> already available.
>>
>> The external dependencies which are not shell built-ins are gzip
>> and grep.
>>
> 
> Eh, and cat, sort, dd, tr, mktemp, head, tail, mv and rm ?

A lot of which are shell builtins .

>> All mkenvimage parameters are implemented and compatible with the
>> C implementation of mkenvimage.
>>
>> Signed-off-by: Marek Vasut <marex@denx.de>
>> ---
>> Cc: Joe Hershberger <joe.hershberger@ni.com>
>> Cc: Tom Rini <trini@konsulko.com>
>> ---
>>   tools/mkenvimage.sh | 126 ++++++++++++++++++++++++++++++++++++++++++++
>>   1 file changed, 126 insertions(+)
>>   create mode 100755 tools/mkenvimage.sh
>>
>> diff --git a/tools/mkenvimage.sh b/tools/mkenvimage.sh
>> new file mode 100755
>> index 00000000000..f83fac92e5d
>> --- /dev/null
>> +++ b/tools/mkenvimage.sh
>> @@ -0,0 +1,126 @@
>> +#!/bin/sh
>> +
>> +print_help() {
>> +	echo "mkenvimage [-h] [-V] [-r] [-b] [-p <byte>] -s <environment partition size> -o <output> <input file>"
> 
> Nit: you included a .sh extension in the filename, probably this output
> should reflect that.

$0 is even better.

>> +
>> +(
>> +# Read input environment file, make sure there is a trailing newline,
>> +# sort the result and remove empty and commented out lines.
>> +( cat "${INFILE}" ; echo ) | sort -u "${INFILE}" | grep -v '^#' | grep -v '^[ \t]*$' | tr '\n' '\0'
> 
> Doesn't sort ignore stdin when given file(s) via argv? Also, the newline
> added by that echo would probably by removed by the grep, and you
> manually insert a \0 right after, so probably the first cat;echo should
> just be removed.

Good point

> I'm also wondering why you even do that sort; AFAIK, mkenvimage doesn't
> do that. And it would make a difference if the input fx had two
> definitions for the same variable
> 
> foo=Y
> foo=X
> 
> because when loaded on target, the last occurring value takes
> precedence.

Indeed

> Are you handling escaping of newlines, in order to allow values to
> contain newline characters?
Uh, no, I don't think so. I don't think I even used something like that 
with plain C mkenvimage either. Do you have an example input env for me?

Re: [PATCH] tools: mkenvimage: Add pure shell implementation of mkenvimage

[Marek Vasut]

On 2/7/25 1:49 AM, Simon Glass wrote:
> Hi Marek,
> 
> On Thu, 6 Feb 2025 at 13:52, Marek Vasut <marex@denx.de> wrote:
>>
>> On 2/6/25 1:38 PM, Simon Glass wrote:
>>> Hi Marek,
>>>
>>> On Wed, 5 Feb 2025 at 07:37, Marek Vasut <marex@denx.de> wrote:
>>>>
>>>> Add implementation of mkenvimage written purely in bourne shell.
>>>>
>>>> This is not a replacement for mkenvimage tool, but rather a simple
>>>> implementation which can be used in environments where mkenvimage
>>>> itself cannot be deployed due to various constraints, like hardware
>>>> manufacturing plants, but where bourne shell and basic tool are
>>>> already available.
>>>>
>>>> The external dependencies which are not shell built-ins are gzip
>>>> and grep.
>>>>
>>>> All mkenvimage parameters are implemented and compatible with the
>>>> C implementation of mkenvimage.
>>>>
>>>> Signed-off-by: Marek Vasut <marex@denx.de>
>>>> ---
>>>> Cc: Joe Hershberger <joe.hershberger@ni.com>
>>>> Cc: Tom Rini <trini@konsulko.com>
>>>> ---
>>>>    tools/mkenvimage.sh | 126 ++++++++++++++++++++++++++++++++++++++++++++
>>>>    1 file changed, 126 insertions(+)
>>>>    create mode 100755 tools/mkenvimage.sh
>>>
>>> Would it be worth adding a simple test for this?
>> Sure, is there an existing test for similar case I can look at ?
> 
> Well test_env_text is something I did for the env2string.awk script,
> so perhaps something like that?

I found some python file in the codebase which contains this function, 
but I couldn't find any documentation how to use it. I am also not a big 
fan of python. Is there any documentation how to use this test ?

Re: [PATCH] tools: mkenvimage: Add pure shell implementation of mkenvimage

[Marek Vasut]

On 2/7/25 8:40 AM, Alexander Dahl wrote:
> Hi Marek,
> 
> had a short look on your script yesterday.  Looks clean and well
> designed, nice job.
> 
> Am Thu, Feb 06, 2025 at 09:38:58PM +0100 schrieb Marek Vasut:
>> On 2/6/25 1:38 PM, Simon Glass wrote:
>>> Hi Marek,
>>>
>>> On Wed, 5 Feb 2025 at 07:37, Marek Vasut <marex@denx.de> wrote:
>>>>
>>>> Add implementation of mkenvimage written purely in bourne shell.
>>>>
>>>> This is not a replacement for mkenvimage tool, but rather a simple
>>>> implementation which can be used in environments where mkenvimage
>>>> itself cannot be deployed due to various constraints, like hardware
>>>> manufacturing plants, but where bourne shell and basic tool are
>>>> already available.
>>>>
>>>> The external dependencies which are not shell built-ins are gzip
>>>> and grep.
>>>>
>>>> All mkenvimage parameters are implemented and compatible with the
>>>> C implementation of mkenvimage.
>>>>
>>>> Signed-off-by: Marek Vasut <marex@denx.de>
>>>> ---
>>>> Cc: Joe Hershberger <joe.hershberger@ni.com>
>>>> Cc: Tom Rini <trini@konsulko.com>
>>>> ---
>>>>    tools/mkenvimage.sh | 126 ++++++++++++++++++++++++++++++++++++++++++++
>>>>    1 file changed, 126 insertions(+)
>>>>    create mode 100755 tools/mkenvimage.sh
>>>
>>> Would it be worth adding a simple test for this?
>> Sure, is there an existing test for similar case I can look at ?
> 
> I thought about testing yesterday, but had no good idea on top, but
> now it's discussed anyways …  How about throwing the same input
> at the c and the sh version and assert the output is the same?  For
> more sophisticated tests it should be tested with both variants.

I do have local shell script which does exactly that -- feed inputs into 
both mkenvimage C and SH variants and compare the checksum of results.