[RFC PATCH 0/1] make cmdline library easier to use

[RFC PATCH 0/1] make cmdline library easier to use

[Bruce Richardson]

And now for something completely different... :-)

While it's nice that DPDK includes a commandline library for easily
adding interactivity to applications, the library itself is a little
clunky to use, due to the massive amounts of boilerplate definitions
required. Having used it recently on an app I was playing with for
testing, I decided the situation can potentially be improved with it
by use of a script to produce the boilerplate when given a simple
list of commands the user wants to add to the app.

This is the result of that effort. Sending it here as a rough-draft to
get feedback on whether this is worth including in DPDK itself.
Hopefully others may find it of use.

Bruce Richardson (1):
  cmdline/dpdk-cmdline-gen: generate boilerplate for simple cmds

 lib/cmdline/dpdk-cmdline-gen.py | 143 ++++++++++++++++++++++++++++++++
 1 file changed, 143 insertions(+)
 create mode 100755 lib/cmdline/dpdk-cmdline-gen.py

--
2.39.2

[RFC PATCH 1/1] cmdline/dpdk-cmdline-gen: generate boilerplate for simple cmds

[Bruce Richardson]

Provide a script for application developers to quickly generate the
boilerplate code necessary for using the cmdline library.

This initial version works only with commands using simple strings and
numbers, but this is sufficient for many use-cases. Future extensions
could, no doubt, expand support to expose more capabilities of the
library.

Example of use:
The script takes an input file with a list of commands the user wants in
the app, where the parameter variables are tagged with the type.
For example:

	$ cat commands.list
	list
	add <UINT16>x <UINT16>y
	echo <STRING>message
	add socket <STRING>path
	quit

When run through the script as "./dpdk-cmdline-gen.py commands.list",
the output will be the contents of a header file with all the
boilerplate necessary for a commandline instance with those commands.

If the flag --stubs is passed, an output header filename must also be
passed, in which case both a header file with the definitions and a C
file with function stubs in it is written to disk. The separation is so
that the header file can be rewritten at any future point to add more
commands, while the C file can be kept as-is and extended by the user
with any additional functions needed.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 lib/cmdline/dpdk-cmdline-gen.py | 143 ++++++++++++++++++++++++++++++++
 1 file changed, 143 insertions(+)
 create mode 100755 lib/cmdline/dpdk-cmdline-gen.py

diff --git a/lib/cmdline/dpdk-cmdline-gen.py b/lib/cmdline/dpdk-cmdline-gen.py
new file mode 100755
index 0000000000..1b81c4f9dc
--- /dev/null
+++ b/lib/cmdline/dpdk-cmdline-gen.py
@@ -0,0 +1,143 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2023 Intel Corporation
+#
+
+import sys
+import os
+import argparse
+
+PARSE_FN_PARAMS = "void *parsed_result, struct cmdline *cl, void *data"
+PARSE_FN_BODY = """
+    /* TODO: command action */
+    RTE_SET_USED(parsed_result);
+    RTE_SET_USED(cl);
+    RTE_SET_USED(data);
+"""
+
+def process_command(tokens, cfile):
+    name = []
+
+    if tokens[0].startswith('<'):
+        print("Error: each command must start with at least one literal string",
+                file=sys.stderr)
+        sys.exit(1)
+    for t in tokens:
+        if t.startswith('<'):
+            break;
+        name.append(t)
+    name = "_".join(name)
+
+    result_struct = []
+    initializers = []
+    token_list = []
+    for t in tokens:
+        if t.startswith('<'):
+            t_type, t_name = t[1:].split('>')
+            t_val = "NULL"
+        else:
+            t_type = "STRING"
+            t_name = t
+            t_val = f'"{t}"'
+
+        if t_type == "STRING":
+            result_struct.append(f"\tcmdline_fixed_string_t {t_name};")
+            initializers.append(f"static cmdline_parse_token_string_t cmd_{name}_{t_name}_tok =\n"
+                    + f"\tTOKEN_STRING_INITIALIZER(struct cmd_{name}_result, {t_name}, {t_val});")
+        elif t_type in ['UINT8', 'UINT16', 'UINT32', 'UINT64',
+                    'INT8', 'INT16', 'INT32', 'INT64']:
+            result_struct.append(f"\t{t_type.lower()}_t {t_name};")
+            initializers.append(f"static cmdline_parse_token_num_t cmd_{name}_{t_name}_tok =\n"
+                    + f"\tTOKEN_NUM_INITIALIZER(struct cmd_{name}_result, {t_name}, RTE_{t_type});")
+        else:
+            print(f"Error: unknown token-type {t}", file=sys.stderr)
+            sys.exit(1)
+        token_list.append(f"cmd_{name}_{t_name}_tok")
+
+    print(f"/* Auto-generated handling for command '{' '.join(tokens)}' */")
+    # output function prototype
+    func_sig = f"void\ncmd_{name}_parsed({PARSE_FN_PARAMS})"
+    print(f"extern {func_sig};\n")
+    # output function template if C file being written
+    if (cfile):
+        print(f"{func_sig}\n{{{PARSE_FN_BODY}}}\n", file=cfile)
+    # output result data structure
+    print(f"struct cmd_{name}_result {{\n"
+            + '\n'.join(result_struct)
+            + "\n};\n")
+    # output the initializer tokens
+    print("\n".join(initializers) + '\n')
+    # output the instance structure
+    print(f"static cmdline_parse_inst_t cmd_{name} = {{\n"
+            + f'\t.f = cmd_{name}_parsed,\n'
+            + f'\t.data = NULL,\n'
+            + f'\t.help_str = "",\n'
+            + f'\t.tokens = {{')
+    for t in token_list:
+        print(f"\t\t(void *)&{t},")
+    print("\t\tNULL\n"
+            + "\t}\n"
+            + "};\n")
+
+    # return the instance structure name
+    return f"cmd_{name}"
+
+def process_commands(infile, hfile, cfile):
+    instances = []
+
+    # redirect stdout to output the header, to save passing file= each print
+    old_sys_stdout = sys.stdout
+    sys.stdout = hfile
+
+    print(f"/* File autogenerated by {sys.argv[0]} */")
+    print('#ifndef GENERATED_COMMANDS_H')
+    print('#define GENERATED_COMMANDS_H')
+    print("#include <cmdline.h>")
+    print("#include <cmdline_parse_string.h>")
+    print("#include <cmdline_parse_num.h>")
+    print("")
+
+    for line in infile.readlines():
+        if line.lstrip().startswith('#'):
+            continue
+        instances.append(process_command(line.strip().split(), cfile))
+
+    print('static cmdline_parse_ctx_t ctx[] = {')
+    for inst in instances:
+        print(f'\t&{inst},')
+    print('\tNULL')
+    print('};\n')
+    print('#endif /* GENERATED_COMMANDS_H */')
+
+    sys.stdout = old_sys_stdout
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--stubs", action="store_true",
+            help="Produce C file with empty function stubs for each command")
+    ap.add_argument("--output-file", "-o", default="-",
+            help="Output header filename [default to stdout]")
+    ap.add_argument("infile", type=argparse.FileType('r'),
+            help="File with list of commands")
+    args = ap.parse_args()
+
+    if not args.stubs:
+        if args.output_file == '-':
+            process_commands(args.infile, sys.stdout, None)
+        else:
+            with open(args.output_file, "w") as hfile:
+                process_commands(args.infile, hfile, None)
+    else:
+        if not args.output_file.endswith('.h'):
+            print("Error: output filename must end with '.h' extension when creating stubs",
+                    file=sys.stderr)
+            sys.exit(1)
+
+        cfilename = args.output_file[:-2] + '.c'
+        with open(args.output_file, "w") as hfile:
+            with open(cfilename, "w") as cfile:
+                print(f"#include \"{args.output_file}\"\n", file=cfile)
+                process_commands(args.infile, hfile, cfile)
+
+if __name__ == "__main__":
+    main()
-- 
2.39.2

Re: [RFC PATCH 0/1] make cmdline library easier to use

[Stephen Hemminger]

On Wed,  2 Aug 2023 18:00:51 +0100
Bruce Richardson <bruce.richardson@intel.com> wrote:

> And now for something completely different... :-)
> 
> While it's nice that DPDK includes a commandline library for easily
> adding interactivity to applications, the library itself is a little
> clunky to use, due to the massive amounts of boilerplate definitions
> required. Having used it recently on an app I was playing with for
> testing, I decided the situation can potentially be improved with it
> by use of a script to produce the boilerplate when given a simple
> list of commands the user wants to add to the app.
> 
> This is the result of that effort. Sending it here as a rough-draft to
> get feedback on whether this is worth including in DPDK itself.
> Hopefully others may find it of use.
> 
> Bruce Richardson (1):
>   cmdline/dpdk-cmdline-gen: generate boilerplate for simple cmds
> 
>  lib/cmdline/dpdk-cmdline-gen.py | 143 ++++++++++++++++++++++++++++++++
>  1 file changed, 143 insertions(+)
>  create mode 100755 lib/cmdline/dpdk-cmdline-gen.py

Ok, but this surely is a solved problem is some other open source library.
Or does every project reinvent this parsing?

Seems to me that something like bison/flex (ie yacc/lex) would be better
in terms of expressing the syntax separate from the implementation.

Re: [RFC PATCH 0/1] make cmdline library easier to use

[Bruce Richardson]

On Wed, Aug 02, 2023 at 11:05:07AM -0700, Stephen Hemminger wrote:
> On Wed,  2 Aug 2023 18:00:51 +0100
> Bruce Richardson <bruce.richardson@intel.com> wrote:
> 
> > And now for something completely different... :-)
> > 
> > While it's nice that DPDK includes a commandline library for easily
> > adding interactivity to applications, the library itself is a little
> > clunky to use, due to the massive amounts of boilerplate definitions
> > required. Having used it recently on an app I was playing with for
> > testing, I decided the situation can potentially be improved with it
> > by use of a script to produce the boilerplate when given a simple
> > list of commands the user wants to add to the app.
> > 
> > This is the result of that effort. Sending it here as a rough-draft to
> > get feedback on whether this is worth including in DPDK itself.
> > Hopefully others may find it of use.
> > 
> > Bruce Richardson (1):
> >   cmdline/dpdk-cmdline-gen: generate boilerplate for simple cmds
> > 
> >  lib/cmdline/dpdk-cmdline-gen.py | 143 ++++++++++++++++++++++++++++++++
> >  1 file changed, 143 insertions(+)
> >  create mode 100755 lib/cmdline/dpdk-cmdline-gen.py
> 
> Ok, but this surely is a solved problem is some other open source library.
> Or does every project reinvent this parsing?
> 
> Seems to me that something like bison/flex (ie yacc/lex) would be better
> in terms of expressing the syntax separate from the implementation.

Sure, but it has been many, many years since I last even looked at those
tools, so it was faster for me to just use this python script for quick
cmdline generation when prototyping things! :-) I wanted something that
worked for me, so I'm just sharing it with the community in case others
find it useful.

If you have time, please do indeed come up with a lex/yacc generator
instead, it would indeed be a better solution. [If not, I suggest we take
what we can get if it works! :-)]

/Bruce

[RFC PATCH v2 0/5] use script to simplify use of cmdline lib

[Bruce Richardson]

The DPDK commandline library is widely used by apps and examples within
DPDK, but requires a lot of boilerplate code definitions in order to
used. We can improve this situation by creating a simple python script
to automatically generate the boilerplate from a list of commands.

This RFC v2 contains the (slightly enhanced from v1) script, as well as
a set of three patches showing its use, by converting three examples
to use the script instead of having the hard-coded boilerplate. Once
the script is used, adding a new command becomes as simple as adding
the desired command to the .list file, and then writing the required
function which will be called for that command. No other boilerplate
coding is necessary.

Script obviously does not cover the full range of capabilities of the
commandline lib, but does cover the most used parts - with one
exception: it does not yet support defining the help text for a
command (hence the fact this is another RFC). If there is a general
acceptance that this is worthwhile, that gap can be closed, making
the shortened boiler-plate-free examples fully equivalent to the
original code.

V2-RFC:
* Add support for IP addresses in commands
* Move to buildtools directory and make installable
* Convert 3 examples to use script, and eliminate their boilerplate

Bruce Richardson (5):
  buildtools/dpdk-cmdline-gen: generate boilerplate for simple cmds
  examples/simple_mp: convert to use cmdline script
  examples/hotplug_mp: auto-generate cmdline boilerplate
  buildtools/dpdk-cmdline-gen: add IP address support
  examples/bond: auto-generate cmdline boilerplate

 buildtools/dpdk-cmdline-gen.py                | 151 ++++++++++++++++
 buildtools/meson.build                        |   7 +
 examples/bond/Makefile                        |  10 +-
 examples/bond/commands.list                   |   6 +
 examples/bond/main.c                          | 161 ++----------------
 examples/bond/main.h                          |  10 --
 examples/bond/meson.build                     |   8 +
 examples/multi_process/hotplug_mp/Makefile    |  10 +-
 examples/multi_process/hotplug_mp/commands.c  | 147 ++--------------
 examples/multi_process/hotplug_mp/commands.h  |  10 --
 .../multi_process/hotplug_mp/commands.list    |   5 +
 examples/multi_process/hotplug_mp/meson.build |   9 +
 examples/multi_process/simple_mp/Makefile     |  10 +-
 examples/multi_process/simple_mp/meson.build  |   9 +
 .../multi_process/simple_mp/mp_commands.c     | 106 ++----------
 .../multi_process/simple_mp/mp_commands.h     |  14 --
 .../multi_process/simple_mp/mp_commands.list  |   3 +
 17 files changed, 257 insertions(+), 419 deletions(-)
 create mode 100755 buildtools/dpdk-cmdline-gen.py
 create mode 100644 examples/bond/commands.list
 delete mode 100644 examples/bond/main.h
 delete mode 100644 examples/multi_process/hotplug_mp/commands.h
 create mode 100644 examples/multi_process/hotplug_mp/commands.list
 delete mode 100644 examples/multi_process/simple_mp/mp_commands.h
 create mode 100644 examples/multi_process/simple_mp/mp_commands.list

--
2.39.2

[RFC PATCH v2 1/5] buildtools/dpdk-cmdline-gen: generate boilerplate for simple cmds

[Bruce Richardson]

Provide a script for application developers to quickly generate the
boilerplate code necessary for using the cmdline library.

This initial version works only with commands using simple strings and
numbers, but this is sufficient for many use-cases. Future extensions
could, no doubt, expand support to expose more capabilities of the
library.

Example of use:
The script takes an input file with a list of commands the user wants in
the app, where the parameter variables are tagged with the type.
For example:

	$ cat commands.list
	list
	add <UINT16>x <UINT16>y
	echo <STRING>message
	add socket <STRING>path
	quit

When run through the script as "./dpdk-cmdline-gen.py commands.list",
the output will be the contents of a header file with all the
boilerplate necessary for a commandline instance with those commands.

If the flag --stubs is passed, an output header filename must also be
passed, in which case both a header file with the definitions and a C
file with function stubs in it is written to disk. The separation is so
that the header file can be rewritten at any future point to add more
commands, while the C file can be kept as-is and extended by the user
with any additional functions needed.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 buildtools/dpdk-cmdline-gen.py | 146 +++++++++++++++++++++++++++++++++
 buildtools/meson.build         |   7 ++
 2 files changed, 153 insertions(+)
 create mode 100755 buildtools/dpdk-cmdline-gen.py

diff --git a/buildtools/dpdk-cmdline-gen.py b/buildtools/dpdk-cmdline-gen.py
new file mode 100755
index 0000000000..1ddd8b6bbb
--- /dev/null
+++ b/buildtools/dpdk-cmdline-gen.py
@@ -0,0 +1,146 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2023 Intel Corporation
+#
+
+import sys
+import os
+import argparse
+
+PARSE_FN_PARAMS = "void *parsed_result, struct cmdline *cl, void *data"
+PARSE_FN_BODY = """
+    /* TODO: command action */
+    RTE_SET_USED(parsed_result);
+    RTE_SET_USED(cl);
+    RTE_SET_USED(data);
+"""
+
+def process_command(tokens, cfile):
+    name = []
+
+    if tokens[0].startswith('<'):
+        print("Error: each command must start with at least one literal string",
+                file=sys.stderr)
+        sys.exit(1)
+    for t in tokens:
+        if t.startswith('<'):
+            break;
+        name.append(t)
+    name = "_".join(name)
+
+    result_struct = []
+    initializers = []
+    token_list = []
+    for t in tokens:
+        if t.startswith('<'):
+            t_type, t_name = t[1:].split('>')
+            t_val = "NULL"
+        else:
+            t_type = "STRING"
+            t_name = t
+            t_val = f'"{t}"'
+
+        if t_type == "STRING":
+            result_struct.append(f"\tcmdline_fixed_string_t {t_name};")
+            initializers.append(f"static cmdline_parse_token_string_t cmd_{name}_{t_name}_tok =\n"
+                    + f"\tTOKEN_STRING_INITIALIZER(struct cmd_{name}_result, {t_name}, {t_val});")
+        elif t_type in ['UINT8', 'UINT16', 'UINT32', 'UINT64',
+                    'INT8', 'INT16', 'INT32', 'INT64']:
+            result_struct.append(f"\t{t_type.lower()}_t {t_name};")
+            initializers.append(f"static cmdline_parse_token_num_t cmd_{name}_{t_name}_tok =\n"
+                    + f"\tTOKEN_NUM_INITIALIZER(struct cmd_{name}_result, {t_name}, RTE_{t_type});")
+        else:
+            print(f"Error: unknown token-type {t}", file=sys.stderr)
+            sys.exit(1)
+        token_list.append(f"cmd_{name}_{t_name}_tok")
+
+    print(f"/* Auto-generated handling for command '{' '.join(tokens)}' */")
+    # output function prototype
+    func_sig = f"void\ncmd_{name}_parsed({PARSE_FN_PARAMS})"
+    print(f"extern {func_sig};\n")
+    # output function template if C file being written
+    if (cfile):
+        print(f"{func_sig}\n{{{PARSE_FN_BODY}}}\n", file=cfile)
+    # output result data structure
+    print(f"struct cmd_{name}_result {{\n"
+            + '\n'.join(result_struct)
+            + "\n};\n")
+    # output the initializer tokens
+    print("\n".join(initializers) + '\n')
+    # output the instance structure
+    print(f"static cmdline_parse_inst_t cmd_{name} = {{\n"
+            + f'\t.f = cmd_{name}_parsed,\n'
+            + f'\t.data = NULL,\n'
+            + f'\t.help_str = "",\n'
+            + f'\t.tokens = {{')
+    for t in token_list:
+        print(f"\t\t(void *)&{t},")
+    print("\t\tNULL\n"
+            + "\t}\n"
+            + "};\n")
+
+    # return the instance structure name
+    return f"cmd_{name}"
+
+def process_commands(infile, hfile, cfile, ctxname):
+    instances = []
+
+    # redirect stdout to output the header, to save passing file= each print
+    old_sys_stdout = sys.stdout
+    sys.stdout = hfile
+
+    print(f"/* File autogenerated by {sys.argv[0]} */")
+    print('#ifndef GENERATED_COMMANDS_H')
+    print('#define GENERATED_COMMANDS_H')
+    print("#include <rte_common.h>")
+    print("#include <cmdline.h>")
+    print("#include <cmdline_parse_string.h>")
+    print("#include <cmdline_parse_num.h>")
+    print("")
+
+    for line in infile.readlines():
+        if line.lstrip().startswith('#'):
+            continue
+        instances.append(process_command(line.strip().split(), cfile))
+
+    print(f'static __rte_used cmdline_parse_ctx_t {ctxname}[] = {{')
+    for inst in instances:
+        print(f'\t&{inst},')
+    print('\tNULL')
+    print('};\n')
+    print('#endif /* GENERATED_COMMANDS_H */')
+
+    sys.stdout = old_sys_stdout
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--stubs", action="store_true",
+            help="Produce C file with empty function stubs for each command")
+    ap.add_argument("--output-file", "-o", default="-",
+            help="Output header filename [default to stdout]")
+    ap.add_argument("--context-name", default="ctx",
+            help="Name given to the cmdline context variable in the output header [default=ctx]")
+    ap.add_argument("infile", type=argparse.FileType('r'),
+            help="File with list of commands")
+    args = ap.parse_args()
+
+    if not args.stubs:
+        if args.output_file == '-':
+            process_commands(args.infile, sys.stdout, None, args.context_name)
+        else:
+            with open(args.output_file, "w") as hfile:
+                process_commands(args.infile, hfile, None, args.context_name)
+    else:
+        if not args.output_file.endswith('.h'):
+            print("Error: output filename must end with '.h' extension when creating stubs",
+                    file=sys.stderr)
+            sys.exit(1)
+
+        cfilename = args.output_file[:-2] + '.c'
+        with open(args.output_file, "w") as hfile:
+            with open(cfilename, "w") as cfile:
+                print(f"#include \"{args.output_file}\"\n", file=cfile)
+                process_commands(args.infile, hfile, cfile, args.context_name)
+
+if __name__ == "__main__":
+    main()
diff --git a/buildtools/meson.build b/buildtools/meson.build
index 948ac17dd2..72447b60a0 100644
--- a/buildtools/meson.build
+++ b/buildtools/meson.build
@@ -19,6 +19,13 @@ get_cpu_count_cmd = py3 + files('get-cpu-count.py')
 get_numa_count_cmd = py3 + files('get-numa-count.py')
 get_test_suites_cmd = py3 + files('get-test-suites.py')
 has_hugepages_cmd = py3 + files('has-hugepages.py')
+cmdline_gen_cmd = py3 + files('dpdk-cmdline-gen.py')
+
+# install any build tools that end-users might want also
+install_data([
+            'dpdk-cmdline-gen.py',
+        ],
+        install_dir: 'bin')
 
 # select library and object file format
 pmdinfo = py3 + files('gen-pmdinfo-cfile.py') + [meson.current_build_dir()]
-- 
2.39.2

[RFC PATCH v2 2/5] examples/simple_mp: auto-generate cmdline boilerplate

[Bruce Richardson]

Use the dpdk-cmdline-gen script to autogenerate all the boilerplate
structs and defines for the commandline part of the app.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 examples/multi_process/simple_mp/Makefile     |  10 +-
 examples/multi_process/simple_mp/meson.build  |   9 ++
 .../multi_process/simple_mp/mp_commands.c     | 106 ++----------------
 .../multi_process/simple_mp/mp_commands.h     |  14 ---
 .../multi_process/simple_mp/mp_commands.list  |   3 +
 5 files changed, 29 insertions(+), 113 deletions(-)
 delete mode 100644 examples/multi_process/simple_mp/mp_commands.h
 create mode 100644 examples/multi_process/simple_mp/mp_commands.list

diff --git a/examples/multi_process/simple_mp/Makefile b/examples/multi_process/simple_mp/Makefile
index 1d0a260e64..b460f53ece 100644
--- a/examples/multi_process/simple_mp/Makefile
+++ b/examples/multi_process/simple_mp/Makefile
@@ -6,6 +6,7 @@ APP = simple_mp

 # all source are stored in SRCS-y
 SRCS-y := main.c mp_commands.c
+SRC-DEPS := build/mp_commands.h

 PKGCONF ?= pkg-config

@@ -22,10 +23,13 @@ static: build/$(APP)-static
 	ln -sf $(APP)-static build/$(APP)

 PC_FILE := $(shell $(PKGCONF) --path libdpdk 2>/dev/null)
-CFLAGS += -O3 $(shell $(PKGCONF) --cflags libdpdk)
+CFLAGS += -O3 $(shell $(PKGCONF) --cflags libdpdk) -I build/
 LDFLAGS_SHARED = $(shell $(PKGCONF) --libs libdpdk)
 LDFLAGS_STATIC = $(shell $(PKGCONF) --static --libs libdpdk)

+build/mp_commands.h: mp_commands.list Makefile
+	dpdk-cmdline-gen.py -o $@ --context-name=simple_mp_ctx $<
+
 ifeq ($(MAKECMDGOALS),static)
 # check for broken pkg-config
 ifeq ($(shell echo $(LDFLAGS_STATIC) | grep 'whole-archive.*l:lib.*no-whole-archive'),)
@@ -36,10 +40,10 @@ endif

 CFLAGS += -DALLOW_EXPERIMENTAL_API

-build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
+build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build $(SRC-DEPS)
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)

-build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build
+build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build $(SRC-DEPS)
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_STATIC)

 build:
diff --git a/examples/multi_process/simple_mp/meson.build b/examples/multi_process/simple_mp/meson.build
index 359af4384d..e99b7a3f6f 100644
--- a/examples/multi_process/simple_mp/meson.build
+++ b/examples/multi_process/simple_mp/meson.build
@@ -7,7 +7,16 @@
 # DPDK instance, use 'make'

 allow_experimental_apis = true
+
+cmd_h = custom_target('commands_hdr',
+	output: 'mp_commands.h',
+	input: files('mp_commands.list'),
+	capture: true,
+	command: [cmdline_gen_cmd, '--context-name=simple_mp_ctx', '@INPUT@']
+)
+
 sources = files(
         'mp_commands.c',
         'main.c',
 )
+sources += cmd_h
diff --git a/examples/multi_process/simple_mp/mp_commands.c b/examples/multi_process/simple_mp/mp_commands.c
index a5f91b00be..df9fa94208 100644
--- a/examples/multi_process/simple_mp/mp_commands.c
+++ b/examples/multi_process/simple_mp/mp_commands.c
@@ -1,44 +1,18 @@
 /* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2010-2014 Intel Corporation
+ * Copyright(c) 2010-2023 Intel Corporation
  */
-#include <stdint.h>
-#include <string.h>
-#include <stdlib.h>
-#include <stdarg.h>
-#include <inttypes.h>
-#include <stdio.h>
-#include <termios.h>
-#include <errno.h>
-#include <sys/queue.h>
-
-#include <rte_common.h>
-#include <rte_memory.h>
-#include <rte_eal.h>
-#include <rte_branch_prediction.h>
-#include <rte_launch.h>
-#include <rte_log.h>
-#include <rte_per_lcore.h>
-#include <rte_lcore.h>
 #include <rte_ring.h>
-#include <rte_debug.h>
 #include <rte_mempool.h>
 #include <rte_string_fns.h>

-#include <cmdline_rdline.h>
-#include <cmdline_parse.h>
-#include <cmdline_parse_string.h>
-#include <cmdline_socket.h>
-#include <cmdline.h>
 #include "mp_commands.h"

-/**********************************************************/
-
-struct cmd_send_result {
-	cmdline_fixed_string_t action;
-	cmdline_fixed_string_t message;
-};
+extern struct rte_ring *send_ring, *recv_ring;
+extern struct rte_mempool *message_pool;
+extern volatile int quit;

-static void cmd_send_parsed(void *parsed_result,
+void
+cmd_send_parsed(void *parsed_result,
 		__rte_unused struct cmdline *cl,
 		__rte_unused void *data)
 {
@@ -54,29 +28,8 @@ static void cmd_send_parsed(void *parsed_result,
 	}
 }

-cmdline_parse_token_string_t cmd_send_action =
-	TOKEN_STRING_INITIALIZER(struct cmd_send_result, action, "send");
-cmdline_parse_token_string_t cmd_send_message =
-	TOKEN_STRING_INITIALIZER(struct cmd_send_result, message, NULL);
-
-cmdline_parse_inst_t cmd_send = {
-	.f = cmd_send_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "send a string to another process",
-	.tokens = {        /* token list, NULL terminated */
-			(void *)&cmd_send_action,
-			(void *)&cmd_send_message,
-			NULL,
-	},
-};
-
-/**********************************************************/
-
-struct cmd_quit_result {
-	cmdline_fixed_string_t quit;
-};
-
-static void cmd_quit_parsed(__rte_unused void *parsed_result,
+void
+cmd_quit_parsed(__rte_unused void *parsed_result,
 			    struct cmdline *cl,
 			    __rte_unused void *data)
 {
@@ -84,26 +37,8 @@ static void cmd_quit_parsed(__rte_unused void *parsed_result,
 	cmdline_quit(cl);
 }

-cmdline_parse_token_string_t cmd_quit_quit =
-	TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit");
-
-cmdline_parse_inst_t cmd_quit = {
-	.f = cmd_quit_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "close the application",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_quit_quit,
-		NULL,
-	},
-};
-
-/**********************************************************/
-
-struct cmd_help_result {
-	cmdline_fixed_string_t help;
-};
-
-static void cmd_help_parsed(__rte_unused void *parsed_result,
+void
+cmd_help_parsed(__rte_unused void *parsed_result,
 			    struct cmdline *cl,
 			    __rte_unused void *data)
 {
@@ -112,24 +47,3 @@ static void cmd_help_parsed(__rte_unused void *parsed_result,
 			"send commands to the simple app. Commands supported are:\n\n"
 			"- send [string]\n" "- help\n" "- quit\n\n");
 }
-
-cmdline_parse_token_string_t cmd_help_help =
-	TOKEN_STRING_INITIALIZER(struct cmd_help_result, help, "help");
-
-cmdline_parse_inst_t cmd_help = {
-	.f = cmd_help_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "show help",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_help_help,
-		NULL,
-	},
-};
-
-/****** CONTEXT (list of instruction) */
-cmdline_parse_ctx_t simple_mp_ctx[] = {
-		(cmdline_parse_inst_t *)&cmd_send,
-		(cmdline_parse_inst_t *)&cmd_quit,
-		(cmdline_parse_inst_t *)&cmd_help,
-	NULL,
-};
diff --git a/examples/multi_process/simple_mp/mp_commands.h b/examples/multi_process/simple_mp/mp_commands.h
deleted file mode 100644
index 5d67413e7c..0000000000
--- a/examples/multi_process/simple_mp/mp_commands.h
+++ /dev/null
@@ -1,14 +0,0 @@
-/* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2010-2014 Intel Corporation
- */
-
-#ifndef _SIMPLE_MP_COMMANDS_H_
-#define _SIMPLE_MP_COMMANDS_H_
-
-extern struct rte_ring *send_ring;
-extern struct rte_mempool *message_pool;
-extern volatile int quit;
-
-extern cmdline_parse_ctx_t simple_mp_ctx[];
-
-#endif /* _SIMPLE_MP_COMMANDS_H_ */
diff --git a/examples/multi_process/simple_mp/mp_commands.list b/examples/multi_process/simple_mp/mp_commands.list
new file mode 100644
index 0000000000..da452249a6
--- /dev/null
+++ b/examples/multi_process/simple_mp/mp_commands.list
@@ -0,0 +1,3 @@
+send <STRING>message
+help
+quit
\ No newline at end of file
--
2.39.2

[RFC PATCH v2 3/5] examples/hotplug_mp: auto-generate cmdline boilerplate

[Bruce Richardson]

Use the dpdk-cmdline-gen script to autogenerate all the boilerplate
structs and defines for the commandline part of the app.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 examples/multi_process/hotplug_mp/Makefile    |  10 +-
 examples/multi_process/hotplug_mp/commands.c  | 147 ++----------------
 examples/multi_process/hotplug_mp/commands.h  |  10 --
 .../multi_process/hotplug_mp/commands.list    |   5 +
 examples/multi_process/hotplug_mp/meson.build |   9 ++
 5 files changed, 34 insertions(+), 147 deletions(-)
 delete mode 100644 examples/multi_process/hotplug_mp/commands.h
 create mode 100644 examples/multi_process/hotplug_mp/commands.list

diff --git a/examples/multi_process/hotplug_mp/Makefile b/examples/multi_process/hotplug_mp/Makefile
index 6b20d6e49a..77e8fe2737 100644
--- a/examples/multi_process/hotplug_mp/Makefile
+++ b/examples/multi_process/hotplug_mp/Makefile
@@ -6,6 +6,7 @@ APP = hotplug_mp
 
 # all source are stored in SRCS-y
 SRCS-y := main.c commands.c
+SRC-DEPS := build/commands.h
 
 PKGCONF ?= pkg-config
 
@@ -22,10 +23,13 @@ static: build/$(APP)-static
 	ln -sf $(APP)-static build/$(APP)
 
 PC_FILE := $(shell $(PKGCONF) --path libdpdk 2>/dev/null)
-CFLAGS += -O3 $(shell $(PKGCONF) --cflags libdpdk)
+CFLAGS += -O3 $(shell $(PKGCONF) --cflags libdpdk) -I build/
 LDFLAGS_SHARED = $(shell $(PKGCONF) --libs libdpdk)
 LDFLAGS_STATIC = $(shell $(PKGCONF) --static --libs libdpdk)
 
+build/commands.h: commands.list Makefile
+	dpdk-cmdline-gen.py -o $@ --context-name=main_ctx $<
+
 ifeq ($(MAKECMDGOALS),static)
 # check for broken pkg-config
 ifeq ($(shell echo $(LDFLAGS_STATIC) | grep 'whole-archive.*l:lib.*no-whole-archive'),)
@@ -36,10 +40,10 @@ endif
 
 CFLAGS += -DALLOW_EXPERIMENTAL_API
 
-build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
+build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build $(SRC-DEPS)
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
 
-build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build
+build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build  $(SRC-DEPS)
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_STATIC)
 
 build:
diff --git a/examples/multi_process/hotplug_mp/commands.c b/examples/multi_process/hotplug_mp/commands.c
index 88f44e00a0..900eb9f774 100644
--- a/examples/multi_process/hotplug_mp/commands.c
+++ b/examples/multi_process/hotplug_mp/commands.c
@@ -1,24 +1,12 @@
 /* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2018 Intel Corporation.
+ * Copyright(c) 2018-2023 Intel Corporation.
  */
 
-#include <cmdline_rdline.h>
-#include <cmdline_parse.h>
-#include <cmdline_parse_ipaddr.h>
-#include <cmdline_parse_num.h>
-#include <cmdline_parse_string.h>
-#include <cmdline.h>
-
 #include <rte_bus.h>
 #include <rte_ethdev.h>
+#include "commands.h"
 
-/**********************************************************/
-
-struct cmd_help_result {
-	cmdline_fixed_string_t help;
-};
-
-static void cmd_help_parsed(__rte_unused void *parsed_result,
+void cmd_help_parsed(__rte_unused void *parsed_result,
 			    struct cmdline *cl,
 			    __rte_unused void *data)
 {
@@ -29,52 +17,16 @@ static void cmd_help_parsed(__rte_unused void *parsed_result,
 		       "- list\n\n");
 }
 
-cmdline_parse_token_string_t cmd_help_help =
-	TOKEN_STRING_INITIALIZER(struct cmd_help_result, help, "help");
-
-cmdline_parse_inst_t cmd_help = {
-	.f = cmd_help_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "show help",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_help_help,
-		NULL,
-	},
-};
-
-/**********************************************************/
-
-struct cmd_quit_result {
-	cmdline_fixed_string_t quit;
-};
-
-static void cmd_quit_parsed(__rte_unused void *parsed_result,
+void
+cmd_quit_parsed(__rte_unused void *parsed_result,
 			    struct cmdline *cl,
 			    __rte_unused void *data)
 {
 	cmdline_quit(cl);
 }
 
-cmdline_parse_token_string_t cmd_quit_quit =
-	TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit");
-
-cmdline_parse_inst_t cmd_quit = {
-	.f = cmd_quit_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "quit",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_quit_quit,
-		NULL,
-	},
-};
-
-/**********************************************************/
-
-struct cmd_list_result {
-	cmdline_fixed_string_t list;
-};
-
-static void cmd_list_parsed(__rte_unused void *parsed_result,
+void
+cmd_list_parsed(__rte_unused void *parsed_result,
 			    struct cmdline *cl,
 			    __rte_unused void *data)
 {
@@ -92,31 +44,12 @@ static void cmd_list_parsed(__rte_unused void *parsed_result,
 	}
 }
 
-cmdline_parse_token_string_t cmd_list_list =
-	TOKEN_STRING_INITIALIZER(struct cmd_list_result, list, "list");
-
-cmdline_parse_inst_t cmd_list = {
-	.f = cmd_list_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "list all devices",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_list_list,
-		NULL,
-	},
-};
-
-/**********************************************************/
-
-struct cmd_dev_attach_result {
-	cmdline_fixed_string_t attach;
-	cmdline_fixed_string_t devargs;
-};
-
-static void cmd_dev_attach_parsed(void *parsed_result,
+void
+cmd_attach_parsed(void *parsed_result,
 				  struct cmdline *cl,
 				  __rte_unused void *data)
 {
-	struct cmd_dev_attach_result *res = parsed_result;
+	struct cmd_attach_result *res = parsed_result;
 	struct rte_devargs da;
 
 	memset(&da, 0, sizeof(da));
@@ -134,35 +67,12 @@ static void cmd_dev_attach_parsed(void *parsed_result,
 	rte_devargs_reset(&da);
 }
 
-cmdline_parse_token_string_t cmd_dev_attach_attach =
-	TOKEN_STRING_INITIALIZER(struct cmd_dev_attach_result, attach,
-				 "attach");
-cmdline_parse_token_string_t cmd_dev_attach_devargs =
-	TOKEN_STRING_INITIALIZER(struct cmd_dev_attach_result, devargs, NULL);
-
-cmdline_parse_inst_t cmd_attach_device = {
-	.f = cmd_dev_attach_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "attach a device",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_dev_attach_attach,
-		(void *)&cmd_dev_attach_devargs,
-		NULL,
-	},
-};
-
-/**********************************************************/
-
-struct cmd_dev_detach_result {
-	cmdline_fixed_string_t detach;
-	cmdline_fixed_string_t devargs;
-};
-
-static void cmd_dev_detach_parsed(void *parsed_result,
+void
+cmd_detach_parsed(void *parsed_result,
 				   struct cmdline *cl,
 				   __rte_unused void *data)
 {
-	struct cmd_dev_detach_result *res = parsed_result;
+	struct cmd_detach_result *res = parsed_result;
 	struct rte_devargs da;
 
 	memset(&da, 0, sizeof(da));
@@ -181,34 +91,3 @@ static void cmd_dev_detach_parsed(void *parsed_result,
 			da.name);
 	rte_devargs_reset(&da);
 }
-
-cmdline_parse_token_string_t cmd_dev_detach_detach =
-	TOKEN_STRING_INITIALIZER(struct cmd_dev_detach_result, detach,
-				 "detach");
-
-cmdline_parse_token_string_t cmd_dev_detach_devargs =
-	TOKEN_STRING_INITIALIZER(struct cmd_dev_detach_result, devargs, NULL);
-
-cmdline_parse_inst_t cmd_detach_device = {
-	.f = cmd_dev_detach_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "detach a device",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_dev_detach_detach,
-		(void *)&cmd_dev_detach_devargs,
-		NULL,
-	},
-};
-
-/**********************************************************/
-/**********************************************************/
-/****** CONTEXT (list of instruction) */
-
-cmdline_parse_ctx_t main_ctx[] = {
-	(cmdline_parse_inst_t *)&cmd_help,
-	(cmdline_parse_inst_t *)&cmd_quit,
-	(cmdline_parse_inst_t *)&cmd_list,
-	(cmdline_parse_inst_t *)&cmd_attach_device,
-	(cmdline_parse_inst_t *)&cmd_detach_device,
-	NULL,
-};
diff --git a/examples/multi_process/hotplug_mp/commands.h b/examples/multi_process/hotplug_mp/commands.h
deleted file mode 100644
index afcf177dba..0000000000
--- a/examples/multi_process/hotplug_mp/commands.h
+++ /dev/null
@@ -1,10 +0,0 @@
-/* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2018 Intel Corporation
- */
-
-#ifndef _COMMANDS_H_
-#define _COMMANDS_H_
-
-extern cmdline_parse_ctx_t main_ctx[];
-
-#endif /* _COMMANDS_H_ */
diff --git a/examples/multi_process/hotplug_mp/commands.list b/examples/multi_process/hotplug_mp/commands.list
new file mode 100644
index 0000000000..683e53cb0e
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/commands.list
@@ -0,0 +1,5 @@
+help
+quit
+list
+attach <STRING>devargs
+detach <STRING>devargs
\ No newline at end of file
diff --git a/examples/multi_process/hotplug_mp/meson.build b/examples/multi_process/hotplug_mp/meson.build
index a1ad98ca2e..7a0e9ca47a 100644
--- a/examples/multi_process/hotplug_mp/meson.build
+++ b/examples/multi_process/hotplug_mp/meson.build
@@ -7,7 +7,16 @@
 # DPDK instance, use 'make'
 
 allow_experimental_apis = true
+
+cmd_h = custom_target('commands_hdr',
+	output: 'commands.h',
+	input: files('commands.list'),
+	capture: true,
+	command: [cmdline_gen_cmd, '--context-name=main_ctx', '@INPUT@']
+)
+
 sources = files(
         'commands.c',
         'main.c',
 )
+sources += cmd_h
-- 
2.39.2

[RFC PATCH v2 4/5] buildtools/dpdk-cmdline-gen: add IP address support

[Bruce Richardson]

Add support for apps to have IP addresses as command parameters

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 buildtools/dpdk-cmdline-gen.py | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/buildtools/dpdk-cmdline-gen.py b/buildtools/dpdk-cmdline-gen.py
index 1ddd8b6bbb..c1f8a6f1da 100755
--- a/buildtools/dpdk-cmdline-gen.py
+++ b/buildtools/dpdk-cmdline-gen.py
@@ -49,6 +49,10 @@ def process_command(tokens, cfile):
             result_struct.append(f"\t{t_type.lower()}_t {t_name};")
             initializers.append(f"static cmdline_parse_token_num_t cmd_{name}_{t_name}_tok =\n"
                     + f"\tTOKEN_NUM_INITIALIZER(struct cmd_{name}_result, {t_name}, RTE_{t_type});")
+        elif t_type in ['IP', 'IP_ADDR', 'IPADDR']:
+            result_struct.append(f"\tcmdline_ipaddr_t {t_name};")
+            initializers.append(f"cmdline_parse_token_ipaddr_t cmd_{name}_{t_name}_tok =\n"
+                    + f"\tTOKEN_IPV4_INITIALIZER(struct cmd_{name}_result, {t_name});")
         else:
             print(f"Error: unknown token-type {t}", file=sys.stderr)
             sys.exit(1)
@@ -96,6 +100,7 @@ def process_commands(infile, hfile, cfile, ctxname):
     print("#include <cmdline.h>")
     print("#include <cmdline_parse_string.h>")
     print("#include <cmdline_parse_num.h>")
+    print("#include <cmdline_parse_ipaddr.h>")
     print("")
 
     for line in infile.readlines():
-- 
2.39.2

[RFC PATCH v2 5/5] examples/bond: auto-generate cmdline boilerplate

[Bruce Richardson]

Use the dpdk-cmdline-gen script to autogenerate all the boilerplate
structs and defines for the commandline part of the app.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 examples/bond/Makefile      |  10 ++-
 examples/bond/commands.list |   6 ++
 examples/bond/main.c        | 161 ++++--------------------------------
 examples/bond/main.h        |  10 ---
 examples/bond/meson.build   |   8 ++
 5 files changed, 36 insertions(+), 159 deletions(-)
 create mode 100644 examples/bond/commands.list
 delete mode 100644 examples/bond/main.h

diff --git a/examples/bond/Makefile b/examples/bond/Makefile
index ad711a5bee..0a559ad82b 100644
--- a/examples/bond/Makefile
+++ b/examples/bond/Makefile
@@ -6,6 +6,7 @@ APP = bond_app
 
 # all source are stored in SRCS-y
 SRCS-y := main.c
+SRC-DEPS := build/commands.h
 
 PKGCONF ?= pkg-config
 
@@ -24,10 +25,13 @@ static: build/$(APP)-static
 LDFLAGS += -lrte_net_bond
 
 PC_FILE := $(shell $(PKGCONF) --path libdpdk 2>/dev/null)
-CFLAGS += -O3 $(shell $(PKGCONF) --cflags libdpdk)
+CFLAGS += -O3 $(shell $(PKGCONF) --cflags libdpdk) -I build/
 LDFLAGS_SHARED = $(shell $(PKGCONF) --libs libdpdk)
 LDFLAGS_STATIC = $(shell $(PKGCONF) --static --libs libdpdk)
 
+build/commands.h: commands.list Makefile
+	dpdk-cmdline-gen.py -o $@ --context-name=main_ctx $<
+
 ifeq ($(MAKECMDGOALS),static)
 # check for broken pkg-config
 ifeq ($(shell echo $(LDFLAGS_STATIC) | grep 'whole-archive.*l:lib.*no-whole-archive'),)
@@ -38,10 +42,10 @@ endif
 
 CFLAGS += -DALLOW_EXPERIMENTAL_API
 
-build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
+build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build $(SRC-DEPS)
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
 
-build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build
+build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build  $(SRC-DEPS)
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_STATIC)
 
 build:
diff --git a/examples/bond/commands.list b/examples/bond/commands.list
new file mode 100644
index 0000000000..08e1a3fea8
--- /dev/null
+++ b/examples/bond/commands.list
@@ -0,0 +1,6 @@
+send <IP>ip
+start
+stop
+show
+help
+quit
\ No newline at end of file
diff --git a/examples/bond/main.c b/examples/bond/main.c
index 9b076bb39f..e893005701 100644
--- a/examples/bond/main.c
+++ b/examples/bond/main.c
@@ -45,16 +45,8 @@
 #include <rte_cpuflags.h>
 #include <rte_eth_bond.h>
 
-#include <cmdline_rdline.h>
-#include <cmdline_parse.h>
-#include <cmdline_parse_num.h>
-#include <cmdline_parse_string.h>
-#include <cmdline_parse_ipaddr.h>
-#include <cmdline_parse_etheraddr.h>
 #include <cmdline_socket.h>
-#include <cmdline.h>
-
-#include "main.h"
+#include "commands.h"
 
 #define RTE_LOGTYPE_DCB RTE_LOGTYPE_USER1
 
@@ -462,11 +454,7 @@ static int lcore_main(__rte_unused void *arg1)
 	return 0;
 }
 
-struct cmd_obj_send_result {
-	cmdline_fixed_string_t action;
-	cmdline_ipaddr_t ip;
-};
-static inline void get_string(struct cmd_obj_send_result *res, char *buf, uint8_t size)
+static inline void get_string(struct cmd_send_result *res, char *buf, uint8_t size)
 {
 	snprintf(buf, size, NIPQUAD_FMT,
 		((unsigned)((unsigned char *)&(res->ip.addr.ipv4))[0]),
@@ -475,12 +463,11 @@ static inline void get_string(struct cmd_obj_send_result *res, char *buf, uint8_
 		((unsigned)((unsigned char *)&(res->ip.addr.ipv4))[3])
 		);
 }
-static void cmd_obj_send_parsed(void *parsed_result,
-		__rte_unused struct cmdline *cl,
-			       __rte_unused void *data)
+void
+cmd_send_parsed(void *parsed_result, __rte_unused struct cmdline *cl, __rte_unused void *data)
 {
 
-	struct cmd_obj_send_result *res = parsed_result;
+	struct cmd_send_result *res = parsed_result;
 	char ip_str[INET6_ADDRSTRLEN];
 
 	struct rte_ether_addr bond_mac_addr;
@@ -544,29 +531,8 @@ static void cmd_obj_send_parsed(void *parsed_result,
 	cmdline_printf(cl, "\n");
 }
 
-cmdline_parse_token_string_t cmd_obj_action_send =
-	TOKEN_STRING_INITIALIZER(struct cmd_obj_send_result, action, "send");
-cmdline_parse_token_ipaddr_t cmd_obj_ip =
-	TOKEN_IPV4_INITIALIZER(struct cmd_obj_send_result, ip);
-
-cmdline_parse_inst_t cmd_obj_send = {
-	.f = cmd_obj_send_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "send client_ip",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_obj_action_send,
-		(void *)&cmd_obj_ip,
-		NULL,
-	},
-};
-
-struct cmd_start_result {
-	cmdline_fixed_string_t start;
-};
-
-static void cmd_start_parsed(__rte_unused void *parsed_result,
-			       struct cmdline *cl,
-			       __rte_unused void *data)
+void
+cmd_start_parsed(__rte_unused void *parsed_result, struct cmdline *cl, __rte_unused void *data)
 {
 	int worker_core_id = rte_lcore_id();
 
@@ -605,26 +571,8 @@ static void cmd_start_parsed(__rte_unused void *parsed_result,
 		);
 }
 
-cmdline_parse_token_string_t cmd_start_start =
-	TOKEN_STRING_INITIALIZER(struct cmd_start_result, start, "start");
-
-cmdline_parse_inst_t cmd_start = {
-	.f = cmd_start_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "starts listening if not started at startup",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_start_start,
-		NULL,
-	},
-};
-
-struct cmd_help_result {
-	cmdline_fixed_string_t help;
-};
-
-static void cmd_help_parsed(__rte_unused void *parsed_result,
-			    struct cmdline *cl,
-			    __rte_unused void *data)
+void
+cmd_help_parsed(__rte_unused void *parsed_result, struct cmdline *cl, __rte_unused void *data)
 {
 	cmdline_printf(cl,
 			"ALB - link bonding mode 6 example\n"
@@ -637,26 +585,8 @@ static void cmd_help_parsed(__rte_unused void *parsed_result,
 		       );
 }
 
-cmdline_parse_token_string_t cmd_help_help =
-	TOKEN_STRING_INITIALIZER(struct cmd_help_result, help, "help");
-
-cmdline_parse_inst_t cmd_help = {
-	.f = cmd_help_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "show help",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_help_help,
-		NULL,
-	},
-};
-
-struct cmd_stop_result {
-	cmdline_fixed_string_t stop;
-};
-
-static void cmd_stop_parsed(__rte_unused void *parsed_result,
-			    struct cmdline *cl,
-			    __rte_unused void *data)
+void
+cmd_stop_parsed(__rte_unused void *parsed_result, struct cmdline *cl, __rte_unused void *data)
 {
 	rte_spinlock_lock(&global_flag_stru_p->lock);
 	if (global_flag_stru_p->LcoreMainIsRunning == 0)	{
@@ -678,26 +608,8 @@ static void cmd_stop_parsed(__rte_unused void *parsed_result,
 	rte_spinlock_unlock(&global_flag_stru_p->lock);
 }
 
-cmdline_parse_token_string_t cmd_stop_stop =
-	TOKEN_STRING_INITIALIZER(struct cmd_stop_result, stop, "stop");
-
-cmdline_parse_inst_t cmd_stop = {
-	.f = cmd_stop_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "this command do not handle any arguments",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_stop_stop,
-		NULL,
-	},
-};
-
-struct cmd_quit_result {
-	cmdline_fixed_string_t quit;
-};
-
-static void cmd_quit_parsed(__rte_unused void *parsed_result,
-			    struct cmdline *cl,
-			    __rte_unused void *data)
+void
+cmd_quit_parsed(__rte_unused void *parsed_result, struct cmdline *cl, __rte_unused void *data)
 {
 	rte_spinlock_lock(&global_flag_stru_p->lock);
 	if (global_flag_stru_p->LcoreMainIsRunning == 0)	{
@@ -721,26 +633,8 @@ static void cmd_quit_parsed(__rte_unused void *parsed_result,
 	cmdline_quit(cl);
 }
 
-cmdline_parse_token_string_t cmd_quit_quit =
-	TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit");
-
-cmdline_parse_inst_t cmd_quit = {
-	.f = cmd_quit_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "this command do not handle any arguments",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_quit_quit,
-		NULL,
-	},
-};
-
-struct cmd_show_result {
-	cmdline_fixed_string_t show;
-};
-
-static void cmd_show_parsed(__rte_unused void *parsed_result,
-			    struct cmdline *cl,
-			    __rte_unused void *data)
+void
+cmd_show_parsed(__rte_unused void *parsed_result, struct cmdline *cl, __rte_unused void *data)
 {
 	uint16_t slaves[16] = {0};
 	uint8_t len = 16;
@@ -772,31 +666,6 @@ static void cmd_show_parsed(__rte_unused void *parsed_result,
 	rte_spinlock_unlock(&global_flag_stru_p->lock);
 }
 
-cmdline_parse_token_string_t cmd_show_show =
-	TOKEN_STRING_INITIALIZER(struct cmd_show_result, show, "show");
-
-cmdline_parse_inst_t cmd_show = {
-	.f = cmd_show_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "this command do not handle any arguments",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_show_show,
-		NULL,
-	},
-};
-
-/****** CONTEXT (list of instruction) */
-
-cmdline_parse_ctx_t main_ctx[] = {
-	(cmdline_parse_inst_t *)&cmd_start,
-	(cmdline_parse_inst_t *)&cmd_obj_send,
-	(cmdline_parse_inst_t *)&cmd_stop,
-	(cmdline_parse_inst_t *)&cmd_show,
-	(cmdline_parse_inst_t *)&cmd_quit,
-	(cmdline_parse_inst_t *)&cmd_help,
-	NULL,
-};
-
 /* prompt function, called from main on MAIN lcore */
 static void prompt(__rte_unused void *arg1)
 {
diff --git a/examples/bond/main.h b/examples/bond/main.h
deleted file mode 100644
index f91ed9c885..0000000000
--- a/examples/bond/main.h
+++ /dev/null
@@ -1,10 +0,0 @@
-/* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2010-2015 Intel Corporation
- */
-
-#ifndef _MAIN_H_
-#define _MAIN_H_
-
-int main(int argc, char *argv[]);
-
-#endif /* ifndef _MAIN_H_ */
diff --git a/examples/bond/meson.build b/examples/bond/meson.build
index ed22b7887d..bfbec04ecd 100644
--- a/examples/bond/meson.build
+++ b/examples/bond/meson.build
@@ -11,3 +11,11 @@ allow_experimental_apis = true
 sources = files(
         'main.c',
 )
+
+cmd_h = custom_target('commands_hdr',
+	output: 'commands.h',
+	input: files('commands.list'),
+	capture: true,
+	command: [cmdline_gen_cmd, '--context-name=main_ctx', '@INPUT@']
+)
+sources += cmd_h
-- 
2.39.2

[PATCH v3 0/5] document and simplify use of cmdline

[Bruce Richardson]

The DPDK commandline library is widely used by apps and examples within
DPDK, but it is not documented in our programmers guide and it requires
a lot of boilerplate code definitions in order to used. We can improve
this situation by creating a simple python script to automatically
generate the boilerplate from a list of commands.

This patchset contains a new documentation chapter on cmdline library,
going through step-by-step how to add commands and create the necessary
token lists and parse contexts.

Following that initial doc patch, the set then contains a
boilerplate-generating script, as well as a set of three patches showing
its use, by converting three examples to use the script instead of
having the hard-coded boilerplate. Once the script is used, adding a new
command becomes as simple as adding the desired command to the .list
file, and then writing the required function which will be called for
that command. No other boilerplate coding is necessary.

Script obviously does not cover the full range of capabilities of the
commandline lib, but does cover the most used parts. The code-saving to
each of the examples by auto-generating the boilerplate is significant,
and probably more examples with commandlines can be converted over in
future.

The "cmdline" example itself, is not converted over, as it should
probably remain as a simple example of direct library use without the
script.

V3:
* Added lots of documentation
* Added support for help text for each command
* Cleaned up script a little so it passes pycodestyle and most flake8
  checks, when line-length is set to max 100.
* Removed RFC tag, as I consider this patchset stable enough for
  consideration in a release.

V2-RFC:
* Add support for IP addresses in commands
* Move to buildtools directory and make installable
* Convert 3 examples to use script, and eliminate their boilerplate

Bruce Richardson (5):
  doc/prog_guide: new chapter on cmdline library
  buildtools: script to generate cmdline boilerplate
  examples/simple_mp: auto-generate cmdline boilerplate
  examples/hotplug_mp: auto-generate cmdline boilerplate
  examples/bond: auto-generate cmdline boilerplate

 app/test/commands.c                           |   2 +
 buildtools/dpdk-cmdline-gen.py                | 167 +++++++
 buildtools/meson.build                        |   7 +
 doc/guides/prog_guide/cmdline.rst             | 466 ++++++++++++++++++
 doc/guides/prog_guide/index.rst               |   1 +
 examples/bond/Makefile                        |  12 +-
 examples/bond/commands.list                   |   6 +
 examples/bond/main.c                          | 161 +-----
 examples/bond/main.h                          |  10 -
 examples/bond/meson.build                     |   8 +
 examples/multi_process/hotplug_mp/Makefile    |  12 +-
 examples/multi_process/hotplug_mp/commands.c  | 147 +-----
 examples/multi_process/hotplug_mp/commands.h  |  10 -
 .../multi_process/hotplug_mp/commands.list    |   5 +
 examples/multi_process/hotplug_mp/meson.build |   9 +
 examples/multi_process/simple_mp/Makefile     |  12 +-
 examples/multi_process/simple_mp/meson.build  |   9 +
 .../multi_process/simple_mp/mp_commands.c     | 106 +---
 .../multi_process/simple_mp/mp_commands.h     |  14 -
 .../multi_process/simple_mp/mp_commands.list  |   3 +
 20 files changed, 745 insertions(+), 422 deletions(-)
 create mode 100755 buildtools/dpdk-cmdline-gen.py
 create mode 100644 doc/guides/prog_guide/cmdline.rst
 create mode 100644 examples/bond/commands.list
 delete mode 100644 examples/bond/main.h
 delete mode 100644 examples/multi_process/hotplug_mp/commands.h
 create mode 100644 examples/multi_process/hotplug_mp/commands.list
 delete mode 100644 examples/multi_process/simple_mp/mp_commands.h
 create mode 100644 examples/multi_process/simple_mp/mp_commands.list

--
2.39.2

[PATCH v3 1/5] doc/prog_guide: new chapter on cmdline library

[Bruce Richardson]

The cmdline library was not documented in our programmers guide, so add
a new chapter on it. This chapter covers step-by-step how to use the
library, rather than focusing on the library internals. This complements
the existing cmdline example app document, providing more details on the
process of using the library.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 app/test/commands.c               |   2 +
 doc/guides/prog_guide/cmdline.rst | 337 ++++++++++++++++++++++++++++++
 doc/guides/prog_guide/index.rst   |   1 +
 3 files changed, 340 insertions(+)
 create mode 100644 doc/guides/prog_guide/cmdline.rst

diff --git a/app/test/commands.c b/app/test/commands.c
index 31259e5c21..497d8e9952 100644
--- a/app/test/commands.c
+++ b/app/test/commands.c
@@ -108,6 +108,7 @@ dump_struct_sizes(void)
 #undef DUMP_SIZE
 }
 
+/* Add the dump_* tests cases 8< */
 static void cmd_dump_parsed(void *parsed_result,
 			    __rte_unused struct cmdline *cl,
 			    __rte_unused void *data)
@@ -155,6 +156,7 @@ cmdline_parse_inst_t cmd_dump = {
 		NULL,
 	},
 };
+/* >8 End of add the dump_* tests cases */
 
 /****************/
 
diff --git a/doc/guides/prog_guide/cmdline.rst b/doc/guides/prog_guide/cmdline.rst
new file mode 100644
index 0000000000..40f49a30cc
--- /dev/null
+++ b/doc/guides/prog_guide/cmdline.rst
@@ -0,0 +1,337 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2023 Intel Corporation.
+
+Command-line Library
+====================
+
+Since its earliest versions, DPDK has included a command-line library -
+primarily for internal use by, for example, ``dpdk-testpmd`` and the ``dpdk-test`` binaries,
+but the library is also exported on install and can be used by any end application.
+This chapter covers the basics of the command-line library and how to use it in an application.
+
+Library Features
+----------------
+
+The DPDK command-line library supports the following features:
+
+* Tab-completion available for interactive terminal sessions
+
+* Ability to read and process commands taken from an input file, e.g. startup script
+
+* Parameterized commands able to take multiple parameters with different datatypes:
+
+   * Strings
+   * Signed/unsigned 16/32/64-bit integers
+   * IP Addresses
+   * Ethernet Addresses
+
+* Ability to multiplex multiple commands to a single callback function
+
+Adding Command-line to an Application
+-------------------------------------
+
+Adding a command-line instance to an application involves a number of coding steps.
+
+1. Define the result structure for the command, specifying the command parameters
+
+2. Provide an initializer for each field in the result
+
+3. Define the callback function for the command
+
+4. Provide a parse result structure instance for the command, linking the callback to the command
+
+5. Add the parse result structure to a command-line context
+
+6. Within your main application code, create a new command-line instance passing in the context.
+
+The next few subsections will cover each of these steps in more detail,
+working through an example to add two commands to a command-line instance.
+Those two commands will be:
+
+1. ``quit`` - as the name suggests, to close the application
+
+2. ``show port stats <n>`` - to display on-screen the statistics for a given ethernet port
+
+.. note::
+
+   For further examples of use of the command-line, see
+   :doc:`cmdline example application <../sample_app_ug/cmd_line>`
+
+Defining Command Result Structure
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The first structure to be defined is the structure which will be created on successful parse of a command.
+This structure contains one member field for each token, or word, in the command.
+The simplest case is for a one-word command, like ``quit``.
+For this, we only need to define a structure with a single string parameter to contain that word.
+
+.. code-block:: c
+
+   struct cmd_quit_result {
+              cmdline_fixed_string_t quit;
+   };
+
+For readability, the name of the struct member should match that of the token in the command.
+
+For our second command, we need a structure with four member fields in it,
+as there are four words/tokens in our command.
+The first three are strings, and the final one is a 16-bit numeric value.
+The resulting struct looks like:
+
+.. code-block:: c
+
+   struct cmd_show_port_stats_result {
+      cmdline_fixed_string_t show;
+      cmdline_fixed_string_t port;
+      cmdline_fixed_string_t stats;
+      uint16_t n;
+   };
+
+As before, we choose names to match the tokens in the command.
+Since our numeric parameter is a 16-bit value, we use ``uint16_t`` type for it.
+Any of the standard sized integer types can be used as parameters, depending on the desired result.
+
+Beyond the standard integer types,
+the library also allows variable parameters to be of a number of other types,
+as called out in the feature list above.
+
+* For variable string parameters,
+  the type should be ``cmdline_fixed_string_t`` - the same as for fixed tokens,
+  but these will be initialized differently (as described below).
+
+* For ethernet addresses use type ``struct rte_ether_addr``
+
+* For IP addresses use type ``cmdline_ipaddr_t``
+
+Providing Field Initializers
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Each field of our result structure needs an initializer.
+For fixed string tokens, like "quit", "show" and "port", the initializer will be the string itself.
+
+.. code-block:: c
+
+   static cmdline_parse_token_string_t cmd_quit_quit_tok =
+      TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit");
+
+The convention for naming used here is to include the base name of the overall result structure -
+``cmd_quit`` in this case,
+as well as the name of the field within that structure - ``quit`` in this case, followed by ``_tok``.
+(This is why there is a double ``quit`` in the name above).
+
+This naming convention is seen in our second example,
+which also demonstrates how to define a numeric initializer.
+
+
+.. code-block:: c
+
+   static cmdline_parse_token_string_t cmd_show_port_stats_show_tok =
+      TOKEN_STRING_INITIALIZER(struct cmd_show_port_stats_result, show, "show");
+   static cmdline_parse_token_string_t cmd_show_port_stats_port_tok =
+      TOKEN_STRING_INITIALIZER(struct cmd_show_port_stats_result, port, "port");
+   static cmdline_parse_token_string_t cmd_show_port_stats_stats_tok =
+      TOKEN_STRING_INITIALIZER(struct cmd_show_port_stats_result, stats, "stats");
+   static cmdline_parse_token_num_t cmd_show_port_stats_n_tok =
+      TOKEN_NUM_INITIALIZER(struct cmd_show_port_stats_result, n, RTE_UINT16);
+
+For variable string tokens, the same ``TOKEN_STRING_INITIALIZER`` macro should be used.
+However, the final parameter should be ``NULL`` rather than a hard-coded token string.
+
+For numeric parameters, the final parameter to the ``TOKEN_NUM_INITIALIZER`` macro should be the
+cmdline type matching the variable type defined in the result structure,
+e.g. RTE_UINT8, RTE_UINT32, etc.
+
+For IP addresses, the macro ``TOKEN_IPADDR_INITIALIZER`` should be used.
+
+For ethernet addresses, the macro ``TOKEN_ETHERADDR_INITIALIZER`` should be used.
+
+Defining Callback Function
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+For each command, we need to define a function to be called once the command has been recognised.
+The callback function should have type:
+
+.. code:: c
+
+   void (*f)(void *, struct cmdline *, void *)
+
+where the first parameter is a pointer to the result structure defined above,
+the second parameter is the command-line instance,
+and the final parameter is a user-defined pointer provided when we associate the callback with the command.
+Most callback functions only use the first parameter, or none at all,
+but the additional two parameters provide some extra flexibility,
+to allow the callback to work with non-global state in your application.
+
+For our two example commands, the relevant callback functions would look very similar in definition.
+However, within the function body,
+we assume that the user would need to reference the result structure to extract the port number in
+the second case.
+
+.. code:: c
+
+   void
+   cmd_quit_parsed(void *parsed_result, struct cmdline *cl, void *data)
+   {
+      quit = 1;
+   }
+   void
+   cmd_show_port_stats_parsed(void *parsed_result, struct cmdline *cl, void *data)
+   {
+      struct cmd_show_port_stats_result *res = parsed_result;
+      uint16_t port_id = res->n;
+      ...
+   }
+
+
+Associating Callback and Command
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The ``cmdline_parse_inst_t`` type defines a "parse instance",
+i.e. a sequence of tokens to be matched and then an associated function to be called.
+Also included in the instance type are a field for help text for the command,
+and any additional user-defined parameter to be passed to the callback functions referenced above.
+For example, for our simple "quit" command:
+
+.. code-block:: c
+
+   static cmdline_parse_inst_t cmd_quit = {
+       .f = cmd_quit_parsed,
+       .data = NULL,
+       .help_str = "Close the application",
+       .tokens = {
+           (void *)&cmd_quit_quit_tok,
+           NULL
+       }
+   };
+
+In this case, we firstly identify the callback function to be called,
+then set the user-defined parameter to NULL,
+provide a help message to be given, on request, to the user explaining the command,
+before finally listing out the single token to be matched for this command instance.
+
+For our second, port stats, example,
+as well as making things a little more complicated by having multiple tokens to be matched,
+we can also demonstrate passing in a parameter to the function.
+Let us suppose that our application does not always use all the ports available to it,
+but instead only uses a subset of the ports, stored in an array called ``active_ports``.
+Our stats command, therefore, should only display stats for the currently in-use ports,
+so we pass this ``active_ports`` array.
+(For simplicity of illustration, we shall assume that the array uses a terminating marker,
+e.g. -1 for the end of the port list, so we don't need to pass in a length parameter too.)
+
+.. code-block:: c
+
+   extern int16_t active_ports[];
+   ...
+   static cmdline_parse_inst_t cmd_show_port_stats = {
+       .f = cmd_show_port_stats_parsed,
+       .data = active_ports,
+       .help_str = "Show statistics for active network ports",
+       .tokens = {
+           (void *)&cmd_show_port_stats_show_tok,
+           (void *)&cmd_show_port_stats_port_tok,
+           (void *)&cmd_show_port_stats_stats_tok,
+           (void *)&cmd_show_port_stats_n_tok,
+           NULL
+       }
+   };
+
+
+Adding Command to Command-line Context
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Now that we have configured each individual command and callback,
+we need to merge these into a single array of command-line "contexts".
+This context array will be used to create the actual command-line instance in the application.
+Thankfully, each context entry is the same as each parse instance,
+so our array is defined by simply listing out the previously defined command parse instances.
+
+.. code-block:: c
+
+   static cmdline_parse_ctx_t ctx[] = {
+       &cmd_quit,
+       &cmd_show_port_stats,
+       NULL
+   };
+
+The context list must be terminated by a NULL entry.
+
+Creating a Command-line Instance
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Once we have our ``ctx`` variable defined,
+we now just need to call the API to create the new command-line instance in our application.
+The basic API is ``cmdline_new`` which will create an interactive command-line with all commands available.
+However, if additional features for interactive use - such as tab-completion -
+are desired, it is recommended that ``cmdline_new_stdin`` be used instead.
+
+A pattern that can be used in applications is to use ``cmdline_new`` for processing any startup commands,
+either from file or from the environment (as is done in the "dpdk-test" application),
+and then using ``cmdline_stdin_new`` thereafter to handle the interactive part.
+For example, to handle a startup file and then provide an interactive prompt:
+
+.. code-block:: c
+
+   struct cmdline *cl;
+   int fd = open(startup_file, O_RDONLY);
+
+   if (fd >= 0) {
+       cl = cmdline_new(ctx, "", fd, STDOUT_FILENO);
+       if (cl == NULL) {
+           /* error handling */
+       }
+       cmdline_interact(cl);
+       cmdline_quit(cl);
+       close(fd);
+   }
+
+   cl = cmdline_stdin_new(ctx, "Proxy>> ");
+   if (cl == NULL) {
+       /* error handling */
+   }
+   cmdline_interact(cl);
+   cmdline_stdin_exit(cl);
+
+
+Multiplexing Multiple Commands to a Single Function
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To reduce the amount of boiler-plate code needed when creating a command-line for an application,
+it is possible to merge a number of commands together to have them call a separate function.
+This can be done in a number of different ways:
+
+* A callback function can be used as the target for a number of different commands.
+  Which command was used for entry to the function can be determined by examining the first parameter,
+  ``parsed_result`` in our examples above.
+
+* For simple string commands, multiple options can be concatenated using the "#" character.
+  For example: ``exit#quit``, specified as a token initializer,
+  will match either on the string "exit" or the string "quit".
+
+As a concrete example,
+these two techniques are used in the DPDK unit test application ``dpdk-test``,
+where a single command ``cmdline_parse_t`` instance is used for all the "dump_<item>" test cases.
+
+.. literalinclude:: ../../../app/test/commands.c
+    :language: c
+    :start-after: Add the dump_* tests cases 8<
+    :end-before: >8 End of add the dump_* tests cases
+
+
+Examples of Command-line Use in DPDK
+------------------------------------
+
+To help the user follow the steps provided above,
+the following DPDK files can be consulted for examples of command-line use.
+
+.. note::
+
+   This is not an exhaustive list of examples of command-line use in DPDK.
+   It is simply a list of a few files that may be of use to the application developer.
+   Some of these referenced files contain more complex examples of use that others.
+
+* ``commands.c/.h`` in ``examples/cmdline``
+
+* ``mp_commands.c/.h`` in ``examples/multi_process/simple_mp``
+
+* ``commands.c`` in ``app/test``
diff --git a/doc/guides/prog_guide/index.rst b/doc/guides/prog_guide/index.rst
index 52a6d9e7aa..b792afc364 100644
--- a/doc/guides/prog_guide/index.rst
+++ b/doc/guides/prog_guide/index.rst
@@ -13,6 +13,7 @@ Programmer's Guide
     source_org
     env_abstraction_layer
     log_lib
+    cmdline
     service_cores
     trace_lib
     rcu_lib
-- 
2.39.2

[PATCH v3 2/5] buildtools: script to generate cmdline boilerplate

[Bruce Richardson]

Provide a "dpdk-cmdline-gen.py" script for application developers to
quickly generate the boilerplate code necessary for using the cmdline
library.

Example of use:
The script takes an input file with a list of commands the user wants in
the app, where the parameter variables are tagged with the type.
For example:

	$ cat commands.list
	list
	add <UINT16>x <UINT16>y
	echo <STRING>message
	add socket <STRING>path
	quit

When run through the script as "./dpdk-cmdline-gen.py commands.list",
the output will be the contents of a header file with all the
boilerplate necessary for a commandline instance with those commands.

If the flag --stubs is passed, an output header filename must also be
passed, in which case both a header file with the definitions and a C
file with function stubs in it is written to disk. The separation is so
that the header file can be rewritten at any future point to add more
commands, while the C file can be kept as-is and extended by the user
with any additional functions needed.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 buildtools/dpdk-cmdline-gen.py    | 167 ++++++++++++++++++++++++++++++
 buildtools/meson.build            |   7 ++
 doc/guides/prog_guide/cmdline.rst | 131 ++++++++++++++++++++++-
 3 files changed, 304 insertions(+), 1 deletion(-)
 create mode 100755 buildtools/dpdk-cmdline-gen.py

diff --git a/buildtools/dpdk-cmdline-gen.py b/buildtools/dpdk-cmdline-gen.py
new file mode 100755
index 0000000000..3b41fb0493
--- /dev/null
+++ b/buildtools/dpdk-cmdline-gen.py
@@ -0,0 +1,167 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2023 Intel Corporation
+#
+"""Script to automatically generate boilerplate for using DPDK cmdline library."""
+
+import argparse
+import sys
+
+PARSE_FN_PARAMS = 'void *parsed_result, struct cmdline *cl, void *data'
+PARSE_FN_BODY = """
+    /* TODO: command action */
+    RTE_SET_USED(parsed_result);
+    RTE_SET_USED(cl);
+    RTE_SET_USED(data);
+"""
+
+
+def process_command(tokens, cfile, comment):
+    """Generate the structures and definitions for a single command."""
+    name = []
+
+    if tokens[0].startswith('<'):
+        print('Error: each command must start with at least one literal string', file=sys.stderr)
+        sys.exit(1)
+    for t in tokens:
+        if t.startswith('<'):
+            break
+        name.append(t)
+    name = '_'.join(name)
+
+    result_struct = []
+    initializers = []
+    token_list = []
+    for t in tokens:
+        if t.startswith('<'):
+            t_type, t_name = t[1:].split('>')
+            t_val = 'NULL'
+        else:
+            t_type = 'STRING'
+            t_name = t
+            t_val = f'"{t}"'
+
+        if t_type == 'STRING':
+            result_struct.append(f'\tcmdline_fixed_string_t {t_name};')
+            initializers.append(
+                    f'static cmdline_parse_token_string_t cmd_{name}_{t_name}_tok =\n' +
+                    f'\tTOKEN_STRING_INITIALIZER(struct cmd_{name}_result, {t_name}, {t_val});')
+        elif t_type in ['UINT8', 'UINT16', 'UINT32', 'UINT64', 'INT8', 'INT16', 'INT32', 'INT64']:
+            result_struct.append(f'\t{t_type.lower()}_t {t_name};')
+            initializers.append(
+                    f'static cmdline_parse_token_num_t cmd_{name}_{t_name}_tok =\n' +
+                    f'\tTOKEN_NUM_INITIALIZER(struct cmd_{name}_result, {t_name}, RTE_{t_type});')
+        elif t_type in ['IP', 'IP_ADDR', 'IPADDR']:
+            result_struct.append(f'\tcmdline_ipaddr_t {t_name};')
+            initializers.append(
+                    f'cmdline_parse_token_ipaddr_t cmd_{name}_{t_name}_tok =\n' +
+                    f'\tTOKEN_IPV4_INITIALIZER(struct cmd_{name}_result, {t_name});')
+        else:
+            print(f'Error: unknown token-type {t}', file=sys.stderr)
+            sys.exit(1)
+        token_list.append(f'cmd_{name}_{t_name}_tok')
+
+    print(f'/* Auto-generated handling for command "{" ".join(tokens)}" */')
+    # output function prototype
+    func_sig = f'void\ncmd_{name}_parsed({PARSE_FN_PARAMS})'
+    print(f'extern {func_sig};\n')
+    # output function template if C file being written
+    if (cfile):
+        print(f'{func_sig}\n{{{PARSE_FN_BODY}}}\n', file=cfile)
+    # output result data structure
+    print(
+            f'struct cmd_{name}_result {{\n' +
+            '\n'.join(result_struct) +
+            '\n};\n')
+    # output the initializer tokens
+    print('\n'.join(initializers) + '\n')
+    # output the instance structure
+    print(
+            f'static cmdline_parse_inst_t cmd_{name} = {{\n' +
+            f'\t.f = cmd_{name}_parsed,\n' +
+            '\t.data = NULL,\n' +
+            f'\t.help_str = "{comment}",\n' +
+            '\t.tokens = {')
+    for t in token_list:
+        print(f'\t\t(void *)&{t},')
+    print('\t\tNULL\n' + '\t}\n' + '};\n')
+
+    # return the instance structure name
+    return f'cmd_{name}'
+
+
+def process_commands(infile, hfile, cfile, ctxname):
+    """Generate boilerplate output for a list of commands from infile."""
+    instances = []
+
+    # redirect stdout to output the header, to save passing file= each print
+    old_sys_stdout = sys.stdout
+    sys.stdout = hfile
+
+    print(f'/* File autogenerated by {sys.argv[0]} */')
+    print('#ifndef GENERATED_COMMANDS_H')
+    print('#define GENERATED_COMMANDS_H')
+    print('#include <rte_common.h>')
+    print('#include <cmdline.h>')
+    print('#include <cmdline_parse_string.h>')
+    print('#include <cmdline_parse_num.h>')
+    print('#include <cmdline_parse_ipaddr.h>')
+    print('')
+
+    for line in infile.readlines():
+        if line.lstrip().startswith('#'):
+            continue
+        if '#' not in line:
+            line = line + '#'  # ensure split always works, even if no help text
+        tokens, comment = line.split('#', 1)
+        instances.append(process_command(tokens.strip().split(), cfile, comment.strip()))
+
+    print(f'static __rte_used cmdline_parse_ctx_t {ctxname}[] = {{')
+    for inst in instances:
+        print(f'\t&{inst},')
+    print('\tNULL')
+    print('};\n')
+    print('#endif /* GENERATED_COMMANDS_H */')
+
+    sys.stdout = old_sys_stdout
+
+
+def main():
+    """Application main entry point."""
+    ap = argparse.ArgumentParser()
+    ap.add_argument(
+            '--stubs', action='store_true',
+            help='Produce C file with empty function stubs for each command')
+    ap.add_argument(
+            '--output-file', '-o', default='-',
+            help='Output header filename [default to stdout]')
+    ap.add_argument(
+            '--context-name', default='ctx',
+            help='Name given to the cmdline context variable in the output header [default=ctx]')
+    ap.add_argument(
+            'infile', type=argparse.FileType('r'),
+            help='File with list of commands')
+    args = ap.parse_args()
+
+    if not args.stubs:
+        if args.output_file == '-':
+            process_commands(args.infile, sys.stdout, None, args.context_name)
+        else:
+            with open(args.output_file, 'w') as hfile:
+                process_commands(args.infile, hfile, None, args.context_name)
+    else:
+        if not args.output_file.endswith('.h'):
+            print(
+                    'Error: output filename must end with ".h" extension when creating stubs',
+                    file=sys.stderr)
+            sys.exit(1)
+
+        cfilename = args.output_file[:-2] + '.c'
+        with open(args.output_file, 'w') as hfile:
+            with open(cfilename, 'w') as cfile:
+                print(f'#include "{args.output_file}"\n', file=cfile)
+                process_commands(args.infile, hfile, cfile, args.context_name)
+
+
+if __name__ == '__main__':
+    main()
diff --git a/buildtools/meson.build b/buildtools/meson.build
index 948ac17dd2..72447b60a0 100644
--- a/buildtools/meson.build
+++ b/buildtools/meson.build
@@ -19,6 +19,13 @@ get_cpu_count_cmd = py3 + files('get-cpu-count.py')
 get_numa_count_cmd = py3 + files('get-numa-count.py')
 get_test_suites_cmd = py3 + files('get-test-suites.py')
 has_hugepages_cmd = py3 + files('has-hugepages.py')
+cmdline_gen_cmd = py3 + files('dpdk-cmdline-gen.py')
+
+# install any build tools that end-users might want also
+install_data([
+            'dpdk-cmdline-gen.py',
+        ],
+        install_dir: 'bin')
 
 # select library and object file format
 pmdinfo = py3 + files('gen-pmdinfo-cfile.py') + [meson.current_build_dir()]
diff --git a/doc/guides/prog_guide/cmdline.rst b/doc/guides/prog_guide/cmdline.rst
index 40f49a30cc..0b96b770e2 100644
--- a/doc/guides/prog_guide/cmdline.rst
+++ b/doc/guides/prog_guide/cmdline.rst
@@ -44,7 +44,136 @@ Adding a command-line instance to an application involves a number of coding ste
 
 6. Within your main application code, create a new command-line instance passing in the context.
 
-The next few subsections will cover each of these steps in more detail,
+Many of these steps can be automated using the script ``dpdk-cmdline-gen.py`` installed by DPDK,
+and found in the ``buildtools`` folder in the source tree.
+This section covers adding a command-line using this script to generate the boiler plate,
+while the following section,
+`Worked Example of Adding Command-line to an Application`_ covers the steps to do so manually.
+
+Creating a Command List File
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The ``dpdk-cmdline-gen.py`` script takes as input a list of commands to be used by the application.
+While these can be piped to it via standard input, using a list file is probably best.
+
+The format of the list file must be:
+
+* Comment lines start with '#' as first non-whitespace character
+
+* One command per line
+
+* Variable fields are prefixed by the type-name in angle-brackets, for example:
+
+  * ``<STRING>message``
+
+  * ``<UINT16>port_id``
+
+  * ``<IP>src_ip``
+
+* The help text for a command is given in the form of a comment on the same line as the command
+
+An example list file, with a variety of (unrelated) commands, is shown below::
+
+   # example list file
+   list                     # show all entries
+   add <UINT16>x <UINT16>y  # add x and y
+   echo <STRING>message     # print message to screen
+   add socket <STRING>path  # add unix socket with the given path
+   quit                     # close the application
+
+Running the Generator Script
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To generate the necessary definitions for a command-line, run ``dpdk-cmdline-gen.py`` passing the list file as parameter.
+The script will output the generated C code to standard output,
+the contents of which are in the form of a C header file.
+Optionally, an output filename may be specified via the ``-o/--output-file`` argument.
+
+The generated content includes:
+
+* The result structure definitions for each command
+
+* The token initializers for each structure field
+
+* An "extern" function prototype for the callback for each command
+
+* A parse context for each command, including the per-command comments as help string
+
+* A command-line context array definition, suitable for passing to ``cmdline_new``
+
+If so desired, the script can also output function stubs for the callback functions for each command.
+This behaviour is triggered by passing the ``--stubs`` flag to the script.
+In this case, an output file must be provided with a filename ending in ".h",
+and the callback stubs will be written to an equivalent ".c" file.
+
+.. note::
+
+   The stubs are written to a separate file,
+   to allow continuous use of the script to regenerate the command-line header,
+   without overwriting any code the user has added to the callback functions.
+   This makes it easy to incrementally add new commands to an existing application.
+
+Providing the Function Callbacks
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+As discussed above, the script output is a header file, containing structure definitions,
+but the callback functions themselves obviously have to be provided by the user.
+These callback functions must be provided as non-static functions in a C file,
+and named ``cmd_<cmdname>_parsed``.
+The function prototypes can be seen in the generated output header.
+
+The "cmdname" part of the function name is built up by combining the non-variable initial tokens in the command.
+So, given the commands in our worked example below: ``quit`` and ``show port stats <n>``,
+the callback functions would be:
+
+.. code:: c
+
+   void
+   cmd_quit_parsed(void *parsed_result, struct cmdline *cl, void *data)
+   {
+        ...
+   }
+
+   void
+   cmd_show_port_stats_parsed(void *parsed_result, struct cmdline *cl, void *data)
+   {
+        ...
+   }
+
+These functions must be provided by the developer, but, as stated above,
+stub functions may be generated by the script automatically using the ``--stubs`` parameter.
+
+The same "cmdname" stem is used in the naming of the generated structures too.
+To get at the results structure for each command above,
+the ``parsed_result`` parameter should be cast to ``struct cmd_quit_result``
+or ``struct cmd_show_port_stats_result`` respectively.
+
+Integrating with the Application
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To integrate the script output with the application,
+we must ``#include`` the generated header into our applications C file,
+and then have the command-line created via either ``cmdline_new`` or ``cmdline_stdin_new``.
+The first parameter to the function call should be the context array in the generated header file,
+``ctx`` by default. (Modifiable via script parameter).
+
+The callback functions may be in this same file, or in a separate one -
+they just need to be available to the linker at build-time.
+
+Limitations of the Script Approach
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The script approach works for most commands that a user may wish to add to an application.
+However, it does not support the full range of functions possible with the DPDK command-line library.
+For example,
+it is not possible using the script to multiplex multiple commands into a single callback function.
+To use this functionality, the user should follow the instructions in the next section
+`Worked Example of Adding Command-line to an Application`_ to manually configure a command-line instance.
+
+Worked Example of Adding Command-line to an Application
+-------------------------------------------------------
+
+The next few subsections will cover each of the steps listed in `Adding Command-line to an Application`_ in more detail,
 working through an example to add two commands to a command-line instance.
 Those two commands will be:
 
-- 
2.39.2

Re: [PATCH v3 2/5] buildtools: script to generate cmdline boilerplate

[Robin Jarry]

Bruce Richardson, Oct 11, 2023 at 15:33:
> Provide a "dpdk-cmdline-gen.py" script for application developers to
> quickly generate the boilerplate code necessary for using the cmdline
> library.
>
> Example of use:
> The script takes an input file with a list of commands the user wants in
> the app, where the parameter variables are tagged with the type.
> For example:
>
> 	$ cat commands.list
> 	list
> 	add <UINT16>x <UINT16>y
> 	echo <STRING>message
> 	add socket <STRING>path
> 	quit
>
> When run through the script as "./dpdk-cmdline-gen.py commands.list",
> the output will be the contents of a header file with all the
> boilerplate necessary for a commandline instance with those commands.
>
> If the flag --stubs is passed, an output header filename must also be
> passed, in which case both a header file with the definitions and a C
> file with function stubs in it is written to disk. The separation is so
> that the header file can be rewritten at any future point to add more
> commands, while the C file can be kept as-is and extended by the user
> with any additional functions needed.
>
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> ---

Hi Bruce,

this is a nice addition, I have a few python style remarks below.

In general, I would advise formatting your code with black[1] to avoid 
debates over coding style. It makes all code homogeneous and lets you 
focus on more important things :)

>  buildtools/dpdk-cmdline-gen.py    | 167 ++++++++++++++++++++++++++++++
>  buildtools/meson.build            |   7 ++
>  doc/guides/prog_guide/cmdline.rst | 131 ++++++++++++++++++++++-
>  3 files changed, 304 insertions(+), 1 deletion(-)
>  create mode 100755 buildtools/dpdk-cmdline-gen.py
>
> diff --git a/buildtools/dpdk-cmdline-gen.py b/buildtools/dpdk-cmdline-gen.py
> new file mode 100755
> index 0000000000..3b41fb0493
> --- /dev/null
> +++ b/buildtools/dpdk-cmdline-gen.py
> @@ -0,0 +1,167 @@
> +#!/usr/bin/env python3
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright(c) 2023 Intel Corporation
> +#
> +"""Script to automatically generate boilerplate for using DPDK cmdline library."""

Multi line (or single line) doc strings are usually formatted as 
follows:

"""
Script to automatically generate boilerplate for using DPDK cmdline library.
"""

It makes adding new lines more readable and saves a bit of characters 
per line.

> +
> +import argparse
> +import sys
> +
> +PARSE_FN_PARAMS = 'void *parsed_result, struct cmdline *cl, void *data'
> +PARSE_FN_BODY = """
> +    /* TODO: command action */
> +    RTE_SET_USED(parsed_result);
> +    RTE_SET_USED(cl);
> +    RTE_SET_USED(data);
> +"""
> +
> +
> +def process_command(tokens, cfile, comment):
> +    """Generate the structures and definitions for a single command."""
> +    name = []
> +
> +    if tokens[0].startswith('<'):
> +        print('Error: each command must start with at least one literal string', file=sys.stderr)
> +        sys.exit(1)

It would be better to raise an exception here and handle it in main() 
for error reporting.

> +    for t in tokens:
> +        if t.startswith('<'):
> +            break
> +        name.append(t)
> +    name = '_'.join(name)
> +
> +    result_struct = []
> +    initializers = []
> +    token_list = []
> +    for t in tokens:
> +        if t.startswith('<'):
> +            t_type, t_name = t[1:].split('>')
> +            t_val = 'NULL'
> +        else:
> +            t_type = 'STRING'
> +            t_name = t
> +            t_val = f'"{t}"'
> +
> +        if t_type == 'STRING':
> +            result_struct.append(f'\tcmdline_fixed_string_t {t_name};')
> +            initializers.append(
> +                    f'static cmdline_parse_token_string_t cmd_{name}_{t_name}_tok =\n' +
> +                    f'\tTOKEN_STRING_INITIALIZER(struct cmd_{name}_result, {t_name}, {t_val});')
> +        elif t_type in ['UINT8', 'UINT16', 'UINT32', 'UINT64', 'INT8', 'INT16', 'INT32', 'INT64']:
> +            result_struct.append(f'\t{t_type.lower()}_t {t_name};')
> +            initializers.append(
> +                    f'static cmdline_parse_token_num_t cmd_{name}_{t_name}_tok =\n' +
> +                    f'\tTOKEN_NUM_INITIALIZER(struct cmd_{name}_result, {t_name}, RTE_{t_type});')
> +        elif t_type in ['IP', 'IP_ADDR', 'IPADDR']:
> +            result_struct.append(f'\tcmdline_ipaddr_t {t_name};')
> +            initializers.append(
> +                    f'cmdline_parse_token_ipaddr_t cmd_{name}_{t_name}_tok =\n' +
> +                    f'\tTOKEN_IPV4_INITIALIZER(struct cmd_{name}_result, {t_name});')
> +        else:
> +            print(f'Error: unknown token-type {t}', file=sys.stderr)
> +            sys.exit(1)
> +        token_list.append(f'cmd_{name}_{t_name}_tok')
> +
> +    print(f'/* Auto-generated handling for command "{" ".join(tokens)}" */')
> +    # output function prototype
> +    func_sig = f'void\ncmd_{name}_parsed({PARSE_FN_PARAMS})'
> +    print(f'extern {func_sig};\n')
> +    # output function template if C file being written
> +    if (cfile):
> +        print(f'{func_sig}\n{{{PARSE_FN_BODY}}}\n', file=cfile)
> +    # output result data structure
> +    print(
> +            f'struct cmd_{name}_result {{\n' +
> +            '\n'.join(result_struct) +
> +            '\n};\n')
> +    # output the initializer tokens
> +    print('\n'.join(initializers) + '\n')
> +    # output the instance structure
> +    print(
> +            f'static cmdline_parse_inst_t cmd_{name} = {{\n' +
> +            f'\t.f = cmd_{name}_parsed,\n' +
> +            '\t.data = NULL,\n' +
> +            f'\t.help_str = "{comment}",\n' +
> +            '\t.tokens = {')
> +    for t in token_list:
> +        print(f'\t\t(void *)&{t},')
> +    print('\t\tNULL\n' + '\t}\n' + '};\n')
> +
> +    # return the instance structure name
> +    return f'cmd_{name}'
> +
> +
> +def process_commands(infile, hfile, cfile, ctxname):
> +    """Generate boilerplate output for a list of commands from infile."""
> +    instances = []
> +
> +    # redirect stdout to output the header, to save passing file= each print
> +    old_sys_stdout = sys.stdout
> +    sys.stdout = hfile

Why not use hfile.write()?

I think the main issue here is to use print() in process_commands(). It 
would probably be cleaner to have process_command() return a list of 
lines and print them in this function.

> +
> +    print(f'/* File autogenerated by {sys.argv[0]} */')
> +    print('#ifndef GENERATED_COMMANDS_H')
> +    print('#define GENERATED_COMMANDS_H')
> +    print('#include <rte_common.h>')
> +    print('#include <cmdline.h>')
> +    print('#include <cmdline_parse_string.h>')
> +    print('#include <cmdline_parse_num.h>')
> +    print('#include <cmdline_parse_ipaddr.h>')
> +    print('')

You can use a multi-line f-string here with a single print/write.

    hfile.write(f"""/* File autogenerated by {sys.argv[0]} */
#ifndef GENERATED_COMMANDS_H
#define GENERATED_COMMANDS_H

#include <rte_common.h>
#include <cmdline.h>
#include <cmdline_parse_string.h>
#include <cmdline_parse_num.h>
#include <cmdline_parse_ipaddr.h>

""")

> +
> +    for line in infile.readlines():
> +        if line.lstrip().startswith('#'):
> +            continue
> +        if '#' not in line:
> +            line = line + '#'  # ensure split always works, even if no help text
> +        tokens, comment = line.split('#', 1)
> +        instances.append(process_command(tokens.strip().split(), cfile, comment.strip()))

If process_command returns a name and a list of lines, that could be 
transformed as:

           name, lines = process_command(tokens.strip().split(), cfile, comment.strip())
           instances.append(name)
           hfile.write("\n".join(lines) + "\n")

> +
> +    print(f'static __rte_used cmdline_parse_ctx_t {ctxname}[] = {{')
> +    for inst in instances:
> +        print(f'\t&{inst},')
> +    print('\tNULL')
> +    print('};\n')
> +    print('#endif /* GENERATED_COMMANDS_H */')

also multi line print here:

hfile.write("""\tNULL
};
#endif /* GENERATED_COMMANDS_H */
""")

> +
> +    sys.stdout = old_sys_stdout
> +
> +
> +def main():
> +    """Application main entry point."""
> +    ap = argparse.ArgumentParser()

Nit to have a nice description of the command with --help:

       ap = argparse.ArgumentParser(description=__doc__)

> +    ap.add_argument(
> +            '--stubs', action='store_true',
> +            help='Produce C file with empty function stubs for each command')
> +    ap.add_argument(
> +            '--output-file', '-o', default='-',
> +            help='Output header filename [default to stdout]')
> +    ap.add_argument(
> +            '--context-name', default='ctx',
> +            help='Name given to the cmdline context variable in the output header [default=ctx]')
> +    ap.add_argument(
> +            'infile', type=argparse.FileType('r'),
> +            help='File with list of commands')
> +    args = ap.parse_args()
> +
> +    if not args.stubs:
> +        if args.output_file == '-':
> +            process_commands(args.infile, sys.stdout, None, args.context_name)
> +        else:
> +            with open(args.output_file, 'w') as hfile:
> +                process_commands(args.infile, hfile, None, args.context_name)
> +    else:
> +        if not args.output_file.endswith('.h'):
> +            print(
> +                    'Error: output filename must end with ".h" extension when creating stubs',
> +                    file=sys.stderr)
> +            sys.exit(1)

You can replace print to stderr + exit with:

ap.error("-o/--output-file: must end with .h extension when creating stubs")

> +
> +        cfilename = args.output_file[:-2] + '.c'
> +        with open(args.output_file, 'w') as hfile:
> +            with open(cfilename, 'w') as cfile:
> +                print(f'#include "{args.output_file}"\n', file=cfile)
> +                process_commands(args.infile, hfile, cfile, args.context_name)
> +
> +
> +if __name__ == '__main__':
> +    main()

I'll stop here ;) Thanks!

Re: [PATCH v3 2/5] buildtools: script to generate cmdline boilerplate

[Bruce Richardson]

On Fri, Oct 13, 2023 at 02:23:13PM +0200, Robin Jarry wrote:
> Bruce Richardson, Oct 11, 2023 at 15:33:
> > Provide a "dpdk-cmdline-gen.py" script for application developers to
> > quickly generate the boilerplate code necessary for using the cmdline
> > library.
> > 
> > Example of use:
> > The script takes an input file with a list of commands the user wants in
> > the app, where the parameter variables are tagged with the type.
> > For example:
> > 
> > 	$ cat commands.list
> > 	list
> > 	add <UINT16>x <UINT16>y
> > 	echo <STRING>message
> > 	add socket <STRING>path
> > 	quit
> > 
> > When run through the script as "./dpdk-cmdline-gen.py commands.list",
> > the output will be the contents of a header file with all the
> > boilerplate necessary for a commandline instance with those commands.
> > 
> > If the flag --stubs is passed, an output header filename must also be
> > passed, in which case both a header file with the definitions and a C
> > file with function stubs in it is written to disk. The separation is so
> > that the header file can be rewritten at any future point to add more
> > commands, while the C file can be kept as-is and extended by the user
> > with any additional functions needed.
> > 
> > Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> > ---
> 
> Hi Bruce,
> 
> this is a nice addition, I have a few python style remarks below.
> 
> In general, I would advise formatting your code with black[1] to avoid
> debates over coding style. It makes all code homogeneous and lets you focus
> on more important things :)
> 

Thanks for the feedback. I'll look into black.

> >  buildtools/dpdk-cmdline-gen.py    | 167 ++++++++++++++++++++++++++++++
> >  buildtools/meson.build            |   7 ++
> >  doc/guides/prog_guide/cmdline.rst | 131 ++++++++++++++++++++++-
> >  3 files changed, 304 insertions(+), 1 deletion(-)
> >  create mode 100755 buildtools/dpdk-cmdline-gen.py
> > 
> > diff --git a/buildtools/dpdk-cmdline-gen.py b/buildtools/dpdk-cmdline-gen.py
> > new file mode 100755
> > index 0000000000..3b41fb0493
> > --- /dev/null
> > +++ b/buildtools/dpdk-cmdline-gen.py
> > @@ -0,0 +1,167 @@
> > +#!/usr/bin/env python3
> > +# SPDX-License-Identifier: BSD-3-Clause
> > +# Copyright(c) 2023 Intel Corporation
> > +#
> > +"""Script to automatically generate boilerplate for using DPDK cmdline library."""
> 
> Multi line (or single line) doc strings are usually formatted as follows:
> 
> """
> Script to automatically generate boilerplate for using DPDK cmdline library.
> """
> 
> It makes adding new lines more readable and saves a bit of characters per
> line.
> 

Sure.

> > +
> > +import argparse
> > +import sys
> > +
> > +PARSE_FN_PARAMS = 'void *parsed_result, struct cmdline *cl, void *data'
> > +PARSE_FN_BODY = """
> > +    /* TODO: command action */
> > +    RTE_SET_USED(parsed_result);
> > +    RTE_SET_USED(cl);
> > +    RTE_SET_USED(data);
> > +"""
> > +
> > +
> > +def process_command(tokens, cfile, comment):
> > +    """Generate the structures and definitions for a single command."""
> > +    name = []
> > +
> > +    if tokens[0].startswith('<'):
> > +        print('Error: each command must start with at least one literal string', file=sys.stderr)
> > +        sys.exit(1)
> 
> It would be better to raise an exception here and handle it in main() for
> error reporting.
> 

I'll look into that. I probably should include the offending line/token in
the error report too.

> > +    for t in tokens:
> > +        if t.startswith('<'):
> > +            break
> > +        name.append(t)
> > +    name = '_'.join(name)
> > +
> > +    result_struct = []
> > +    initializers = []
> > +    token_list = []
> > +    for t in tokens:
> > +        if t.startswith('<'):
> > +            t_type, t_name = t[1:].split('>')
> > +            t_val = 'NULL'
> > +        else:
> > +            t_type = 'STRING'
> > +            t_name = t
> > +            t_val = f'"{t}"'
> > +
> > +        if t_type == 'STRING':
> > +            result_struct.append(f'\tcmdline_fixed_string_t {t_name};')
> > +            initializers.append(
> > +                    f'static cmdline_parse_token_string_t cmd_{name}_{t_name}_tok =\n' +
> > +                    f'\tTOKEN_STRING_INITIALIZER(struct cmd_{name}_result, {t_name}, {t_val});')
> > +        elif t_type in ['UINT8', 'UINT16', 'UINT32', 'UINT64', 'INT8', 'INT16', 'INT32', 'INT64']:
> > +            result_struct.append(f'\t{t_type.lower()}_t {t_name};')
> > +            initializers.append(
> > +                    f'static cmdline_parse_token_num_t cmd_{name}_{t_name}_tok =\n' +
> > +                    f'\tTOKEN_NUM_INITIALIZER(struct cmd_{name}_result, {t_name}, RTE_{t_type});')
> > +        elif t_type in ['IP', 'IP_ADDR', 'IPADDR']:
> > +            result_struct.append(f'\tcmdline_ipaddr_t {t_name};')
> > +            initializers.append(
> > +                    f'cmdline_parse_token_ipaddr_t cmd_{name}_{t_name}_tok =\n' +
> > +                    f'\tTOKEN_IPV4_INITIALIZER(struct cmd_{name}_result, {t_name});')
> > +        else:
> > +            print(f'Error: unknown token-type {t}', file=sys.stderr)
> > +            sys.exit(1)
> > +        token_list.append(f'cmd_{name}_{t_name}_tok')
> > +
> > +    print(f'/* Auto-generated handling for command "{" ".join(tokens)}" */')
> > +    # output function prototype
> > +    func_sig = f'void\ncmd_{name}_parsed({PARSE_FN_PARAMS})'
> > +    print(f'extern {func_sig};\n')
> > +    # output function template if C file being written
> > +    if (cfile):
> > +        print(f'{func_sig}\n{{{PARSE_FN_BODY}}}\n', file=cfile)
> > +    # output result data structure
> > +    print(
> > +            f'struct cmd_{name}_result {{\n' +
> > +            '\n'.join(result_struct) +
> > +            '\n};\n')
> > +    # output the initializer tokens
> > +    print('\n'.join(initializers) + '\n')
> > +    # output the instance structure
> > +    print(
> > +            f'static cmdline_parse_inst_t cmd_{name} = {{\n' +
> > +            f'\t.f = cmd_{name}_parsed,\n' +
> > +            '\t.data = NULL,\n' +
> > +            f'\t.help_str = "{comment}",\n' +
> > +            '\t.tokens = {')
> > +    for t in token_list:
> > +        print(f'\t\t(void *)&{t},')
> > +    print('\t\tNULL\n' + '\t}\n' + '};\n')
> > +
> > +    # return the instance structure name
> > +    return f'cmd_{name}'
> > +
> > +
> > +def process_commands(infile, hfile, cfile, ctxname):
> > +    """Generate boilerplate output for a list of commands from infile."""
> > +    instances = []
> > +
> > +    # redirect stdout to output the header, to save passing file= each print
> > +    old_sys_stdout = sys.stdout
> > +    sys.stdout = hfile
> 
> Why not use hfile.write()?
> 
> I think the main issue here is to use print() in process_commands(). It
> would probably be cleaner to have process_command() return a list of lines
> and print them in this function.
> 

Good idea, let me look into that. Just using print was shorter than
hfile.write, and I felt it would save me some line-breaking.

> > +
> > +    print(f'/* File autogenerated by {sys.argv[0]} */')
> > +    print('#ifndef GENERATED_COMMANDS_H')
> > +    print('#define GENERATED_COMMANDS_H')
> > +    print('#include <rte_common.h>')
> > +    print('#include <cmdline.h>')
> > +    print('#include <cmdline_parse_string.h>')
> > +    print('#include <cmdline_parse_num.h>')
> > +    print('#include <cmdline_parse_ipaddr.h>')
> > +    print('')
> 
> You can use a multi-line f-string here with a single print/write.
> 
>    hfile.write(f"""/* File autogenerated by {sys.argv[0]} */
> #ifndef GENERATED_COMMANDS_H
> #define GENERATED_COMMANDS_H
> 
> #include <rte_common.h>
> #include <cmdline.h>
> #include <cmdline_parse_string.h>
> #include <cmdline_parse_num.h>
> #include <cmdline_parse_ipaddr.h>
> 
> """)
> 

Yes, I was aware of that.  However, I deliberately didn't use it, because
the indentation looks completely wrong with it (at least to my eyes). I
really don't like having each line start at column zero in the middle of a
block of code.

> > +
> > +    for line in infile.readlines():
> > +        if line.lstrip().startswith('#'):
> > +            continue
> > +        if '#' not in line:
> > +            line = line + '#'  # ensure split always works, even if no help text
> > +        tokens, comment = line.split('#', 1)
> > +        instances.append(process_command(tokens.strip().split(), cfile, comment.strip()))
> 
> If process_command returns a name and a list of lines, that could be
> transformed as:
> 
>           name, lines = process_command(tokens.strip().split(), cfile, comment.strip())
>           instances.append(name)
>           hfile.write("\n".join(lines) + "\n")
>

Will investigate
 
> > +
> > +    print(f'static __rte_used cmdline_parse_ctx_t {ctxname}[] = {{')
> > +    for inst in instances:
> > +        print(f'\t&{inst},')
> > +    print('\tNULL')
> > +    print('};\n')
> > +    print('#endif /* GENERATED_COMMANDS_H */')
> 
> also multi line print here:
> 
> hfile.write("""\tNULL
> };
> #endif /* GENERATED_COMMANDS_H */
> """)
> 

As above, I didn't like the non-indented nature of it.

> > +
> > +    sys.stdout = old_sys_stdout
> > +
> > +
> > +def main():
> > +    """Application main entry point."""
> > +    ap = argparse.ArgumentParser()
> 
> Nit to have a nice description of the command with --help:
> 
>       ap = argparse.ArgumentParser(description=__doc__)
> 
> > +    ap.add_argument(
> > +            '--stubs', action='store_true',
> > +            help='Produce C file with empty function stubs for each command')
> > +    ap.add_argument(
> > +            '--output-file', '-o', default='-',
> > +            help='Output header filename [default to stdout]')
> > +    ap.add_argument(
> > +            '--context-name', default='ctx',
> > +            help='Name given to the cmdline context variable in the output header [default=ctx]')
> > +    ap.add_argument(
> > +            'infile', type=argparse.FileType('r'),
> > +            help='File with list of commands')
> > +    args = ap.parse_args()
> > +
> > +    if not args.stubs:
> > +        if args.output_file == '-':
> > +            process_commands(args.infile, sys.stdout, None, args.context_name)
> > +        else:
> > +            with open(args.output_file, 'w') as hfile:
> > +                process_commands(args.infile, hfile, None, args.context_name)
> > +    else:
> > +        if not args.output_file.endswith('.h'):
> > +            print(
> > +                    'Error: output filename must end with ".h" extension when creating stubs',
> > +                    file=sys.stderr)
> > +            sys.exit(1)
> 
> You can replace print to stderr + exit with:
> 
> ap.error("-o/--output-file: must end with .h extension when creating stubs")
> 

Ack.

> > +
> > +        cfilename = args.output_file[:-2] + '.c'
> > +        with open(args.output_file, 'w') as hfile:
> > +            with open(cfilename, 'w') as cfile:
> > +                print(f'#include "{args.output_file}"\n', file=cfile)
> > +                process_commands(args.infile, hfile, cfile, args.context_name)
> > +
> > +
> > +if __name__ == '__main__':
> > +    main()
> 
> I'll stop here ;) Thanks!
> 

[PATCH v3 3/5] examples/simple_mp: auto-generate cmdline boilerplate

[Bruce Richardson]

Use the dpdk-cmdline-gen script to autogenerate all the boilerplate
structs and defines for the commandline part of the app.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 examples/multi_process/simple_mp/Makefile     |  12 +-
 examples/multi_process/simple_mp/meson.build  |   9 ++
 .../multi_process/simple_mp/mp_commands.c     | 106 ++----------------
 .../multi_process/simple_mp/mp_commands.h     |  14 ---
 .../multi_process/simple_mp/mp_commands.list  |   3 +
 5 files changed, 30 insertions(+), 114 deletions(-)
 delete mode 100644 examples/multi_process/simple_mp/mp_commands.h
 create mode 100644 examples/multi_process/simple_mp/mp_commands.list

diff --git a/examples/multi_process/simple_mp/Makefile b/examples/multi_process/simple_mp/Makefile
index 1d0a260e64..890b6b7e62 100644
--- a/examples/multi_process/simple_mp/Makefile
+++ b/examples/multi_process/simple_mp/Makefile
@@ -6,6 +6,7 @@ APP = simple_mp
 
 # all source are stored in SRCS-y
 SRCS-y := main.c mp_commands.c
+SRC-DEPS := build/mp_commands.h
 
 PKGCONF ?= pkg-config
 
@@ -22,10 +23,13 @@ static: build/$(APP)-static
 	ln -sf $(APP)-static build/$(APP)
 
 PC_FILE := $(shell $(PKGCONF) --path libdpdk 2>/dev/null)
-CFLAGS += -O3 $(shell $(PKGCONF) --cflags libdpdk)
+CFLAGS += -O3 $(shell $(PKGCONF) --cflags libdpdk) -I build/
 LDFLAGS_SHARED = $(shell $(PKGCONF) --libs libdpdk)
 LDFLAGS_STATIC = $(shell $(PKGCONF) --static --libs libdpdk)
 
+build/mp_commands.h: mp_commands.list Makefile
+	dpdk-cmdline-gen.py -o $@ --context-name=simple_mp_ctx $<
+
 ifeq ($(MAKECMDGOALS),static)
 # check for broken pkg-config
 ifeq ($(shell echo $(LDFLAGS_STATIC) | grep 'whole-archive.*l:lib.*no-whole-archive'),)
@@ -36,10 +40,10 @@ endif
 
 CFLAGS += -DALLOW_EXPERIMENTAL_API
 
-build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
+build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build $(SRC-DEPS)
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
 
-build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build
+build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build $(SRC-DEPS)
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_STATIC)
 
 build:
@@ -47,5 +51,5 @@ build:
 
 .PHONY: clean
 clean:
-	rm -f build/$(APP) build/$(APP)-static build/$(APP)-shared
+	rm -f build/$(APP) build/$(APP)-static build/$(APP)-shared build/mp_commands.h
 	test -d build && rmdir -p build || true
diff --git a/examples/multi_process/simple_mp/meson.build b/examples/multi_process/simple_mp/meson.build
index 359af4384d..e99b7a3f6f 100644
--- a/examples/multi_process/simple_mp/meson.build
+++ b/examples/multi_process/simple_mp/meson.build
@@ -7,7 +7,16 @@
 # DPDK instance, use 'make'
 
 allow_experimental_apis = true
+
+cmd_h = custom_target('commands_hdr',
+	output: 'mp_commands.h',
+	input: files('mp_commands.list'),
+	capture: true,
+	command: [cmdline_gen_cmd, '--context-name=simple_mp_ctx', '@INPUT@']
+)
+
 sources = files(
         'mp_commands.c',
         'main.c',
 )
+sources += cmd_h
diff --git a/examples/multi_process/simple_mp/mp_commands.c b/examples/multi_process/simple_mp/mp_commands.c
index a5f91b00be..df9fa94208 100644
--- a/examples/multi_process/simple_mp/mp_commands.c
+++ b/examples/multi_process/simple_mp/mp_commands.c
@@ -1,44 +1,18 @@
 /* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2010-2014 Intel Corporation
+ * Copyright(c) 2010-2023 Intel Corporation
  */
-#include <stdint.h>
-#include <string.h>
-#include <stdlib.h>
-#include <stdarg.h>
-#include <inttypes.h>
-#include <stdio.h>
-#include <termios.h>
-#include <errno.h>
-#include <sys/queue.h>
-
-#include <rte_common.h>
-#include <rte_memory.h>
-#include <rte_eal.h>
-#include <rte_branch_prediction.h>
-#include <rte_launch.h>
-#include <rte_log.h>
-#include <rte_per_lcore.h>
-#include <rte_lcore.h>
 #include <rte_ring.h>
-#include <rte_debug.h>
 #include <rte_mempool.h>
 #include <rte_string_fns.h>
 
-#include <cmdline_rdline.h>
-#include <cmdline_parse.h>
-#include <cmdline_parse_string.h>
-#include <cmdline_socket.h>
-#include <cmdline.h>
 #include "mp_commands.h"
 
-/**********************************************************/
-
-struct cmd_send_result {
-	cmdline_fixed_string_t action;
-	cmdline_fixed_string_t message;
-};
+extern struct rte_ring *send_ring, *recv_ring;
+extern struct rte_mempool *message_pool;
+extern volatile int quit;
 
-static void cmd_send_parsed(void *parsed_result,
+void
+cmd_send_parsed(void *parsed_result,
 		__rte_unused struct cmdline *cl,
 		__rte_unused void *data)
 {
@@ -54,29 +28,8 @@ static void cmd_send_parsed(void *parsed_result,
 	}
 }
 
-cmdline_parse_token_string_t cmd_send_action =
-	TOKEN_STRING_INITIALIZER(struct cmd_send_result, action, "send");
-cmdline_parse_token_string_t cmd_send_message =
-	TOKEN_STRING_INITIALIZER(struct cmd_send_result, message, NULL);
-
-cmdline_parse_inst_t cmd_send = {
-	.f = cmd_send_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "send a string to another process",
-	.tokens = {        /* token list, NULL terminated */
-			(void *)&cmd_send_action,
-			(void *)&cmd_send_message,
-			NULL,
-	},
-};
-
-/**********************************************************/
-
-struct cmd_quit_result {
-	cmdline_fixed_string_t quit;
-};
-
-static void cmd_quit_parsed(__rte_unused void *parsed_result,
+void
+cmd_quit_parsed(__rte_unused void *parsed_result,
 			    struct cmdline *cl,
 			    __rte_unused void *data)
 {
@@ -84,26 +37,8 @@ static void cmd_quit_parsed(__rte_unused void *parsed_result,
 	cmdline_quit(cl);
 }
 
-cmdline_parse_token_string_t cmd_quit_quit =
-	TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit");
-
-cmdline_parse_inst_t cmd_quit = {
-	.f = cmd_quit_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "close the application",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_quit_quit,
-		NULL,
-	},
-};
-
-/**********************************************************/
-
-struct cmd_help_result {
-	cmdline_fixed_string_t help;
-};
-
-static void cmd_help_parsed(__rte_unused void *parsed_result,
+void
+cmd_help_parsed(__rte_unused void *parsed_result,
 			    struct cmdline *cl,
 			    __rte_unused void *data)
 {
@@ -112,24 +47,3 @@ static void cmd_help_parsed(__rte_unused void *parsed_result,
 			"send commands to the simple app. Commands supported are:\n\n"
 			"- send [string]\n" "- help\n" "- quit\n\n");
 }
-
-cmdline_parse_token_string_t cmd_help_help =
-	TOKEN_STRING_INITIALIZER(struct cmd_help_result, help, "help");
-
-cmdline_parse_inst_t cmd_help = {
-	.f = cmd_help_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "show help",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_help_help,
-		NULL,
-	},
-};
-
-/****** CONTEXT (list of instruction) */
-cmdline_parse_ctx_t simple_mp_ctx[] = {
-		(cmdline_parse_inst_t *)&cmd_send,
-		(cmdline_parse_inst_t *)&cmd_quit,
-		(cmdline_parse_inst_t *)&cmd_help,
-	NULL,
-};
diff --git a/examples/multi_process/simple_mp/mp_commands.h b/examples/multi_process/simple_mp/mp_commands.h
deleted file mode 100644
index 5d67413e7c..0000000000
--- a/examples/multi_process/simple_mp/mp_commands.h
+++ /dev/null
@@ -1,14 +0,0 @@
-/* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2010-2014 Intel Corporation
- */
-
-#ifndef _SIMPLE_MP_COMMANDS_H_
-#define _SIMPLE_MP_COMMANDS_H_
-
-extern struct rte_ring *send_ring;
-extern struct rte_mempool *message_pool;
-extern volatile int quit;
-
-extern cmdline_parse_ctx_t simple_mp_ctx[];
-
-#endif /* _SIMPLE_MP_COMMANDS_H_ */
diff --git a/examples/multi_process/simple_mp/mp_commands.list b/examples/multi_process/simple_mp/mp_commands.list
new file mode 100644
index 0000000000..c8890cb071
--- /dev/null
+++ b/examples/multi_process/simple_mp/mp_commands.list
@@ -0,0 +1,3 @@
+send <STRING>message  # send a string to another process
+help                  # show help
+quit                  # close the application
-- 
2.39.2

[PATCH v3 4/5] examples/hotplug_mp: auto-generate cmdline boilerplate

[Bruce Richardson]

Use the dpdk-cmdline-gen script to autogenerate all the boilerplate
structs and defines for the commandline part of the app.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 examples/multi_process/hotplug_mp/Makefile    |  12 +-
 examples/multi_process/hotplug_mp/commands.c  | 147 ++----------------
 examples/multi_process/hotplug_mp/commands.h  |  10 --
 .../multi_process/hotplug_mp/commands.list    |   5 +
 examples/multi_process/hotplug_mp/meson.build |   9 ++
 5 files changed, 35 insertions(+), 148 deletions(-)
 delete mode 100644 examples/multi_process/hotplug_mp/commands.h
 create mode 100644 examples/multi_process/hotplug_mp/commands.list

diff --git a/examples/multi_process/hotplug_mp/Makefile b/examples/multi_process/hotplug_mp/Makefile
index 6b20d6e49a..81ee85cd6b 100644
--- a/examples/multi_process/hotplug_mp/Makefile
+++ b/examples/multi_process/hotplug_mp/Makefile
@@ -6,6 +6,7 @@ APP = hotplug_mp
 
 # all source are stored in SRCS-y
 SRCS-y := main.c commands.c
+SRC-DEPS := build/commands.h
 
 PKGCONF ?= pkg-config
 
@@ -22,10 +23,13 @@ static: build/$(APP)-static
 	ln -sf $(APP)-static build/$(APP)
 
 PC_FILE := $(shell $(PKGCONF) --path libdpdk 2>/dev/null)
-CFLAGS += -O3 $(shell $(PKGCONF) --cflags libdpdk)
+CFLAGS += -O3 $(shell $(PKGCONF) --cflags libdpdk) -I build/
 LDFLAGS_SHARED = $(shell $(PKGCONF) --libs libdpdk)
 LDFLAGS_STATIC = $(shell $(PKGCONF) --static --libs libdpdk)
 
+build/commands.h: commands.list Makefile
+	dpdk-cmdline-gen.py -o $@ --context-name=main_ctx $<
+
 ifeq ($(MAKECMDGOALS),static)
 # check for broken pkg-config
 ifeq ($(shell echo $(LDFLAGS_STATIC) | grep 'whole-archive.*l:lib.*no-whole-archive'),)
@@ -36,10 +40,10 @@ endif
 
 CFLAGS += -DALLOW_EXPERIMENTAL_API
 
-build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
+build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build $(SRC-DEPS)
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
 
-build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build
+build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build  $(SRC-DEPS)
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_STATIC)
 
 build:
@@ -47,5 +51,5 @@ build:
 
 .PHONY: clean
 clean:
-	rm -f build/$(APP) build/$(APP)-static build/$(APP)-shared
+	rm -f build/$(APP) build/$(APP)-static build/$(APP)-shared build/commands.h
 	test -d build && rmdir -p build || true
diff --git a/examples/multi_process/hotplug_mp/commands.c b/examples/multi_process/hotplug_mp/commands.c
index 88f44e00a0..900eb9f774 100644
--- a/examples/multi_process/hotplug_mp/commands.c
+++ b/examples/multi_process/hotplug_mp/commands.c
@@ -1,24 +1,12 @@
 /* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2018 Intel Corporation.
+ * Copyright(c) 2018-2023 Intel Corporation.
  */
 
-#include <cmdline_rdline.h>
-#include <cmdline_parse.h>
-#include <cmdline_parse_ipaddr.h>
-#include <cmdline_parse_num.h>
-#include <cmdline_parse_string.h>
-#include <cmdline.h>
-
 #include <rte_bus.h>
 #include <rte_ethdev.h>
+#include "commands.h"
 
-/**********************************************************/
-
-struct cmd_help_result {
-	cmdline_fixed_string_t help;
-};
-
-static void cmd_help_parsed(__rte_unused void *parsed_result,
+void cmd_help_parsed(__rte_unused void *parsed_result,
 			    struct cmdline *cl,
 			    __rte_unused void *data)
 {
@@ -29,52 +17,16 @@ static void cmd_help_parsed(__rte_unused void *parsed_result,
 		       "- list\n\n");
 }
 
-cmdline_parse_token_string_t cmd_help_help =
-	TOKEN_STRING_INITIALIZER(struct cmd_help_result, help, "help");
-
-cmdline_parse_inst_t cmd_help = {
-	.f = cmd_help_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "show help",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_help_help,
-		NULL,
-	},
-};
-
-/**********************************************************/
-
-struct cmd_quit_result {
-	cmdline_fixed_string_t quit;
-};
-
-static void cmd_quit_parsed(__rte_unused void *parsed_result,
+void
+cmd_quit_parsed(__rte_unused void *parsed_result,
 			    struct cmdline *cl,
 			    __rte_unused void *data)
 {
 	cmdline_quit(cl);
 }
 
-cmdline_parse_token_string_t cmd_quit_quit =
-	TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit");
-
-cmdline_parse_inst_t cmd_quit = {
-	.f = cmd_quit_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "quit",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_quit_quit,
-		NULL,
-	},
-};
-
-/**********************************************************/
-
-struct cmd_list_result {
-	cmdline_fixed_string_t list;
-};
-
-static void cmd_list_parsed(__rte_unused void *parsed_result,
+void
+cmd_list_parsed(__rte_unused void *parsed_result,
 			    struct cmdline *cl,
 			    __rte_unused void *data)
 {
@@ -92,31 +44,12 @@ static void cmd_list_parsed(__rte_unused void *parsed_result,
 	}
 }
 
-cmdline_parse_token_string_t cmd_list_list =
-	TOKEN_STRING_INITIALIZER(struct cmd_list_result, list, "list");
-
-cmdline_parse_inst_t cmd_list = {
-	.f = cmd_list_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "list all devices",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_list_list,
-		NULL,
-	},
-};
-
-/**********************************************************/
-
-struct cmd_dev_attach_result {
-	cmdline_fixed_string_t attach;
-	cmdline_fixed_string_t devargs;
-};
-
-static void cmd_dev_attach_parsed(void *parsed_result,
+void
+cmd_attach_parsed(void *parsed_result,
 				  struct cmdline *cl,
 				  __rte_unused void *data)
 {
-	struct cmd_dev_attach_result *res = parsed_result;
+	struct cmd_attach_result *res = parsed_result;
 	struct rte_devargs da;
 
 	memset(&da, 0, sizeof(da));
@@ -134,35 +67,12 @@ static void cmd_dev_attach_parsed(void *parsed_result,
 	rte_devargs_reset(&da);
 }
 
-cmdline_parse_token_string_t cmd_dev_attach_attach =
-	TOKEN_STRING_INITIALIZER(struct cmd_dev_attach_result, attach,
-				 "attach");
-cmdline_parse_token_string_t cmd_dev_attach_devargs =
-	TOKEN_STRING_INITIALIZER(struct cmd_dev_attach_result, devargs, NULL);
-
-cmdline_parse_inst_t cmd_attach_device = {
-	.f = cmd_dev_attach_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "attach a device",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_dev_attach_attach,
-		(void *)&cmd_dev_attach_devargs,
-		NULL,
-	},
-};
-
-/**********************************************************/
-
-struct cmd_dev_detach_result {
-	cmdline_fixed_string_t detach;
-	cmdline_fixed_string_t devargs;
-};
-
-static void cmd_dev_detach_parsed(void *parsed_result,
+void
+cmd_detach_parsed(void *parsed_result,
 				   struct cmdline *cl,
 				   __rte_unused void *data)
 {
-	struct cmd_dev_detach_result *res = parsed_result;
+	struct cmd_detach_result *res = parsed_result;
 	struct rte_devargs da;
 
 	memset(&da, 0, sizeof(da));
@@ -181,34 +91,3 @@ static void cmd_dev_detach_parsed(void *parsed_result,
 			da.name);
 	rte_devargs_reset(&da);
 }
-
-cmdline_parse_token_string_t cmd_dev_detach_detach =
-	TOKEN_STRING_INITIALIZER(struct cmd_dev_detach_result, detach,
-				 "detach");
-
-cmdline_parse_token_string_t cmd_dev_detach_devargs =
-	TOKEN_STRING_INITIALIZER(struct cmd_dev_detach_result, devargs, NULL);
-
-cmdline_parse_inst_t cmd_detach_device = {
-	.f = cmd_dev_detach_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "detach a device",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_dev_detach_detach,
-		(void *)&cmd_dev_detach_devargs,
-		NULL,
-	},
-};
-
-/**********************************************************/
-/**********************************************************/
-/****** CONTEXT (list of instruction) */
-
-cmdline_parse_ctx_t main_ctx[] = {
-	(cmdline_parse_inst_t *)&cmd_help,
-	(cmdline_parse_inst_t *)&cmd_quit,
-	(cmdline_parse_inst_t *)&cmd_list,
-	(cmdline_parse_inst_t *)&cmd_attach_device,
-	(cmdline_parse_inst_t *)&cmd_detach_device,
-	NULL,
-};
diff --git a/examples/multi_process/hotplug_mp/commands.h b/examples/multi_process/hotplug_mp/commands.h
deleted file mode 100644
index afcf177dba..0000000000
--- a/examples/multi_process/hotplug_mp/commands.h
+++ /dev/null
@@ -1,10 +0,0 @@
-/* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2018 Intel Corporation
- */
-
-#ifndef _COMMANDS_H_
-#define _COMMANDS_H_
-
-extern cmdline_parse_ctx_t main_ctx[];
-
-#endif /* _COMMANDS_H_ */
diff --git a/examples/multi_process/hotplug_mp/commands.list b/examples/multi_process/hotplug_mp/commands.list
new file mode 100644
index 0000000000..e3de503965
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/commands.list
@@ -0,0 +1,5 @@
+attach <STRING>devargs     # attach a device
+detach <STRING>devargs     # detach a device
+list                       # list all devices
+help                       # show help
+quit                       # quit
diff --git a/examples/multi_process/hotplug_mp/meson.build b/examples/multi_process/hotplug_mp/meson.build
index a1ad98ca2e..7a0e9ca47a 100644
--- a/examples/multi_process/hotplug_mp/meson.build
+++ b/examples/multi_process/hotplug_mp/meson.build
@@ -7,7 +7,16 @@
 # DPDK instance, use 'make'
 
 allow_experimental_apis = true
+
+cmd_h = custom_target('commands_hdr',
+	output: 'commands.h',
+	input: files('commands.list'),
+	capture: true,
+	command: [cmdline_gen_cmd, '--context-name=main_ctx', '@INPUT@']
+)
+
 sources = files(
         'commands.c',
         'main.c',
 )
+sources += cmd_h
-- 
2.39.2

[PATCH v3 5/5] examples/bond: auto-generate cmdline boilerplate

[Bruce Richardson]

Use the dpdk-cmdline-gen script to autogenerate all the boilerplate
structs and defines for the commandline part of the app.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>

---
Note: the original help text on some of the commands in this example
  were not useful "this command do not handle any arguments". Therefore,
  when converting over to the command script, the per-command help
  info has been updated with reference to the code rather than a literal
  transfer of the existing help text, as was done with the previous 2
  example apps.
---
 examples/bond/Makefile      |  12 ++-
 examples/bond/commands.list |   6 ++
 examples/bond/main.c        | 161 ++++--------------------------------
 examples/bond/main.h        |  10 ---
 examples/bond/meson.build   |   8 ++
 5 files changed, 37 insertions(+), 160 deletions(-)
 create mode 100644 examples/bond/commands.list
 delete mode 100644 examples/bond/main.h

diff --git a/examples/bond/Makefile b/examples/bond/Makefile
index ad711a5bee..d87c7a32ba 100644
--- a/examples/bond/Makefile
+++ b/examples/bond/Makefile
@@ -6,6 +6,7 @@ APP = bond_app
 
 # all source are stored in SRCS-y
 SRCS-y := main.c
+SRC-DEPS := build/commands.h
 
 PKGCONF ?= pkg-config
 
@@ -24,10 +25,13 @@ static: build/$(APP)-static
 LDFLAGS += -lrte_net_bond
 
 PC_FILE := $(shell $(PKGCONF) --path libdpdk 2>/dev/null)
-CFLAGS += -O3 $(shell $(PKGCONF) --cflags libdpdk)
+CFLAGS += -O3 $(shell $(PKGCONF) --cflags libdpdk) -I build/
 LDFLAGS_SHARED = $(shell $(PKGCONF) --libs libdpdk)
 LDFLAGS_STATIC = $(shell $(PKGCONF) --static --libs libdpdk)
 
+build/commands.h: commands.list Makefile
+	dpdk-cmdline-gen.py -o $@ --context-name=main_ctx $<
+
 ifeq ($(MAKECMDGOALS),static)
 # check for broken pkg-config
 ifeq ($(shell echo $(LDFLAGS_STATIC) | grep 'whole-archive.*l:lib.*no-whole-archive'),)
@@ -38,10 +42,10 @@ endif
 
 CFLAGS += -DALLOW_EXPERIMENTAL_API
 
-build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
+build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build $(SRC-DEPS)
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
 
-build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build
+build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build  $(SRC-DEPS)
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_STATIC)
 
 build:
@@ -49,5 +53,5 @@ build:
 
 .PHONY: clean
 clean:
-	rm -f build/$(APP) build/$(APP)-static build/$(APP)-shared
+	rm -f build/$(APP) build/$(APP)-static build/$(APP)-shared build/*.h
 	test -d build && rmdir -p build || true
diff --git a/examples/bond/commands.list b/examples/bond/commands.list
new file mode 100644
index 0000000000..a10bf75708
--- /dev/null
+++ b/examples/bond/commands.list
@@ -0,0 +1,6 @@
+send <IP>ip   # sends one ARPrequest through bonding for IP
+start         # starts listening if not started at startup
+stop          # stops listening
+show          # shows some bond info, e.g. active members
+help          # show help
+quit          # close application
diff --git a/examples/bond/main.c b/examples/bond/main.c
index 90f422ec11..8528abf675 100644
--- a/examples/bond/main.c
+++ b/examples/bond/main.c
@@ -45,16 +45,8 @@
 #include <rte_cpuflags.h>
 #include <rte_eth_bond.h>
 
-#include <cmdline_rdline.h>
-#include <cmdline_parse.h>
-#include <cmdline_parse_num.h>
-#include <cmdline_parse_string.h>
-#include <cmdline_parse_ipaddr.h>
-#include <cmdline_parse_etheraddr.h>
 #include <cmdline_socket.h>
-#include <cmdline.h>
-
-#include "main.h"
+#include "commands.h"
 
 #define RTE_LOGTYPE_DCB RTE_LOGTYPE_USER1
 
@@ -462,11 +454,7 @@ static int lcore_main(__rte_unused void *arg1)
 	return 0;
 }
 
-struct cmd_obj_send_result {
-	cmdline_fixed_string_t action;
-	cmdline_ipaddr_t ip;
-};
-static inline void get_string(struct cmd_obj_send_result *res, char *buf, uint8_t size)
+static inline void get_string(struct cmd_send_result *res, char *buf, uint8_t size)
 {
 	snprintf(buf, size, NIPQUAD_FMT,
 		((unsigned)((unsigned char *)&(res->ip.addr.ipv4))[0]),
@@ -475,12 +463,11 @@ static inline void get_string(struct cmd_obj_send_result *res, char *buf, uint8_
 		((unsigned)((unsigned char *)&(res->ip.addr.ipv4))[3])
 		);
 }
-static void cmd_obj_send_parsed(void *parsed_result,
-		__rte_unused struct cmdline *cl,
-			       __rte_unused void *data)
+void
+cmd_send_parsed(void *parsed_result, __rte_unused struct cmdline *cl, __rte_unused void *data)
 {
 
-	struct cmd_obj_send_result *res = parsed_result;
+	struct cmd_send_result *res = parsed_result;
 	char ip_str[INET6_ADDRSTRLEN];
 
 	struct rte_ether_addr bond_mac_addr;
@@ -544,29 +531,8 @@ static void cmd_obj_send_parsed(void *parsed_result,
 	cmdline_printf(cl, "\n");
 }
 
-cmdline_parse_token_string_t cmd_obj_action_send =
-	TOKEN_STRING_INITIALIZER(struct cmd_obj_send_result, action, "send");
-cmdline_parse_token_ipaddr_t cmd_obj_ip =
-	TOKEN_IPV4_INITIALIZER(struct cmd_obj_send_result, ip);
-
-cmdline_parse_inst_t cmd_obj_send = {
-	.f = cmd_obj_send_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "send client_ip",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_obj_action_send,
-		(void *)&cmd_obj_ip,
-		NULL,
-	},
-};
-
-struct cmd_start_result {
-	cmdline_fixed_string_t start;
-};
-
-static void cmd_start_parsed(__rte_unused void *parsed_result,
-			       struct cmdline *cl,
-			       __rte_unused void *data)
+void
+cmd_start_parsed(__rte_unused void *parsed_result, struct cmdline *cl, __rte_unused void *data)
 {
 	int worker_core_id = rte_lcore_id();
 
@@ -605,26 +571,8 @@ static void cmd_start_parsed(__rte_unused void *parsed_result,
 		);
 }
 
-cmdline_parse_token_string_t cmd_start_start =
-	TOKEN_STRING_INITIALIZER(struct cmd_start_result, start, "start");
-
-cmdline_parse_inst_t cmd_start = {
-	.f = cmd_start_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "starts listening if not started at startup",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_start_start,
-		NULL,
-	},
-};
-
-struct cmd_help_result {
-	cmdline_fixed_string_t help;
-};
-
-static void cmd_help_parsed(__rte_unused void *parsed_result,
-			    struct cmdline *cl,
-			    __rte_unused void *data)
+void
+cmd_help_parsed(__rte_unused void *parsed_result, struct cmdline *cl, __rte_unused void *data)
 {
 	cmdline_printf(cl,
 			"ALB - link bonding mode 6 example\n"
@@ -637,26 +585,8 @@ static void cmd_help_parsed(__rte_unused void *parsed_result,
 		       );
 }
 
-cmdline_parse_token_string_t cmd_help_help =
-	TOKEN_STRING_INITIALIZER(struct cmd_help_result, help, "help");
-
-cmdline_parse_inst_t cmd_help = {
-	.f = cmd_help_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "show help",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_help_help,
-		NULL,
-	},
-};
-
-struct cmd_stop_result {
-	cmdline_fixed_string_t stop;
-};
-
-static void cmd_stop_parsed(__rte_unused void *parsed_result,
-			    struct cmdline *cl,
-			    __rte_unused void *data)
+void
+cmd_stop_parsed(__rte_unused void *parsed_result, struct cmdline *cl, __rte_unused void *data)
 {
 	rte_spinlock_lock(&global_flag_stru_p->lock);
 	if (global_flag_stru_p->LcoreMainIsRunning == 0)	{
@@ -678,26 +608,8 @@ static void cmd_stop_parsed(__rte_unused void *parsed_result,
 	rte_spinlock_unlock(&global_flag_stru_p->lock);
 }
 
-cmdline_parse_token_string_t cmd_stop_stop =
-	TOKEN_STRING_INITIALIZER(struct cmd_stop_result, stop, "stop");
-
-cmdline_parse_inst_t cmd_stop = {
-	.f = cmd_stop_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "this command do not handle any arguments",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_stop_stop,
-		NULL,
-	},
-};
-
-struct cmd_quit_result {
-	cmdline_fixed_string_t quit;
-};
-
-static void cmd_quit_parsed(__rte_unused void *parsed_result,
-			    struct cmdline *cl,
-			    __rte_unused void *data)
+void
+cmd_quit_parsed(__rte_unused void *parsed_result, struct cmdline *cl, __rte_unused void *data)
 {
 	rte_spinlock_lock(&global_flag_stru_p->lock);
 	if (global_flag_stru_p->LcoreMainIsRunning == 0)	{
@@ -721,26 +633,8 @@ static void cmd_quit_parsed(__rte_unused void *parsed_result,
 	cmdline_quit(cl);
 }
 
-cmdline_parse_token_string_t cmd_quit_quit =
-	TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit");
-
-cmdline_parse_inst_t cmd_quit = {
-	.f = cmd_quit_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "this command do not handle any arguments",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_quit_quit,
-		NULL,
-	},
-};
-
-struct cmd_show_result {
-	cmdline_fixed_string_t show;
-};
-
-static void cmd_show_parsed(__rte_unused void *parsed_result,
-			    struct cmdline *cl,
-			    __rte_unused void *data)
+void
+cmd_show_parsed(__rte_unused void *parsed_result, struct cmdline *cl, __rte_unused void *data)
 {
 	uint16_t members[16] = {0};
 	uint8_t len = 16;
@@ -772,31 +666,6 @@ static void cmd_show_parsed(__rte_unused void *parsed_result,
 	rte_spinlock_unlock(&global_flag_stru_p->lock);
 }
 
-cmdline_parse_token_string_t cmd_show_show =
-	TOKEN_STRING_INITIALIZER(struct cmd_show_result, show, "show");
-
-cmdline_parse_inst_t cmd_show = {
-	.f = cmd_show_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "this command do not handle any arguments",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_show_show,
-		NULL,
-	},
-};
-
-/****** CONTEXT (list of instruction) */
-
-cmdline_parse_ctx_t main_ctx[] = {
-	(cmdline_parse_inst_t *)&cmd_start,
-	(cmdline_parse_inst_t *)&cmd_obj_send,
-	(cmdline_parse_inst_t *)&cmd_stop,
-	(cmdline_parse_inst_t *)&cmd_show,
-	(cmdline_parse_inst_t *)&cmd_quit,
-	(cmdline_parse_inst_t *)&cmd_help,
-	NULL,
-};
-
 /* prompt function, called from main on MAIN lcore */
 static void prompt(__rte_unused void *arg1)
 {
diff --git a/examples/bond/main.h b/examples/bond/main.h
deleted file mode 100644
index f91ed9c885..0000000000
--- a/examples/bond/main.h
+++ /dev/null
@@ -1,10 +0,0 @@
-/* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2010-2015 Intel Corporation
- */
-
-#ifndef _MAIN_H_
-#define _MAIN_H_
-
-int main(int argc, char *argv[]);
-
-#endif /* ifndef _MAIN_H_ */
diff --git a/examples/bond/meson.build b/examples/bond/meson.build
index ed22b7887d..bfbec04ecd 100644
--- a/examples/bond/meson.build
+++ b/examples/bond/meson.build
@@ -11,3 +11,11 @@ allow_experimental_apis = true
 sources = files(
         'main.c',
 )
+
+cmd_h = custom_target('commands_hdr',
+	output: 'commands.h',
+	input: files('commands.list'),
+	capture: true,
+	command: [cmdline_gen_cmd, '--context-name=main_ctx', '@INPUT@']
+)
+sources += cmd_h
-- 
2.39.2

Re: [PATCH v3 0/5] document and simplify use of cmdline

[David Marchand]

On Wed, Oct 11, 2023 at 3:34 PM Bruce Richardson
<bruce.richardson@intel.com> wrote:
>
> The DPDK commandline library is widely used by apps and examples within
> DPDK, but it is not documented in our programmers guide and it requires
> a lot of boilerplate code definitions in order to used. We can improve
> this situation by creating a simple python script to automatically
> generate the boilerplate from a list of commands.
>
> This patchset contains a new documentation chapter on cmdline library,
> going through step-by-step how to add commands and create the necessary
> token lists and parse contexts.
>
> Following that initial doc patch, the set then contains a
> boilerplate-generating script, as well as a set of three patches showing
> its use, by converting three examples to use the script instead of
> having the hard-coded boilerplate. Once the script is used, adding a new
> command becomes as simple as adding the desired command to the .list
> file, and then writing the required function which will be called for
> that command. No other boilerplate coding is necessary.
>
> Script obviously does not cover the full range of capabilities of the
> commandline lib, but does cover the most used parts. The code-saving to
> each of the examples by auto-generating the boilerplate is significant,
> and probably more examples with commandlines can be converted over in
> future.
>
> The "cmdline" example itself, is not converted over, as it should
> probably remain as a simple example of direct library use without the
> script.
>
> V3:
> * Added lots of documentation
> * Added support for help text for each command
> * Cleaned up script a little so it passes pycodestyle and most flake8
>   checks, when line-length is set to max 100.
> * Removed RFC tag, as I consider this patchset stable enough for
>   consideration in a release.
>
> V2-RFC:
> * Add support for IP addresses in commands
> * Move to buildtools directory and make installable
> * Convert 3 examples to use script, and eliminate their boilerplate
>
> Bruce Richardson (5):
>   doc/prog_guide: new chapter on cmdline library
>   buildtools: script to generate cmdline boilerplate
>   examples/simple_mp: auto-generate cmdline boilerplate
>   examples/hotplug_mp: auto-generate cmdline boilerplate
>   examples/bond: auto-generate cmdline boilerplate
>
>  app/test/commands.c                           |   2 +
>  buildtools/dpdk-cmdline-gen.py                | 167 +++++++
>  buildtools/meson.build                        |   7 +
>  doc/guides/prog_guide/cmdline.rst             | 466 ++++++++++++++++++
>  doc/guides/prog_guide/index.rst               |   1 +
>  examples/bond/Makefile                        |  12 +-
>  examples/bond/commands.list                   |   6 +
>  examples/bond/main.c                          | 161 +-----
>  examples/bond/main.h                          |  10 -
>  examples/bond/meson.build                     |   8 +
>  examples/multi_process/hotplug_mp/Makefile    |  12 +-
>  examples/multi_process/hotplug_mp/commands.c  | 147 +-----
>  examples/multi_process/hotplug_mp/commands.h  |  10 -
>  .../multi_process/hotplug_mp/commands.list    |   5 +
>  examples/multi_process/hotplug_mp/meson.build |   9 +
>  examples/multi_process/simple_mp/Makefile     |  12 +-
>  examples/multi_process/simple_mp/meson.build  |   9 +
>  .../multi_process/simple_mp/mp_commands.c     | 106 +---
>  .../multi_process/simple_mp/mp_commands.h     |  14 -
>  .../multi_process/simple_mp/mp_commands.list  |   3 +
>  20 files changed, 745 insertions(+), 422 deletions(-)

Interesting series.
So if we remove the doc addition, this patch is removing loc, so +1 from me :-).

There is a problem though with externally building the examples.
Maybe the dpdk-cmdline-gen.py has not been exported in the install process.


>  create mode 100755 buildtools/dpdk-cmdline-gen.py
>  create mode 100644 doc/guides/prog_guide/cmdline.rst
>  create mode 100644 examples/bond/commands.list
>  delete mode 100644 examples/bond/main.h
>  delete mode 100644 examples/multi_process/hotplug_mp/commands.h
>  create mode 100644 examples/multi_process/hotplug_mp/commands.list
>  delete mode 100644 examples/multi_process/simple_mp/mp_commands.h
>  create mode 100644 examples/multi_process/simple_mp/mp_commands.list
>
> --
> 2.39.2
>


-- 
David Marchand

Re: [PATCH v3 0/5] document and simplify use of cmdline

[Bruce Richardson]

On Thu, Oct 12, 2023 at 03:21:08PM +0200, David Marchand wrote:
> On Wed, Oct 11, 2023 at 3:34 PM Bruce Richardson
> <bruce.richardson@intel.com> wrote:
> >
> > The DPDK commandline library is widely used by apps and examples within
> > DPDK, but it is not documented in our programmers guide and it requires
> > a lot of boilerplate code definitions in order to used. We can improve
> > this situation by creating a simple python script to automatically
> > generate the boilerplate from a list of commands.
> >
> > This patchset contains a new documentation chapter on cmdline library,
> > going through step-by-step how to add commands and create the necessary
> > token lists and parse contexts.
> >
> > Following that initial doc patch, the set then contains a
> > boilerplate-generating script, as well as a set of three patches showing
> > its use, by converting three examples to use the script instead of
> > having the hard-coded boilerplate. Once the script is used, adding a new
> > command becomes as simple as adding the desired command to the .list
> > file, and then writing the required function which will be called for
> > that command. No other boilerplate coding is necessary.
> >
> > Script obviously does not cover the full range of capabilities of the
> > commandline lib, but does cover the most used parts. The code-saving to
> > each of the examples by auto-generating the boilerplate is significant,
> > and probably more examples with commandlines can be converted over in
> > future.
> >
> > The "cmdline" example itself, is not converted over, as it should
> > probably remain as a simple example of direct library use without the
> > script.
> >
> > V3:
> > * Added lots of documentation
> > * Added support for help text for each command
> > * Cleaned up script a little so it passes pycodestyle and most flake8
> >   checks, when line-length is set to max 100.
> > * Removed RFC tag, as I consider this patchset stable enough for
> >   consideration in a release.
> >
> > V2-RFC:
> > * Add support for IP addresses in commands
> > * Move to buildtools directory and make installable
> > * Convert 3 examples to use script, and eliminate their boilerplate
> >
> > Bruce Richardson (5):
> >   doc/prog_guide: new chapter on cmdline library
> >   buildtools: script to generate cmdline boilerplate
> >   examples/simple_mp: auto-generate cmdline boilerplate
> >   examples/hotplug_mp: auto-generate cmdline boilerplate
> >   examples/bond: auto-generate cmdline boilerplate
> >
> >  app/test/commands.c                           |   2 +
> >  buildtools/dpdk-cmdline-gen.py                | 167 +++++++
> >  buildtools/meson.build                        |   7 +
> >  doc/guides/prog_guide/cmdline.rst             | 466 ++++++++++++++++++
> >  doc/guides/prog_guide/index.rst               |   1 +
> >  examples/bond/Makefile                        |  12 +-
> >  examples/bond/commands.list                   |   6 +
> >  examples/bond/main.c                          | 161 +-----
> >  examples/bond/main.h                          |  10 -
> >  examples/bond/meson.build                     |   8 +
> >  examples/multi_process/hotplug_mp/Makefile    |  12 +-
> >  examples/multi_process/hotplug_mp/commands.c  | 147 +-----
> >  examples/multi_process/hotplug_mp/commands.h  |  10 -
> >  .../multi_process/hotplug_mp/commands.list    |   5 +
> >  examples/multi_process/hotplug_mp/meson.build |   9 +
> >  examples/multi_process/simple_mp/Makefile     |  12 +-
> >  examples/multi_process/simple_mp/meson.build  |   9 +
> >  .../multi_process/simple_mp/mp_commands.c     | 106 +---
> >  .../multi_process/simple_mp/mp_commands.h     |  14 -
> >  .../multi_process/simple_mp/mp_commands.list  |   3 +
> >  20 files changed, 745 insertions(+), 422 deletions(-)
> 
> Interesting series.
> So if we remove the doc addition, this patch is removing loc, so +1 from me :-).
> 
> There is a problem though with externally building the examples.
> Maybe the dpdk-cmdline-gen.py has not been exported in the install process.
> 
It has, but it's not visible to the external builds in github actions. The
following one-line fix should correct that, I believe.
I was testing it in my own github yesterday, but never sent a v4 series.

/Bruce

diff --git a/.ci/linux-build.sh b/.ci/linux-build.sh
index e0b62bac90..3db9d9de6e 100755
--- a/.ci/linux-build.sh
+++ b/.ci/linux-build.sh
@@ -174,6 +174,7 @@ fi
 if [ "$BUILD_EXAMPLES" = "true" ]; then
     [ -d install ] || DESTDIR=$(pwd)/install meson install -C build
     export LD_LIBRARY_PATH=$(dirname $(find $(pwd)/install -name librte_eal.so)):$LD_LIBRARY_PATH
+    export PATH=$(dirname $(find $(pwd)/install -name dpdk-devbind.py)):$PATH
     export PKG_CONFIG_PATH=$(dirname $(find $(pwd)/install -name libdpdk.pc)):$PKG_CONFIG_PATH
     export PKGCONF="pkg-config --define-prefix"
     find build/examples -maxdepth 1 -type f -name "dpdk-*" |

Re: [PATCH v3 0/5] document and simplify use of cmdline

[Bruce Richardson]

On Thu, Oct 12, 2023 at 03:21:08PM +0200, David Marchand wrote:
> On Wed, Oct 11, 2023 at 3:34 PM Bruce Richardson
> <bruce.richardson@intel.com> wrote:
> >
> > The DPDK commandline library is widely used by apps and examples within
> > DPDK, but it is not documented in our programmers guide and it requires
> > a lot of boilerplate code definitions in order to used. We can improve
> > this situation by creating a simple python script to automatically
> > generate the boilerplate from a list of commands.
> >
> > This patchset contains a new documentation chapter on cmdline library,
> > going through step-by-step how to add commands and create the necessary
> > token lists and parse contexts.
> >
> > Following that initial doc patch, the set then contains a
> > boilerplate-generating script, as well as a set of three patches showing
> > its use, by converting three examples to use the script instead of
> > having the hard-coded boilerplate. Once the script is used, adding a new
> > command becomes as simple as adding the desired command to the .list
> > file, and then writing the required function which will be called for
> > that command. No other boilerplate coding is necessary.
> >
> > Script obviously does not cover the full range of capabilities of the
> > commandline lib, but does cover the most used parts. The code-saving to
> > each of the examples by auto-generating the boilerplate is significant,
> > and probably more examples with commandlines can be converted over in
> > future.
> >
> > The "cmdline" example itself, is not converted over, as it should
> > probably remain as a simple example of direct library use without the
> > script.
> >
> > V3:
> > * Added lots of documentation
> > * Added support for help text for each command
> > * Cleaned up script a little so it passes pycodestyle and most flake8
> >   checks, when line-length is set to max 100.
> > * Removed RFC tag, as I consider this patchset stable enough for
> >   consideration in a release.
> >
> > V2-RFC:
> > * Add support for IP addresses in commands
> > * Move to buildtools directory and make installable
> > * Convert 3 examples to use script, and eliminate their boilerplate
> >
> > Bruce Richardson (5):
> >   doc/prog_guide: new chapter on cmdline library
> >   buildtools: script to generate cmdline boilerplate
> >   examples/simple_mp: auto-generate cmdline boilerplate
> >   examples/hotplug_mp: auto-generate cmdline boilerplate
> >   examples/bond: auto-generate cmdline boilerplate
> >
> >  app/test/commands.c                           |   2 +
> >  buildtools/dpdk-cmdline-gen.py                | 167 +++++++
> >  buildtools/meson.build                        |   7 +
> >  doc/guides/prog_guide/cmdline.rst             | 466 ++++++++++++++++++
> >  doc/guides/prog_guide/index.rst               |   1 +
> >  examples/bond/Makefile                        |  12 +-
> >  examples/bond/commands.list                   |   6 +
> >  examples/bond/main.c                          | 161 +-----
> >  examples/bond/main.h                          |  10 -
> >  examples/bond/meson.build                     |   8 +
> >  examples/multi_process/hotplug_mp/Makefile    |  12 +-
> >  examples/multi_process/hotplug_mp/commands.c  | 147 +-----
> >  examples/multi_process/hotplug_mp/commands.h  |  10 -
> >  .../multi_process/hotplug_mp/commands.list    |   5 +
> >  examples/multi_process/hotplug_mp/meson.build |   9 +
> >  examples/multi_process/simple_mp/Makefile     |  12 +-
> >  examples/multi_process/simple_mp/meson.build  |   9 +
> >  .../multi_process/simple_mp/mp_commands.c     | 106 +---
> >  .../multi_process/simple_mp/mp_commands.h     |  14 -
> >  .../multi_process/simple_mp/mp_commands.list  |   3 +
> >  20 files changed, 745 insertions(+), 422 deletions(-)
> 
> Interesting series.
> So if we remove the doc addition, this patch is removing loc, so +1 from me :-).
> 
There's probably lots more LOC that can be removed beyond this, if we put
in a bit of effort looking. I just took three simple examples, and managed
to save ~100LOC each.

However, the big benefit I find is for anyone working on new apps - using a
script makes it SOOOO  much easier to add a commandline and then
subsequently to extend with new commands as you go! Just add the new command to
the list file, and write the new callback function and you're done.

Since cmdline library is not going to be going anywhere, anytime soon,
since it's so integrated into testpmd and dpdk-test, we might as well make
it useful for others.

/Bruce

[PATCH v4 0/7] document and simplify use of cmdline

[Bruce Richardson]

The DPDK commandline library is widely used by apps and examples within
DPDK, but it is not documented in our programmers guide and it requires
a lot of boilerplate code definitions in order to used. We can improve
this situation by creating a simple python script to automatically
generate the boilerplate from a list of commands.

This patchset contains a new documentation chapter on cmdline library,
going through step-by-step how to add commands and create the necessary
token lists and parse contexts.

Following that initial doc patch, the set then contains a
boilerplate-generating script, as well as a set of four patches showing
its use, by converting four examples to use the script instead of
having the hard-coded boilerplate. Once the script is used, adding a new
command becomes as simple as adding the desired command to the .list
file, and then writing the required function which will be called for
that command. No other boilerplate coding is necessary.

Script obviously does not cover the full range of capabilities of the
commandline lib, but does cover the most used parts. The code-saving to
each of the examples by auto-generating the boilerplate is significant,
and probably more examples with commandlines can be converted over in
future.

The "cmdline" example itself, is not converted over, as it should
probably remain as a simple example of direct library use without the
script.

V4:
* Reworked script following feedback from Robin J.
   - formatted code with black
   - used multi-line strings
   - replaced sys.exit(1) with proper error handling
   - per-command function returns lists rather than printing directly
   - other small cleanups
* Added change to CI script so the cmdline script is in PATH
* Converted VDPA example to script, saving another 100 LOC

V3:
* Added lots of documentation
* Added support for help text for each command
* Cleaned up script a little so it passes pycodestyle and most flake8
  checks, when line-length is set to max 100.
* Removed RFC tag, as I consider this patchset stable enough for
  consideration in a release.

V2-RFC:
* Add support for IP addresses in commands
* Move to buildtools directory and make installable
* Convert 3 examples to use script, and eliminate their boilerplate


Bruce Richardson (7):
  doc/prog_guide: new chapter on cmdline library
  buildtools: script to generate cmdline boilerplate
  ci: allow use of DPDK tools when building examples
  examples/simple_mp: auto-generate cmdline boilerplate
  examples/hotplug_mp: auto-generate cmdline boilerplate
  examples/bond: auto-generate cmdline boilerplate
  examples/vdpa: auto-generate cmdline boilerplate

 .ci/linux-build.sh                            |   1 +
 app/test/commands.c                           |   2 +
 buildtools/dpdk-cmdline-gen.py                | 190 +++++++
 buildtools/meson.build                        |   7 +
 doc/guides/prog_guide/cmdline.rst             | 466 ++++++++++++++++++
 doc/guides/prog_guide/index.rst               |   1 +
 examples/bond/Makefile                        |  12 +-
 examples/bond/commands.list                   |   6 +
 examples/bond/main.c                          | 161 +-----
 examples/bond/main.h                          |  10 -
 examples/bond/meson.build                     |   8 +
 examples/multi_process/hotplug_mp/Makefile    |  12 +-
 examples/multi_process/hotplug_mp/commands.c  | 147 +-----
 examples/multi_process/hotplug_mp/commands.h  |  10 -
 .../multi_process/hotplug_mp/commands.list    |   5 +
 examples/multi_process/hotplug_mp/meson.build |   9 +
 examples/multi_process/simple_mp/Makefile     |  12 +-
 examples/multi_process/simple_mp/meson.build  |   9 +
 .../multi_process/simple_mp/mp_commands.c     | 106 +---
 .../multi_process/simple_mp/mp_commands.h     |  14 -
 .../multi_process/simple_mp/mp_commands.list  |   3 +
 examples/vdpa/Makefile                        |  12 +-
 examples/vdpa/commands.list                   |   5 +
 examples/vdpa/main.c                          | 131 +----
 examples/vdpa/meson.build                     |   7 +
 25 files changed, 796 insertions(+), 550 deletions(-)
 create mode 100755 buildtools/dpdk-cmdline-gen.py
 create mode 100644 doc/guides/prog_guide/cmdline.rst
 create mode 100644 examples/bond/commands.list
 delete mode 100644 examples/bond/main.h
 delete mode 100644 examples/multi_process/hotplug_mp/commands.h
 create mode 100644 examples/multi_process/hotplug_mp/commands.list
 delete mode 100644 examples/multi_process/simple_mp/mp_commands.h
 create mode 100644 examples/multi_process/simple_mp/mp_commands.list
 create mode 100644 examples/vdpa/commands.list

--
2.39.2

[PATCH v4 1/7] doc/prog_guide: new chapter on cmdline library

[Bruce Richardson]

The cmdline library was not documented in our programmers guide, so add
a new chapter on it. This chapter covers step-by-step how to use the
library, rather than focusing on the library internals. This complements
the existing cmdline example app document, providing more details on the
process of using the library.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 app/test/commands.c               |   2 +
 doc/guides/prog_guide/cmdline.rst | 337 ++++++++++++++++++++++++++++++
 doc/guides/prog_guide/index.rst   |   1 +
 3 files changed, 340 insertions(+)
 create mode 100644 doc/guides/prog_guide/cmdline.rst

diff --git a/app/test/commands.c b/app/test/commands.c
index 31259e5c21..497d8e9952 100644
--- a/app/test/commands.c
+++ b/app/test/commands.c
@@ -108,6 +108,7 @@ dump_struct_sizes(void)
 #undef DUMP_SIZE
 }
 
+/* Add the dump_* tests cases 8< */
 static void cmd_dump_parsed(void *parsed_result,
 			    __rte_unused struct cmdline *cl,
 			    __rte_unused void *data)
@@ -155,6 +156,7 @@ cmdline_parse_inst_t cmd_dump = {
 		NULL,
 	},
 };
+/* >8 End of add the dump_* tests cases */
 
 /****************/
 
diff --git a/doc/guides/prog_guide/cmdline.rst b/doc/guides/prog_guide/cmdline.rst
new file mode 100644
index 0000000000..40f49a30cc
--- /dev/null
+++ b/doc/guides/prog_guide/cmdline.rst
@@ -0,0 +1,337 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2023 Intel Corporation.
+
+Command-line Library
+====================
+
+Since its earliest versions, DPDK has included a command-line library -
+primarily for internal use by, for example, ``dpdk-testpmd`` and the ``dpdk-test`` binaries,
+but the library is also exported on install and can be used by any end application.
+This chapter covers the basics of the command-line library and how to use it in an application.
+
+Library Features
+----------------
+
+The DPDK command-line library supports the following features:
+
+* Tab-completion available for interactive terminal sessions
+
+* Ability to read and process commands taken from an input file, e.g. startup script
+
+* Parameterized commands able to take multiple parameters with different datatypes:
+
+   * Strings
+   * Signed/unsigned 16/32/64-bit integers
+   * IP Addresses
+   * Ethernet Addresses
+
+* Ability to multiplex multiple commands to a single callback function
+
+Adding Command-line to an Application
+-------------------------------------
+
+Adding a command-line instance to an application involves a number of coding steps.
+
+1. Define the result structure for the command, specifying the command parameters
+
+2. Provide an initializer for each field in the result
+
+3. Define the callback function for the command
+
+4. Provide a parse result structure instance for the command, linking the callback to the command
+
+5. Add the parse result structure to a command-line context
+
+6. Within your main application code, create a new command-line instance passing in the context.
+
+The next few subsections will cover each of these steps in more detail,
+working through an example to add two commands to a command-line instance.
+Those two commands will be:
+
+1. ``quit`` - as the name suggests, to close the application
+
+2. ``show port stats <n>`` - to display on-screen the statistics for a given ethernet port
+
+.. note::
+
+   For further examples of use of the command-line, see
+   :doc:`cmdline example application <../sample_app_ug/cmd_line>`
+
+Defining Command Result Structure
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The first structure to be defined is the structure which will be created on successful parse of a command.
+This structure contains one member field for each token, or word, in the command.
+The simplest case is for a one-word command, like ``quit``.
+For this, we only need to define a structure with a single string parameter to contain that word.
+
+.. code-block:: c
+
+   struct cmd_quit_result {
+              cmdline_fixed_string_t quit;
+   };
+
+For readability, the name of the struct member should match that of the token in the command.
+
+For our second command, we need a structure with four member fields in it,
+as there are four words/tokens in our command.
+The first three are strings, and the final one is a 16-bit numeric value.
+The resulting struct looks like:
+
+.. code-block:: c
+
+   struct cmd_show_port_stats_result {
+      cmdline_fixed_string_t show;
+      cmdline_fixed_string_t port;
+      cmdline_fixed_string_t stats;
+      uint16_t n;
+   };
+
+As before, we choose names to match the tokens in the command.
+Since our numeric parameter is a 16-bit value, we use ``uint16_t`` type for it.
+Any of the standard sized integer types can be used as parameters, depending on the desired result.
+
+Beyond the standard integer types,
+the library also allows variable parameters to be of a number of other types,
+as called out in the feature list above.
+
+* For variable string parameters,
+  the type should be ``cmdline_fixed_string_t`` - the same as for fixed tokens,
+  but these will be initialized differently (as described below).
+
+* For ethernet addresses use type ``struct rte_ether_addr``
+
+* For IP addresses use type ``cmdline_ipaddr_t``
+
+Providing Field Initializers
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Each field of our result structure needs an initializer.
+For fixed string tokens, like "quit", "show" and "port", the initializer will be the string itself.
+
+.. code-block:: c
+
+   static cmdline_parse_token_string_t cmd_quit_quit_tok =
+      TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit");
+
+The convention for naming used here is to include the base name of the overall result structure -
+``cmd_quit`` in this case,
+as well as the name of the field within that structure - ``quit`` in this case, followed by ``_tok``.
+(This is why there is a double ``quit`` in the name above).
+
+This naming convention is seen in our second example,
+which also demonstrates how to define a numeric initializer.
+
+
+.. code-block:: c
+
+   static cmdline_parse_token_string_t cmd_show_port_stats_show_tok =
+      TOKEN_STRING_INITIALIZER(struct cmd_show_port_stats_result, show, "show");
+   static cmdline_parse_token_string_t cmd_show_port_stats_port_tok =
+      TOKEN_STRING_INITIALIZER(struct cmd_show_port_stats_result, port, "port");
+   static cmdline_parse_token_string_t cmd_show_port_stats_stats_tok =
+      TOKEN_STRING_INITIALIZER(struct cmd_show_port_stats_result, stats, "stats");
+   static cmdline_parse_token_num_t cmd_show_port_stats_n_tok =
+      TOKEN_NUM_INITIALIZER(struct cmd_show_port_stats_result, n, RTE_UINT16);
+
+For variable string tokens, the same ``TOKEN_STRING_INITIALIZER`` macro should be used.
+However, the final parameter should be ``NULL`` rather than a hard-coded token string.
+
+For numeric parameters, the final parameter to the ``TOKEN_NUM_INITIALIZER`` macro should be the
+cmdline type matching the variable type defined in the result structure,
+e.g. RTE_UINT8, RTE_UINT32, etc.
+
+For IP addresses, the macro ``TOKEN_IPADDR_INITIALIZER`` should be used.
+
+For ethernet addresses, the macro ``TOKEN_ETHERADDR_INITIALIZER`` should be used.
+
+Defining Callback Function
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+For each command, we need to define a function to be called once the command has been recognised.
+The callback function should have type:
+
+.. code:: c
+
+   void (*f)(void *, struct cmdline *, void *)
+
+where the first parameter is a pointer to the result structure defined above,
+the second parameter is the command-line instance,
+and the final parameter is a user-defined pointer provided when we associate the callback with the command.
+Most callback functions only use the first parameter, or none at all,
+but the additional two parameters provide some extra flexibility,
+to allow the callback to work with non-global state in your application.
+
+For our two example commands, the relevant callback functions would look very similar in definition.
+However, within the function body,
+we assume that the user would need to reference the result structure to extract the port number in
+the second case.
+
+.. code:: c
+
+   void
+   cmd_quit_parsed(void *parsed_result, struct cmdline *cl, void *data)
+   {
+      quit = 1;
+   }
+   void
+   cmd_show_port_stats_parsed(void *parsed_result, struct cmdline *cl, void *data)
+   {
+      struct cmd_show_port_stats_result *res = parsed_result;
+      uint16_t port_id = res->n;
+      ...
+   }
+
+
+Associating Callback and Command
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The ``cmdline_parse_inst_t`` type defines a "parse instance",
+i.e. a sequence of tokens to be matched and then an associated function to be called.
+Also included in the instance type are a field for help text for the command,
+and any additional user-defined parameter to be passed to the callback functions referenced above.
+For example, for our simple "quit" command:
+
+.. code-block:: c
+
+   static cmdline_parse_inst_t cmd_quit = {
+       .f = cmd_quit_parsed,
+       .data = NULL,
+       .help_str = "Close the application",
+       .tokens = {
+           (void *)&cmd_quit_quit_tok,
+           NULL
+       }
+   };
+
+In this case, we firstly identify the callback function to be called,
+then set the user-defined parameter to NULL,
+provide a help message to be given, on request, to the user explaining the command,
+before finally listing out the single token to be matched for this command instance.
+
+For our second, port stats, example,
+as well as making things a little more complicated by having multiple tokens to be matched,
+we can also demonstrate passing in a parameter to the function.
+Let us suppose that our application does not always use all the ports available to it,
+but instead only uses a subset of the ports, stored in an array called ``active_ports``.
+Our stats command, therefore, should only display stats for the currently in-use ports,
+so we pass this ``active_ports`` array.
+(For simplicity of illustration, we shall assume that the array uses a terminating marker,
+e.g. -1 for the end of the port list, so we don't need to pass in a length parameter too.)
+
+.. code-block:: c
+
+   extern int16_t active_ports[];
+   ...
+   static cmdline_parse_inst_t cmd_show_port_stats = {
+       .f = cmd_show_port_stats_parsed,
+       .data = active_ports,
+       .help_str = "Show statistics for active network ports",
+       .tokens = {
+           (void *)&cmd_show_port_stats_show_tok,
+           (void *)&cmd_show_port_stats_port_tok,
+           (void *)&cmd_show_port_stats_stats_tok,
+           (void *)&cmd_show_port_stats_n_tok,
+           NULL
+       }
+   };
+
+
+Adding Command to Command-line Context
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Now that we have configured each individual command and callback,
+we need to merge these into a single array of command-line "contexts".
+This context array will be used to create the actual command-line instance in the application.
+Thankfully, each context entry is the same as each parse instance,
+so our array is defined by simply listing out the previously defined command parse instances.
+
+.. code-block:: c
+
+   static cmdline_parse_ctx_t ctx[] = {
+       &cmd_quit,
+       &cmd_show_port_stats,
+       NULL
+   };
+
+The context list must be terminated by a NULL entry.
+
+Creating a Command-line Instance
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Once we have our ``ctx`` variable defined,
+we now just need to call the API to create the new command-line instance in our application.
+The basic API is ``cmdline_new`` which will create an interactive command-line with all commands available.
+However, if additional features for interactive use - such as tab-completion -
+are desired, it is recommended that ``cmdline_new_stdin`` be used instead.
+
+A pattern that can be used in applications is to use ``cmdline_new`` for processing any startup commands,
+either from file or from the environment (as is done in the "dpdk-test" application),
+and then using ``cmdline_stdin_new`` thereafter to handle the interactive part.
+For example, to handle a startup file and then provide an interactive prompt:
+
+.. code-block:: c
+
+   struct cmdline *cl;
+   int fd = open(startup_file, O_RDONLY);
+
+   if (fd >= 0) {
+       cl = cmdline_new(ctx, "", fd, STDOUT_FILENO);
+       if (cl == NULL) {
+           /* error handling */
+       }
+       cmdline_interact(cl);
+       cmdline_quit(cl);
+       close(fd);
+   }
+
+   cl = cmdline_stdin_new(ctx, "Proxy>> ");
+   if (cl == NULL) {
+       /* error handling */
+   }
+   cmdline_interact(cl);
+   cmdline_stdin_exit(cl);
+
+
+Multiplexing Multiple Commands to a Single Function
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To reduce the amount of boiler-plate code needed when creating a command-line for an application,
+it is possible to merge a number of commands together to have them call a separate function.
+This can be done in a number of different ways:
+
+* A callback function can be used as the target for a number of different commands.
+  Which command was used for entry to the function can be determined by examining the first parameter,
+  ``parsed_result`` in our examples above.
+
+* For simple string commands, multiple options can be concatenated using the "#" character.
+  For example: ``exit#quit``, specified as a token initializer,
+  will match either on the string "exit" or the string "quit".
+
+As a concrete example,
+these two techniques are used in the DPDK unit test application ``dpdk-test``,
+where a single command ``cmdline_parse_t`` instance is used for all the "dump_<item>" test cases.
+
+.. literalinclude:: ../../../app/test/commands.c
+    :language: c
+    :start-after: Add the dump_* tests cases 8<
+    :end-before: >8 End of add the dump_* tests cases
+
+
+Examples of Command-line Use in DPDK
+------------------------------------
+
+To help the user follow the steps provided above,
+the following DPDK files can be consulted for examples of command-line use.
+
+.. note::
+
+   This is not an exhaustive list of examples of command-line use in DPDK.
+   It is simply a list of a few files that may be of use to the application developer.
+   Some of these referenced files contain more complex examples of use that others.
+
+* ``commands.c/.h`` in ``examples/cmdline``
+
+* ``mp_commands.c/.h`` in ``examples/multi_process/simple_mp``
+
+* ``commands.c`` in ``app/test``
diff --git a/doc/guides/prog_guide/index.rst b/doc/guides/prog_guide/index.rst
index e517f0e259..94964357ff 100644
--- a/doc/guides/prog_guide/index.rst
+++ b/doc/guides/prog_guide/index.rst
@@ -13,6 +13,7 @@ Programmer's Guide
     source_org
     env_abstraction_layer
     log_lib
+    cmdline
     service_cores
     trace_lib
     rcu_lib
-- 
2.39.2

[PATCH v4 2/7] buildtools: script to generate cmdline boilerplate

[Bruce Richardson]

Provide a "dpdk-cmdline-gen.py" script for application developers to
quickly generate the boilerplate code necessary for using the cmdline
library.

Example of use:
The script takes an input file with a list of commands the user wants in
the app, where the parameter variables are tagged with the type.
For example:

	$ cat commands.list
	list
	add <UINT16>x <UINT16>y
	echo <STRING>message
	add socket <STRING>path
	quit

When run through the script as "./dpdk-cmdline-gen.py commands.list",
the output will be the contents of a header file with all the
boilerplate necessary for a commandline instance with those commands.

If the flag --stubs is passed, an output header filename must also be
passed, in which case both a header file with the definitions and a C
file with function stubs in it is written to disk. The separation is so
that the header file can be rewritten at any future point to add more
commands, while the C file can be kept as-is and extended by the user
with any additional functions needed.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 buildtools/dpdk-cmdline-gen.py    | 190 ++++++++++++++++++++++++++++++
 buildtools/meson.build            |   7 ++
 doc/guides/prog_guide/cmdline.rst | 131 +++++++++++++++++++-
 3 files changed, 327 insertions(+), 1 deletion(-)
 create mode 100755 buildtools/dpdk-cmdline-gen.py

diff --git a/buildtools/dpdk-cmdline-gen.py b/buildtools/dpdk-cmdline-gen.py
new file mode 100755
index 0000000000..6cb7610de4
--- /dev/null
+++ b/buildtools/dpdk-cmdline-gen.py
@@ -0,0 +1,190 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2023 Intel Corporation
+#
+"""
+Script to automatically generate boilerplate for using DPDK cmdline library.
+"""
+
+import argparse
+import sys
+
+PARSE_FN_PARAMS = "void *parsed_result, struct cmdline *cl, void *data"
+PARSE_FN_BODY = """
+    /* TODO: command action */
+    RTE_SET_USED(parsed_result);
+    RTE_SET_USED(cl);
+    RTE_SET_USED(data);
+"""
+NUMERIC_TYPES = [
+    "UINT8",
+    "UINT16",
+    "UINT32",
+    "UINT64",
+    "INT8",
+    "INT16",
+    "INT32",
+    "INT64",
+]
+
+
+def process_command(lineno, tokens, comment):
+    """Generate the structures and definitions for a single command."""
+    out = []
+    cfile_out = []
+
+    if tokens[0].startswith("<"):
+        raise ValueError(f"Error line {lineno + 1}: command must start with a literal string")
+
+    name_tokens = []
+    for t in tokens:
+        if t.startswith("<"):
+            break
+        name_tokens.append(t)
+    name = "_".join(name_tokens)
+
+    result_struct = []
+    initializers = []
+    token_list = []
+    for t in tokens:
+        if t.startswith("<"):
+            t_type, t_name = t[1:].split(">")
+            t_val = "NULL"
+        else:
+            t_type = "STRING"
+            t_name = t
+            t_val = f'"{t}"'
+
+        if t_type == "STRING":
+            result_struct.append(f"\tcmdline_fixed_string_t {t_name};")
+            initializers.append(
+                f"static cmdline_parse_token_string_t cmd_{name}_{t_name}_tok =\n"
+                + f"\tTOKEN_STRING_INITIALIZER(struct cmd_{name}_result, {t_name}, {t_val});"
+            )
+        elif t_type in NUMERIC_TYPES:
+            result_struct.append(f"\t{t_type.lower()}_t {t_name};")
+            initializers.append(
+                f"static cmdline_parse_token_num_t cmd_{name}_{t_name}_tok =\n"
+                + f"\tTOKEN_NUM_INITIALIZER(struct cmd_{name}_result, {t_name}, RTE_{t_type});"
+            )
+        elif t_type in ["IP", "IP_ADDR", "IPADDR"]:
+            result_struct.append(f"\tcmdline_ipaddr_t {t_name};")
+            initializers.append(
+                f"cmdline_parse_token_ipaddr_t cmd_{name}_{t_name}_tok =\n"
+                + f"\tTOKEN_IPV4_INITIALIZER(struct cmd_{name}_result, {t_name});"
+            )
+        else:
+            raise TypeError(f"Error line {lineno + 1}: unknown token type '{t_type}'")
+        token_list.append(f"cmd_{name}_{t_name}_tok")
+
+    out.append(f'/* Auto-generated handling for command "{" ".join(tokens)}" */')
+    # output function prototype
+    func_sig = f"void\ncmd_{name}_parsed({PARSE_FN_PARAMS})"
+    out.append(f"extern {func_sig};\n")
+    # output result data structure
+    out.append(f"struct cmd_{name}_result {{\n" + "\n".join(result_struct) + "\n};\n")
+    # output the initializer tokens
+    out.append("\n".join(initializers) + "\n")
+    # output the instance structure
+    out.append(
+        f"static cmdline_parse_inst_t cmd_{name} = {{\n"
+        + f"\t.f = cmd_{name}_parsed,\n"
+        + "\t.data = NULL,\n"
+        + f'\t.help_str = "{comment}",\n'
+        + "\t.tokens = {"
+    )
+    for t in token_list:
+        out.append(f"\t\t(void *)&{t},")
+    out.append("\t\tNULL\n" + "\t}\n" + "};\n")
+    # output function template if C file being written
+    cfile_out.append(f"{func_sig}\n{{{PARSE_FN_BODY}}}\n")
+
+    # return the instance structure name
+    return (f"cmd_{name}", out, cfile_out)
+
+
+def process_commands(infile, hfile, cfile, ctxname):
+    """Generate boilerplate output for a list of commands from infile."""
+    instances = []
+
+    hfile.write(
+        f"""/* File autogenerated by {sys.argv[0]} */
+#ifndef GENERATED_COMMANDS_H
+#define GENERATED_COMMANDS_H
+#include <rte_common.h>
+#include <cmdline.h>
+#include <cmdline_parse_string.h>
+#include <cmdline_parse_num.h>
+#include <cmdline_parse_ipaddr.h>
+
+"""
+    )
+
+    for lineno, line in enumerate(infile.readlines()):
+        if line.lstrip().startswith("#"):
+            continue
+        if "#" not in line:
+            line = line + "#"  # ensure split always works, even if no help text
+        tokens, comment = line.split("#", 1)
+        cmd_inst, h_out, c_out = process_command(lineno, tokens.strip().split(), comment.strip())
+        hfile.write("\n".join(h_out))
+        if cfile:
+            cfile.write("\n".join(c_out))
+        instances.append(cmd_inst)
+
+    inst_join_str = ",\n\t&"
+    hfile.write(
+        f"""
+static __rte_used cmdline_parse_ctx_t {ctxname}[] = {{
+\t&{inst_join_str.join(instances)},
+\tNULL
+}};
+
+#endif /* GENERATED_COMMANDS_H */
+"""
+    )
+
+
+def main():
+    """Application main entry point."""
+    ap = argparse.ArgumentParser(description=__doc__)
+    ap.add_argument(
+        "--stubs",
+        action="store_true",
+        help="Produce C file with empty function stubs for each command",
+    )
+    ap.add_argument(
+        "--output-file",
+        "-o",
+        default="-",
+        help="Output header filename [default to stdout]",
+    )
+    ap.add_argument(
+        "--context-name",
+        default="ctx",
+        help="Name given to the cmdline context variable in the output header [default=ctx]",
+    )
+    ap.add_argument("infile", type=argparse.FileType("r"), help="File with list of commands")
+    args = ap.parse_args()
+
+    if not args.stubs:
+        if args.output_file == "-":
+            process_commands(args.infile, sys.stdout, None, args.context_name)
+        else:
+            with open(args.output_file, "w") as hfile:
+                process_commands(args.infile, hfile, None, args.context_name)
+    else:
+        if not args.output_file.endswith(".h"):
+            ap.error(
+                "-o/--output-file: specify an output filename ending with .h when creating stubs"
+            )
+
+        cfilename = args.output_file[:-2] + ".c"
+        with open(args.output_file, "w") as hfile:
+            with open(cfilename, "w") as cfile:
+                cfile.write(f'#include "{args.output_file}"\n\n')
+                process_commands(args.infile, hfile, cfile, args.context_name)
+
+
+if __name__ == "__main__":
+    main()
diff --git a/buildtools/meson.build b/buildtools/meson.build
index 948ac17dd2..72447b60a0 100644
--- a/buildtools/meson.build
+++ b/buildtools/meson.build
@@ -19,6 +19,13 @@ get_cpu_count_cmd = py3 + files('get-cpu-count.py')
 get_numa_count_cmd = py3 + files('get-numa-count.py')
 get_test_suites_cmd = py3 + files('get-test-suites.py')
 has_hugepages_cmd = py3 + files('has-hugepages.py')
+cmdline_gen_cmd = py3 + files('dpdk-cmdline-gen.py')
+
+# install any build tools that end-users might want also
+install_data([
+            'dpdk-cmdline-gen.py',
+        ],
+        install_dir: 'bin')
 
 # select library and object file format
 pmdinfo = py3 + files('gen-pmdinfo-cfile.py') + [meson.current_build_dir()]
diff --git a/doc/guides/prog_guide/cmdline.rst b/doc/guides/prog_guide/cmdline.rst
index 40f49a30cc..0b96b770e2 100644
--- a/doc/guides/prog_guide/cmdline.rst
+++ b/doc/guides/prog_guide/cmdline.rst
@@ -44,7 +44,136 @@ Adding a command-line instance to an application involves a number of coding ste
 
 6. Within your main application code, create a new command-line instance passing in the context.
 
-The next few subsections will cover each of these steps in more detail,
+Many of these steps can be automated using the script ``dpdk-cmdline-gen.py`` installed by DPDK,
+and found in the ``buildtools`` folder in the source tree.
+This section covers adding a command-line using this script to generate the boiler plate,
+while the following section,
+`Worked Example of Adding Command-line to an Application`_ covers the steps to do so manually.
+
+Creating a Command List File
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The ``dpdk-cmdline-gen.py`` script takes as input a list of commands to be used by the application.
+While these can be piped to it via standard input, using a list file is probably best.
+
+The format of the list file must be:
+
+* Comment lines start with '#' as first non-whitespace character
+
+* One command per line
+
+* Variable fields are prefixed by the type-name in angle-brackets, for example:
+
+  * ``<STRING>message``
+
+  * ``<UINT16>port_id``
+
+  * ``<IP>src_ip``
+
+* The help text for a command is given in the form of a comment on the same line as the command
+
+An example list file, with a variety of (unrelated) commands, is shown below::
+
+   # example list file
+   list                     # show all entries
+   add <UINT16>x <UINT16>y  # add x and y
+   echo <STRING>message     # print message to screen
+   add socket <STRING>path  # add unix socket with the given path
+   quit                     # close the application
+
+Running the Generator Script
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To generate the necessary definitions for a command-line, run ``dpdk-cmdline-gen.py`` passing the list file as parameter.
+The script will output the generated C code to standard output,
+the contents of which are in the form of a C header file.
+Optionally, an output filename may be specified via the ``-o/--output-file`` argument.
+
+The generated content includes:
+
+* The result structure definitions for each command
+
+* The token initializers for each structure field
+
+* An "extern" function prototype for the callback for each command
+
+* A parse context for each command, including the per-command comments as help string
+
+* A command-line context array definition, suitable for passing to ``cmdline_new``
+
+If so desired, the script can also output function stubs for the callback functions for each command.
+This behaviour is triggered by passing the ``--stubs`` flag to the script.
+In this case, an output file must be provided with a filename ending in ".h",
+and the callback stubs will be written to an equivalent ".c" file.
+
+.. note::
+
+   The stubs are written to a separate file,
+   to allow continuous use of the script to regenerate the command-line header,
+   without overwriting any code the user has added to the callback functions.
+   This makes it easy to incrementally add new commands to an existing application.
+
+Providing the Function Callbacks
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+As discussed above, the script output is a header file, containing structure definitions,
+but the callback functions themselves obviously have to be provided by the user.
+These callback functions must be provided as non-static functions in a C file,
+and named ``cmd_<cmdname>_parsed``.
+The function prototypes can be seen in the generated output header.
+
+The "cmdname" part of the function name is built up by combining the non-variable initial tokens in the command.
+So, given the commands in our worked example below: ``quit`` and ``show port stats <n>``,
+the callback functions would be:
+
+.. code:: c
+
+   void
+   cmd_quit_parsed(void *parsed_result, struct cmdline *cl, void *data)
+   {
+        ...
+   }
+
+   void
+   cmd_show_port_stats_parsed(void *parsed_result, struct cmdline *cl, void *data)
+   {
+        ...
+   }
+
+These functions must be provided by the developer, but, as stated above,
+stub functions may be generated by the script automatically using the ``--stubs`` parameter.
+
+The same "cmdname" stem is used in the naming of the generated structures too.
+To get at the results structure for each command above,
+the ``parsed_result`` parameter should be cast to ``struct cmd_quit_result``
+or ``struct cmd_show_port_stats_result`` respectively.
+
+Integrating with the Application
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To integrate the script output with the application,
+we must ``#include`` the generated header into our applications C file,
+and then have the command-line created via either ``cmdline_new`` or ``cmdline_stdin_new``.
+The first parameter to the function call should be the context array in the generated header file,
+``ctx`` by default. (Modifiable via script parameter).
+
+The callback functions may be in this same file, or in a separate one -
+they just need to be available to the linker at build-time.
+
+Limitations of the Script Approach
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The script approach works for most commands that a user may wish to add to an application.
+However, it does not support the full range of functions possible with the DPDK command-line library.
+For example,
+it is not possible using the script to multiplex multiple commands into a single callback function.
+To use this functionality, the user should follow the instructions in the next section
+`Worked Example of Adding Command-line to an Application`_ to manually configure a command-line instance.
+
+Worked Example of Adding Command-line to an Application
+-------------------------------------------------------
+
+The next few subsections will cover each of the steps listed in `Adding Command-line to an Application`_ in more detail,
 working through an example to add two commands to a command-line instance.
 Those two commands will be:
 
-- 
2.39.2

[PATCH v4 3/7] ci: allow use of DPDK tools when building examples

[Bruce Richardson]

To allow use of the DPDK python scripts (installed in $(prefix)/bin)
from within the makefiles of our examples, we need to export the PATH
variable with the location of our installed scripts from within our CI
scripts. This matches what is already done for other paths e.g. the
PKG_CONFIG_PATH variable.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 .ci/linux-build.sh | 1 +
 1 file changed, 1 insertion(+)

diff --git a/.ci/linux-build.sh b/.ci/linux-build.sh
index e0b62bac90..3db9d9de6e 100755
--- a/.ci/linux-build.sh
+++ b/.ci/linux-build.sh
@@ -174,6 +174,7 @@ fi
 if [ "$BUILD_EXAMPLES" = "true" ]; then
     [ -d install ] || DESTDIR=$(pwd)/install meson install -C build
     export LD_LIBRARY_PATH=$(dirname $(find $(pwd)/install -name librte_eal.so)):$LD_LIBRARY_PATH
+    export PATH=$(dirname $(find $(pwd)/install -name dpdk-devbind.py)):$PATH
     export PKG_CONFIG_PATH=$(dirname $(find $(pwd)/install -name libdpdk.pc)):$PKG_CONFIG_PATH
     export PKGCONF="pkg-config --define-prefix"
     find build/examples -maxdepth 1 -type f -name "dpdk-*" |
-- 
2.39.2

Re: [PATCH v4 3/7] ci: allow use of DPDK tools when building examples

[Aaron Conole]

Bruce Richardson <bruce.richardson@intel.com> writes:

> To allow use of the DPDK python scripts (installed in $(prefix)/bin)
> from within the makefiles of our examples, we need to export the PATH
> variable with the location of our installed scripts from within our CI
> scripts. This matches what is already done for other paths e.g. the
> PKG_CONFIG_PATH variable.
>
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> ---

I guess this should work, but it's a bit strange that we need to key off
something like dpdk-devbind.py.  Not a complaint, just an observation.

Maybe a future change would be to update the meson.build to set the
exec_prefix variable - but I guess for now that might be a much bigger
change.  But it means we could get the path after setting the pkgconfig
path and then just run something like:

  pkg-config libdpdk --variable=exec_prefix

and wouldn't need to search for the python utility.

As I wrote - it's a more involved change, and I don't think it should
hold up this patch.  Just something to consider for the future (maybe
Michael or I could look at it)

>  .ci/linux-build.sh | 1 +
>  1 file changed, 1 insertion(+)
>
> diff --git a/.ci/linux-build.sh b/.ci/linux-build.sh
> index e0b62bac90..3db9d9de6e 100755
> --- a/.ci/linux-build.sh
> +++ b/.ci/linux-build.sh
> @@ -174,6 +174,7 @@ fi
>  if [ "$BUILD_EXAMPLES" = "true" ]; then
>      [ -d install ] || DESTDIR=$(pwd)/install meson install -C build
>      export LD_LIBRARY_PATH=$(dirname $(find $(pwd)/install -name librte_eal.so)):$LD_LIBRARY_PATH
> +    export PATH=$(dirname $(find $(pwd)/install -name dpdk-devbind.py)):$PATH
>      export PKG_CONFIG_PATH=$(dirname $(find $(pwd)/install -name libdpdk.pc)):$PKG_CONFIG_PATH
>      export PKGCONF="pkg-config --define-prefix"
>      find build/examples -maxdepth 1 -type f -name "dpdk-*" |

Re: [PATCH v4 3/7] ci: allow use of DPDK tools when building examples

[Bruce Richardson]

On Tue, Oct 17, 2023 at 08:24:36AM -0400, Aaron Conole wrote:
> Bruce Richardson <bruce.richardson@intel.com> writes:
> 
> > To allow use of the DPDK python scripts (installed in $(prefix)/bin)
> > from within the makefiles of our examples, we need to export the PATH
> > variable with the location of our installed scripts from within our CI
> > scripts. This matches what is already done for other paths e.g. the
> > PKG_CONFIG_PATH variable.
> >
> > Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> > ---
> 
> I guess this should work, but it's a bit strange that we need to key off
> something like dpdk-devbind.py.  Not a complaint, just an observation.
> 

Yes, I also find it a bit strange, but I decided to just copy the style
used for the other paths which also key off a "known-good" file. As you
say, it also works though, which is nice :-)

> Maybe a future change would be to update the meson.build to set the
> exec_prefix variable - but I guess for now that might be a much bigger
> change.  But it means we could get the path after setting the pkgconfig
> path and then just run something like:
> 
>   pkg-config libdpdk --variable=exec_prefix
> 
> and wouldn't need to search for the python utility.
> 
> As I wrote - it's a more involved change, and I don't think it should
> hold up this patch.  Just something to consider for the future (maybe
> Michael or I could look at it)

Thanks for the input.

/Bruce

[PATCH v4 4/7] examples/simple_mp: auto-generate cmdline boilerplate

[Bruce Richardson]

Use the dpdk-cmdline-gen script to autogenerate all the boilerplate
structs and defines for the commandline part of the app.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 examples/multi_process/simple_mp/Makefile     |  12 +-
 examples/multi_process/simple_mp/meson.build  |   9 ++
 .../multi_process/simple_mp/mp_commands.c     | 106 ++----------------
 .../multi_process/simple_mp/mp_commands.h     |  14 ---
 .../multi_process/simple_mp/mp_commands.list  |   3 +
 5 files changed, 30 insertions(+), 114 deletions(-)
 delete mode 100644 examples/multi_process/simple_mp/mp_commands.h
 create mode 100644 examples/multi_process/simple_mp/mp_commands.list

diff --git a/examples/multi_process/simple_mp/Makefile b/examples/multi_process/simple_mp/Makefile
index 1d0a260e64..890b6b7e62 100644
--- a/examples/multi_process/simple_mp/Makefile
+++ b/examples/multi_process/simple_mp/Makefile
@@ -6,6 +6,7 @@ APP = simple_mp
 
 # all source are stored in SRCS-y
 SRCS-y := main.c mp_commands.c
+SRC-DEPS := build/mp_commands.h
 
 PKGCONF ?= pkg-config
 
@@ -22,10 +23,13 @@ static: build/$(APP)-static
 	ln -sf $(APP)-static build/$(APP)
 
 PC_FILE := $(shell $(PKGCONF) --path libdpdk 2>/dev/null)
-CFLAGS += -O3 $(shell $(PKGCONF) --cflags libdpdk)
+CFLAGS += -O3 $(shell $(PKGCONF) --cflags libdpdk) -I build/
 LDFLAGS_SHARED = $(shell $(PKGCONF) --libs libdpdk)
 LDFLAGS_STATIC = $(shell $(PKGCONF) --static --libs libdpdk)
 
+build/mp_commands.h: mp_commands.list Makefile
+	dpdk-cmdline-gen.py -o $@ --context-name=simple_mp_ctx $<
+
 ifeq ($(MAKECMDGOALS),static)
 # check for broken pkg-config
 ifeq ($(shell echo $(LDFLAGS_STATIC) | grep 'whole-archive.*l:lib.*no-whole-archive'),)
@@ -36,10 +40,10 @@ endif
 
 CFLAGS += -DALLOW_EXPERIMENTAL_API
 
-build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
+build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build $(SRC-DEPS)
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
 
-build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build
+build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build $(SRC-DEPS)
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_STATIC)
 
 build:
@@ -47,5 +51,5 @@ build:
 
 .PHONY: clean
 clean:
-	rm -f build/$(APP) build/$(APP)-static build/$(APP)-shared
+	rm -f build/$(APP) build/$(APP)-static build/$(APP)-shared build/mp_commands.h
 	test -d build && rmdir -p build || true
diff --git a/examples/multi_process/simple_mp/meson.build b/examples/multi_process/simple_mp/meson.build
index 359af4384d..e99b7a3f6f 100644
--- a/examples/multi_process/simple_mp/meson.build
+++ b/examples/multi_process/simple_mp/meson.build
@@ -7,7 +7,16 @@
 # DPDK instance, use 'make'
 
 allow_experimental_apis = true
+
+cmd_h = custom_target('commands_hdr',
+	output: 'mp_commands.h',
+	input: files('mp_commands.list'),
+	capture: true,
+	command: [cmdline_gen_cmd, '--context-name=simple_mp_ctx', '@INPUT@']
+)
+
 sources = files(
         'mp_commands.c',
         'main.c',
 )
+sources += cmd_h
diff --git a/examples/multi_process/simple_mp/mp_commands.c b/examples/multi_process/simple_mp/mp_commands.c
index a5f91b00be..df9fa94208 100644
--- a/examples/multi_process/simple_mp/mp_commands.c
+++ b/examples/multi_process/simple_mp/mp_commands.c
@@ -1,44 +1,18 @@
 /* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2010-2014 Intel Corporation
+ * Copyright(c) 2010-2023 Intel Corporation
  */
-#include <stdint.h>
-#include <string.h>
-#include <stdlib.h>
-#include <stdarg.h>
-#include <inttypes.h>
-#include <stdio.h>
-#include <termios.h>
-#include <errno.h>
-#include <sys/queue.h>
-
-#include <rte_common.h>
-#include <rte_memory.h>
-#include <rte_eal.h>
-#include <rte_branch_prediction.h>
-#include <rte_launch.h>
-#include <rte_log.h>
-#include <rte_per_lcore.h>
-#include <rte_lcore.h>
 #include <rte_ring.h>
-#include <rte_debug.h>
 #include <rte_mempool.h>
 #include <rte_string_fns.h>
 
-#include <cmdline_rdline.h>
-#include <cmdline_parse.h>
-#include <cmdline_parse_string.h>
-#include <cmdline_socket.h>
-#include <cmdline.h>
 #include "mp_commands.h"
 
-/**********************************************************/
-
-struct cmd_send_result {
-	cmdline_fixed_string_t action;
-	cmdline_fixed_string_t message;
-};
+extern struct rte_ring *send_ring, *recv_ring;
+extern struct rte_mempool *message_pool;
+extern volatile int quit;
 
-static void cmd_send_parsed(void *parsed_result,
+void
+cmd_send_parsed(void *parsed_result,
 		__rte_unused struct cmdline *cl,
 		__rte_unused void *data)
 {
@@ -54,29 +28,8 @@ static void cmd_send_parsed(void *parsed_result,
 	}
 }
 
-cmdline_parse_token_string_t cmd_send_action =
-	TOKEN_STRING_INITIALIZER(struct cmd_send_result, action, "send");
-cmdline_parse_token_string_t cmd_send_message =
-	TOKEN_STRING_INITIALIZER(struct cmd_send_result, message, NULL);
-
-cmdline_parse_inst_t cmd_send = {
-	.f = cmd_send_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "send a string to another process",
-	.tokens = {        /* token list, NULL terminated */
-			(void *)&cmd_send_action,
-			(void *)&cmd_send_message,
-			NULL,
-	},
-};
-
-/**********************************************************/
-
-struct cmd_quit_result {
-	cmdline_fixed_string_t quit;
-};
-
-static void cmd_quit_parsed(__rte_unused void *parsed_result,
+void
+cmd_quit_parsed(__rte_unused void *parsed_result,
 			    struct cmdline *cl,
 			    __rte_unused void *data)
 {
@@ -84,26 +37,8 @@ static void cmd_quit_parsed(__rte_unused void *parsed_result,
 	cmdline_quit(cl);
 }
 
-cmdline_parse_token_string_t cmd_quit_quit =
-	TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit");
-
-cmdline_parse_inst_t cmd_quit = {
-	.f = cmd_quit_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "close the application",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_quit_quit,
-		NULL,
-	},
-};
-
-/**********************************************************/
-
-struct cmd_help_result {
-	cmdline_fixed_string_t help;
-};
-
-static void cmd_help_parsed(__rte_unused void *parsed_result,
+void
+cmd_help_parsed(__rte_unused void *parsed_result,
 			    struct cmdline *cl,
 			    __rte_unused void *data)
 {
@@ -112,24 +47,3 @@ static void cmd_help_parsed(__rte_unused void *parsed_result,
 			"send commands to the simple app. Commands supported are:\n\n"
 			"- send [string]\n" "- help\n" "- quit\n\n");
 }
-
-cmdline_parse_token_string_t cmd_help_help =
-	TOKEN_STRING_INITIALIZER(struct cmd_help_result, help, "help");
-
-cmdline_parse_inst_t cmd_help = {
-	.f = cmd_help_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "show help",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_help_help,
-		NULL,
-	},
-};
-
-/****** CONTEXT (list of instruction) */
-cmdline_parse_ctx_t simple_mp_ctx[] = {
-		(cmdline_parse_inst_t *)&cmd_send,
-		(cmdline_parse_inst_t *)&cmd_quit,
-		(cmdline_parse_inst_t *)&cmd_help,
-	NULL,
-};
diff --git a/examples/multi_process/simple_mp/mp_commands.h b/examples/multi_process/simple_mp/mp_commands.h
deleted file mode 100644
index 5d67413e7c..0000000000
--- a/examples/multi_process/simple_mp/mp_commands.h
+++ /dev/null
@@ -1,14 +0,0 @@
-/* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2010-2014 Intel Corporation
- */
-
-#ifndef _SIMPLE_MP_COMMANDS_H_
-#define _SIMPLE_MP_COMMANDS_H_
-
-extern struct rte_ring *send_ring;
-extern struct rte_mempool *message_pool;
-extern volatile int quit;
-
-extern cmdline_parse_ctx_t simple_mp_ctx[];
-
-#endif /* _SIMPLE_MP_COMMANDS_H_ */
diff --git a/examples/multi_process/simple_mp/mp_commands.list b/examples/multi_process/simple_mp/mp_commands.list
new file mode 100644
index 0000000000..c8890cb071
--- /dev/null
+++ b/examples/multi_process/simple_mp/mp_commands.list
@@ -0,0 +1,3 @@
+send <STRING>message  # send a string to another process
+help                  # show help
+quit                  # close the application
-- 
2.39.2

[PATCH v4 5/7] examples/hotplug_mp: auto-generate cmdline boilerplate

[Bruce Richardson]

Use the dpdk-cmdline-gen script to autogenerate all the boilerplate
structs and defines for the commandline part of the app.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 examples/multi_process/hotplug_mp/Makefile    |  12 +-
 examples/multi_process/hotplug_mp/commands.c  | 147 ++----------------
 examples/multi_process/hotplug_mp/commands.h  |  10 --
 .../multi_process/hotplug_mp/commands.list    |   5 +
 examples/multi_process/hotplug_mp/meson.build |   9 ++
 5 files changed, 35 insertions(+), 148 deletions(-)
 delete mode 100644 examples/multi_process/hotplug_mp/commands.h
 create mode 100644 examples/multi_process/hotplug_mp/commands.list

diff --git a/examples/multi_process/hotplug_mp/Makefile b/examples/multi_process/hotplug_mp/Makefile
index 6b20d6e49a..81ee85cd6b 100644
--- a/examples/multi_process/hotplug_mp/Makefile
+++ b/examples/multi_process/hotplug_mp/Makefile
@@ -6,6 +6,7 @@ APP = hotplug_mp
 
 # all source are stored in SRCS-y
 SRCS-y := main.c commands.c
+SRC-DEPS := build/commands.h
 
 PKGCONF ?= pkg-config
 
@@ -22,10 +23,13 @@ static: build/$(APP)-static
 	ln -sf $(APP)-static build/$(APP)
 
 PC_FILE := $(shell $(PKGCONF) --path libdpdk 2>/dev/null)
-CFLAGS += -O3 $(shell $(PKGCONF) --cflags libdpdk)
+CFLAGS += -O3 $(shell $(PKGCONF) --cflags libdpdk) -I build/
 LDFLAGS_SHARED = $(shell $(PKGCONF) --libs libdpdk)
 LDFLAGS_STATIC = $(shell $(PKGCONF) --static --libs libdpdk)
 
+build/commands.h: commands.list Makefile
+	dpdk-cmdline-gen.py -o $@ --context-name=main_ctx $<
+
 ifeq ($(MAKECMDGOALS),static)
 # check for broken pkg-config
 ifeq ($(shell echo $(LDFLAGS_STATIC) | grep 'whole-archive.*l:lib.*no-whole-archive'),)
@@ -36,10 +40,10 @@ endif
 
 CFLAGS += -DALLOW_EXPERIMENTAL_API
 
-build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
+build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build $(SRC-DEPS)
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
 
-build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build
+build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build  $(SRC-DEPS)
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_STATIC)
 
 build:
@@ -47,5 +51,5 @@ build:
 
 .PHONY: clean
 clean:
-	rm -f build/$(APP) build/$(APP)-static build/$(APP)-shared
+	rm -f build/$(APP) build/$(APP)-static build/$(APP)-shared build/commands.h
 	test -d build && rmdir -p build || true
diff --git a/examples/multi_process/hotplug_mp/commands.c b/examples/multi_process/hotplug_mp/commands.c
index 88f44e00a0..900eb9f774 100644
--- a/examples/multi_process/hotplug_mp/commands.c
+++ b/examples/multi_process/hotplug_mp/commands.c
@@ -1,24 +1,12 @@
 /* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2018 Intel Corporation.
+ * Copyright(c) 2018-2023 Intel Corporation.
  */
 
-#include <cmdline_rdline.h>
-#include <cmdline_parse.h>
-#include <cmdline_parse_ipaddr.h>
-#include <cmdline_parse_num.h>
-#include <cmdline_parse_string.h>
-#include <cmdline.h>
-
 #include <rte_bus.h>
 #include <rte_ethdev.h>
+#include "commands.h"
 
-/**********************************************************/
-
-struct cmd_help_result {
-	cmdline_fixed_string_t help;
-};
-
-static void cmd_help_parsed(__rte_unused void *parsed_result,
+void cmd_help_parsed(__rte_unused void *parsed_result,
 			    struct cmdline *cl,
 			    __rte_unused void *data)
 {
@@ -29,52 +17,16 @@ static void cmd_help_parsed(__rte_unused void *parsed_result,
 		       "- list\n\n");
 }
 
-cmdline_parse_token_string_t cmd_help_help =
-	TOKEN_STRING_INITIALIZER(struct cmd_help_result, help, "help");
-
-cmdline_parse_inst_t cmd_help = {
-	.f = cmd_help_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "show help",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_help_help,
-		NULL,
-	},
-};
-
-/**********************************************************/
-
-struct cmd_quit_result {
-	cmdline_fixed_string_t quit;
-};
-
-static void cmd_quit_parsed(__rte_unused void *parsed_result,
+void
+cmd_quit_parsed(__rte_unused void *parsed_result,
 			    struct cmdline *cl,
 			    __rte_unused void *data)
 {
 	cmdline_quit(cl);
 }
 
-cmdline_parse_token_string_t cmd_quit_quit =
-	TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit");
-
-cmdline_parse_inst_t cmd_quit = {
-	.f = cmd_quit_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "quit",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_quit_quit,
-		NULL,
-	},
-};
-
-/**********************************************************/
-
-struct cmd_list_result {
-	cmdline_fixed_string_t list;
-};
-
-static void cmd_list_parsed(__rte_unused void *parsed_result,
+void
+cmd_list_parsed(__rte_unused void *parsed_result,
 			    struct cmdline *cl,
 			    __rte_unused void *data)
 {
@@ -92,31 +44,12 @@ static void cmd_list_parsed(__rte_unused void *parsed_result,
 	}
 }
 
-cmdline_parse_token_string_t cmd_list_list =
-	TOKEN_STRING_INITIALIZER(struct cmd_list_result, list, "list");
-
-cmdline_parse_inst_t cmd_list = {
-	.f = cmd_list_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "list all devices",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_list_list,
-		NULL,
-	},
-};
-
-/**********************************************************/
-
-struct cmd_dev_attach_result {
-	cmdline_fixed_string_t attach;
-	cmdline_fixed_string_t devargs;
-};
-
-static void cmd_dev_attach_parsed(void *parsed_result,
+void
+cmd_attach_parsed(void *parsed_result,
 				  struct cmdline *cl,
 				  __rte_unused void *data)
 {
-	struct cmd_dev_attach_result *res = parsed_result;
+	struct cmd_attach_result *res = parsed_result;
 	struct rte_devargs da;
 
 	memset(&da, 0, sizeof(da));
@@ -134,35 +67,12 @@ static void cmd_dev_attach_parsed(void *parsed_result,
 	rte_devargs_reset(&da);
 }
 
-cmdline_parse_token_string_t cmd_dev_attach_attach =
-	TOKEN_STRING_INITIALIZER(struct cmd_dev_attach_result, attach,
-				 "attach");
-cmdline_parse_token_string_t cmd_dev_attach_devargs =
-	TOKEN_STRING_INITIALIZER(struct cmd_dev_attach_result, devargs, NULL);
-
-cmdline_parse_inst_t cmd_attach_device = {
-	.f = cmd_dev_attach_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "attach a device",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_dev_attach_attach,
-		(void *)&cmd_dev_attach_devargs,
-		NULL,
-	},
-};
-
-/**********************************************************/
-
-struct cmd_dev_detach_result {
-	cmdline_fixed_string_t detach;
-	cmdline_fixed_string_t devargs;
-};
-
-static void cmd_dev_detach_parsed(void *parsed_result,
+void
+cmd_detach_parsed(void *parsed_result,
 				   struct cmdline *cl,
 				   __rte_unused void *data)
 {
-	struct cmd_dev_detach_result *res = parsed_result;
+	struct cmd_detach_result *res = parsed_result;
 	struct rte_devargs da;
 
 	memset(&da, 0, sizeof(da));
@@ -181,34 +91,3 @@ static void cmd_dev_detach_parsed(void *parsed_result,
 			da.name);
 	rte_devargs_reset(&da);
 }
-
-cmdline_parse_token_string_t cmd_dev_detach_detach =
-	TOKEN_STRING_INITIALIZER(struct cmd_dev_detach_result, detach,
-				 "detach");
-
-cmdline_parse_token_string_t cmd_dev_detach_devargs =
-	TOKEN_STRING_INITIALIZER(struct cmd_dev_detach_result, devargs, NULL);
-
-cmdline_parse_inst_t cmd_detach_device = {
-	.f = cmd_dev_detach_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "detach a device",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_dev_detach_detach,
-		(void *)&cmd_dev_detach_devargs,
-		NULL,
-	},
-};
-
-/**********************************************************/
-/**********************************************************/
-/****** CONTEXT (list of instruction) */
-
-cmdline_parse_ctx_t main_ctx[] = {
-	(cmdline_parse_inst_t *)&cmd_help,
-	(cmdline_parse_inst_t *)&cmd_quit,
-	(cmdline_parse_inst_t *)&cmd_list,
-	(cmdline_parse_inst_t *)&cmd_attach_device,
-	(cmdline_parse_inst_t *)&cmd_detach_device,
-	NULL,
-};
diff --git a/examples/multi_process/hotplug_mp/commands.h b/examples/multi_process/hotplug_mp/commands.h
deleted file mode 100644
index afcf177dba..0000000000
--- a/examples/multi_process/hotplug_mp/commands.h
+++ /dev/null
@@ -1,10 +0,0 @@
-/* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2018 Intel Corporation
- */
-
-#ifndef _COMMANDS_H_
-#define _COMMANDS_H_
-
-extern cmdline_parse_ctx_t main_ctx[];
-
-#endif /* _COMMANDS_H_ */
diff --git a/examples/multi_process/hotplug_mp/commands.list b/examples/multi_process/hotplug_mp/commands.list
new file mode 100644
index 0000000000..e3de503965
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/commands.list
@@ -0,0 +1,5 @@
+attach <STRING>devargs     # attach a device
+detach <STRING>devargs     # detach a device
+list                       # list all devices
+help                       # show help
+quit                       # quit
diff --git a/examples/multi_process/hotplug_mp/meson.build b/examples/multi_process/hotplug_mp/meson.build
index a1ad98ca2e..7a0e9ca47a 100644
--- a/examples/multi_process/hotplug_mp/meson.build
+++ b/examples/multi_process/hotplug_mp/meson.build
@@ -7,7 +7,16 @@
 # DPDK instance, use 'make'
 
 allow_experimental_apis = true
+
+cmd_h = custom_target('commands_hdr',
+	output: 'commands.h',
+	input: files('commands.list'),
+	capture: true,
+	command: [cmdline_gen_cmd, '--context-name=main_ctx', '@INPUT@']
+)
+
 sources = files(
         'commands.c',
         'main.c',
 )
+sources += cmd_h
-- 
2.39.2

[PATCH v4 6/7] examples/bond: auto-generate cmdline boilerplate

[Bruce Richardson]

Use the dpdk-cmdline-gen script to autogenerate all the boilerplate
structs and defines for the commandline part of the app.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>

---
Note: the original help text on some of the commands in this example
  were not useful "this command do not handle any arguments". Therefore,
  when converting over to the command script, the per-command help
  info has been updated with reference to the code rather than a literal
  transfer of the existing help text, as was done with the previous 2
  example apps.
---
 examples/bond/Makefile      |  12 ++-
 examples/bond/commands.list |   6 ++
 examples/bond/main.c        | 161 ++++--------------------------------
 examples/bond/main.h        |  10 ---
 examples/bond/meson.build   |   8 ++
 5 files changed, 37 insertions(+), 160 deletions(-)
 create mode 100644 examples/bond/commands.list
 delete mode 100644 examples/bond/main.h

diff --git a/examples/bond/Makefile b/examples/bond/Makefile
index ad711a5bee..d87c7a32ba 100644
--- a/examples/bond/Makefile
+++ b/examples/bond/Makefile
@@ -6,6 +6,7 @@ APP = bond_app
 
 # all source are stored in SRCS-y
 SRCS-y := main.c
+SRC-DEPS := build/commands.h
 
 PKGCONF ?= pkg-config
 
@@ -24,10 +25,13 @@ static: build/$(APP)-static
 LDFLAGS += -lrte_net_bond
 
 PC_FILE := $(shell $(PKGCONF) --path libdpdk 2>/dev/null)
-CFLAGS += -O3 $(shell $(PKGCONF) --cflags libdpdk)
+CFLAGS += -O3 $(shell $(PKGCONF) --cflags libdpdk) -I build/
 LDFLAGS_SHARED = $(shell $(PKGCONF) --libs libdpdk)
 LDFLAGS_STATIC = $(shell $(PKGCONF) --static --libs libdpdk)
 
+build/commands.h: commands.list Makefile
+	dpdk-cmdline-gen.py -o $@ --context-name=main_ctx $<
+
 ifeq ($(MAKECMDGOALS),static)
 # check for broken pkg-config
 ifeq ($(shell echo $(LDFLAGS_STATIC) | grep 'whole-archive.*l:lib.*no-whole-archive'),)
@@ -38,10 +42,10 @@ endif
 
 CFLAGS += -DALLOW_EXPERIMENTAL_API
 
-build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
+build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build $(SRC-DEPS)
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
 
-build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build
+build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build  $(SRC-DEPS)
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_STATIC)
 
 build:
@@ -49,5 +53,5 @@ build:
 
 .PHONY: clean
 clean:
-	rm -f build/$(APP) build/$(APP)-static build/$(APP)-shared
+	rm -f build/$(APP) build/$(APP)-static build/$(APP)-shared build/*.h
 	test -d build && rmdir -p build || true
diff --git a/examples/bond/commands.list b/examples/bond/commands.list
new file mode 100644
index 0000000000..a10bf75708
--- /dev/null
+++ b/examples/bond/commands.list
@@ -0,0 +1,6 @@
+send <IP>ip   # sends one ARPrequest through bonding for IP
+start         # starts listening if not started at startup
+stop          # stops listening
+show          # shows some bond info, e.g. active members
+help          # show help
+quit          # close application
diff --git a/examples/bond/main.c b/examples/bond/main.c
index 90f422ec11..8528abf675 100644
--- a/examples/bond/main.c
+++ b/examples/bond/main.c
@@ -45,16 +45,8 @@
 #include <rte_cpuflags.h>
 #include <rte_eth_bond.h>
 
-#include <cmdline_rdline.h>
-#include <cmdline_parse.h>
-#include <cmdline_parse_num.h>
-#include <cmdline_parse_string.h>
-#include <cmdline_parse_ipaddr.h>
-#include <cmdline_parse_etheraddr.h>
 #include <cmdline_socket.h>
-#include <cmdline.h>
-
-#include "main.h"
+#include "commands.h"
 
 #define RTE_LOGTYPE_DCB RTE_LOGTYPE_USER1
 
@@ -462,11 +454,7 @@ static int lcore_main(__rte_unused void *arg1)
 	return 0;
 }
 
-struct cmd_obj_send_result {
-	cmdline_fixed_string_t action;
-	cmdline_ipaddr_t ip;
-};
-static inline void get_string(struct cmd_obj_send_result *res, char *buf, uint8_t size)
+static inline void get_string(struct cmd_send_result *res, char *buf, uint8_t size)
 {
 	snprintf(buf, size, NIPQUAD_FMT,
 		((unsigned)((unsigned char *)&(res->ip.addr.ipv4))[0]),
@@ -475,12 +463,11 @@ static inline void get_string(struct cmd_obj_send_result *res, char *buf, uint8_
 		((unsigned)((unsigned char *)&(res->ip.addr.ipv4))[3])
 		);
 }
-static void cmd_obj_send_parsed(void *parsed_result,
-		__rte_unused struct cmdline *cl,
-			       __rte_unused void *data)
+void
+cmd_send_parsed(void *parsed_result, __rte_unused struct cmdline *cl, __rte_unused void *data)
 {
 
-	struct cmd_obj_send_result *res = parsed_result;
+	struct cmd_send_result *res = parsed_result;
 	char ip_str[INET6_ADDRSTRLEN];
 
 	struct rte_ether_addr bond_mac_addr;
@@ -544,29 +531,8 @@ static void cmd_obj_send_parsed(void *parsed_result,
 	cmdline_printf(cl, "\n");
 }
 
-cmdline_parse_token_string_t cmd_obj_action_send =
-	TOKEN_STRING_INITIALIZER(struct cmd_obj_send_result, action, "send");
-cmdline_parse_token_ipaddr_t cmd_obj_ip =
-	TOKEN_IPV4_INITIALIZER(struct cmd_obj_send_result, ip);
-
-cmdline_parse_inst_t cmd_obj_send = {
-	.f = cmd_obj_send_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "send client_ip",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_obj_action_send,
-		(void *)&cmd_obj_ip,
-		NULL,
-	},
-};
-
-struct cmd_start_result {
-	cmdline_fixed_string_t start;
-};
-
-static void cmd_start_parsed(__rte_unused void *parsed_result,
-			       struct cmdline *cl,
-			       __rte_unused void *data)
+void
+cmd_start_parsed(__rte_unused void *parsed_result, struct cmdline *cl, __rte_unused void *data)
 {
 	int worker_core_id = rte_lcore_id();
 
@@ -605,26 +571,8 @@ static void cmd_start_parsed(__rte_unused void *parsed_result,
 		);
 }
 
-cmdline_parse_token_string_t cmd_start_start =
-	TOKEN_STRING_INITIALIZER(struct cmd_start_result, start, "start");
-
-cmdline_parse_inst_t cmd_start = {
-	.f = cmd_start_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "starts listening if not started at startup",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_start_start,
-		NULL,
-	},
-};
-
-struct cmd_help_result {
-	cmdline_fixed_string_t help;
-};
-
-static void cmd_help_parsed(__rte_unused void *parsed_result,
-			    struct cmdline *cl,
-			    __rte_unused void *data)
+void
+cmd_help_parsed(__rte_unused void *parsed_result, struct cmdline *cl, __rte_unused void *data)
 {
 	cmdline_printf(cl,
 			"ALB - link bonding mode 6 example\n"
@@ -637,26 +585,8 @@ static void cmd_help_parsed(__rte_unused void *parsed_result,
 		       );
 }
 
-cmdline_parse_token_string_t cmd_help_help =
-	TOKEN_STRING_INITIALIZER(struct cmd_help_result, help, "help");
-
-cmdline_parse_inst_t cmd_help = {
-	.f = cmd_help_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "show help",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_help_help,
-		NULL,
-	},
-};
-
-struct cmd_stop_result {
-	cmdline_fixed_string_t stop;
-};
-
-static void cmd_stop_parsed(__rte_unused void *parsed_result,
-			    struct cmdline *cl,
-			    __rte_unused void *data)
+void
+cmd_stop_parsed(__rte_unused void *parsed_result, struct cmdline *cl, __rte_unused void *data)
 {
 	rte_spinlock_lock(&global_flag_stru_p->lock);
 	if (global_flag_stru_p->LcoreMainIsRunning == 0)	{
@@ -678,26 +608,8 @@ static void cmd_stop_parsed(__rte_unused void *parsed_result,
 	rte_spinlock_unlock(&global_flag_stru_p->lock);
 }
 
-cmdline_parse_token_string_t cmd_stop_stop =
-	TOKEN_STRING_INITIALIZER(struct cmd_stop_result, stop, "stop");
-
-cmdline_parse_inst_t cmd_stop = {
-	.f = cmd_stop_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "this command do not handle any arguments",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_stop_stop,
-		NULL,
-	},
-};
-
-struct cmd_quit_result {
-	cmdline_fixed_string_t quit;
-};
-
-static void cmd_quit_parsed(__rte_unused void *parsed_result,
-			    struct cmdline *cl,
-			    __rte_unused void *data)
+void
+cmd_quit_parsed(__rte_unused void *parsed_result, struct cmdline *cl, __rte_unused void *data)
 {
 	rte_spinlock_lock(&global_flag_stru_p->lock);
 	if (global_flag_stru_p->LcoreMainIsRunning == 0)	{
@@ -721,26 +633,8 @@ static void cmd_quit_parsed(__rte_unused void *parsed_result,
 	cmdline_quit(cl);
 }
 
-cmdline_parse_token_string_t cmd_quit_quit =
-	TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit");
-
-cmdline_parse_inst_t cmd_quit = {
-	.f = cmd_quit_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "this command do not handle any arguments",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_quit_quit,
-		NULL,
-	},
-};
-
-struct cmd_show_result {
-	cmdline_fixed_string_t show;
-};
-
-static void cmd_show_parsed(__rte_unused void *parsed_result,
-			    struct cmdline *cl,
-			    __rte_unused void *data)
+void
+cmd_show_parsed(__rte_unused void *parsed_result, struct cmdline *cl, __rte_unused void *data)
 {
 	uint16_t members[16] = {0};
 	uint8_t len = 16;
@@ -772,31 +666,6 @@ static void cmd_show_parsed(__rte_unused void *parsed_result,
 	rte_spinlock_unlock(&global_flag_stru_p->lock);
 }
 
-cmdline_parse_token_string_t cmd_show_show =
-	TOKEN_STRING_INITIALIZER(struct cmd_show_result, show, "show");
-
-cmdline_parse_inst_t cmd_show = {
-	.f = cmd_show_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "this command do not handle any arguments",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_show_show,
-		NULL,
-	},
-};
-
-/****** CONTEXT (list of instruction) */
-
-cmdline_parse_ctx_t main_ctx[] = {
-	(cmdline_parse_inst_t *)&cmd_start,
-	(cmdline_parse_inst_t *)&cmd_obj_send,
-	(cmdline_parse_inst_t *)&cmd_stop,
-	(cmdline_parse_inst_t *)&cmd_show,
-	(cmdline_parse_inst_t *)&cmd_quit,
-	(cmdline_parse_inst_t *)&cmd_help,
-	NULL,
-};
-
 /* prompt function, called from main on MAIN lcore */
 static void prompt(__rte_unused void *arg1)
 {
diff --git a/examples/bond/main.h b/examples/bond/main.h
deleted file mode 100644
index f91ed9c885..0000000000
--- a/examples/bond/main.h
+++ /dev/null
@@ -1,10 +0,0 @@
-/* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2010-2015 Intel Corporation
- */
-
-#ifndef _MAIN_H_
-#define _MAIN_H_
-
-int main(int argc, char *argv[]);
-
-#endif /* ifndef _MAIN_H_ */
diff --git a/examples/bond/meson.build b/examples/bond/meson.build
index ed22b7887d..bfbec04ecd 100644
--- a/examples/bond/meson.build
+++ b/examples/bond/meson.build
@@ -11,3 +11,11 @@ allow_experimental_apis = true
 sources = files(
         'main.c',
 )
+
+cmd_h = custom_target('commands_hdr',
+	output: 'commands.h',
+	input: files('commands.list'),
+	capture: true,
+	command: [cmdline_gen_cmd, '--context-name=main_ctx', '@INPUT@']
+)
+sources += cmd_h
-- 
2.39.2

[PATCH v4 7/7] examples/vdpa: auto-generate cmdline boilerplate

[Bruce Richardson]

Use the dpdk-cmdline-gen script to autogenerate all the boilerplate
structs and defines for the commandline part of the app.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 examples/vdpa/Makefile      |  12 ++--
 examples/vdpa/commands.list |   5 ++
 examples/vdpa/main.c        | 131 ++----------------------------------
 examples/vdpa/meson.build   |   7 ++
 4 files changed, 27 insertions(+), 128 deletions(-)
 create mode 100644 examples/vdpa/commands.list

diff --git a/examples/vdpa/Makefile b/examples/vdpa/Makefile
index d974db4f40..aa60a000cf 100644
--- a/examples/vdpa/Makefile
+++ b/examples/vdpa/Makefile
@@ -6,6 +6,7 @@ APP = vdpa
 
 # all source are stored in SRCS-y
 SRCS-y := main.c
+SRC-DEPS := build/commands.h
 CFLAGS += -DALLOW_EXPERIMENTAL_API
 
 PKGCONF ?= pkg-config
@@ -23,10 +24,13 @@ static: build/$(APP)-static
 	ln -sf $(APP)-static build/$(APP)
 
 PC_FILE := $(shell $(PKGCONF) --path libdpdk 2>/dev/null)
-CFLAGS += -O3 $(shell $(PKGCONF) --cflags libdpdk)
+CFLAGS += -O3 $(shell $(PKGCONF) --cflags libdpdk) -I build/
 LDFLAGS_SHARED = $(shell $(PKGCONF) --libs libdpdk)
 LDFLAGS_STATIC = $(shell $(PKGCONF) --static --libs libdpdk)
 
+build/commands.h: commands.list Makefile
+	dpdk-cmdline-gen.py -o $@ --context-name=main_ctx $<
+
 ifeq ($(MAKECMDGOALS),static)
 # check for broken pkg-config
 ifeq ($(shell echo $(LDFLAGS_STATIC) | grep 'whole-archive.*l:lib.*no-whole-archive'),)
@@ -35,10 +39,10 @@ $(error "Cannot generate statically-linked binaries with this version of pkg-con
 endif
 endif
 
-build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
+build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build $(SRC-DEPS)
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
 
-build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build
+build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build $(SRC-DEPS)
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_STATIC)
 
 build:
@@ -46,5 +50,5 @@ build:
 
 .PHONY: clean
 clean:
-	rm -f build/$(APP) build/$(APP)-static build/$(APP)-shared
+	rm -f build/$(APP) build/$(APP)-static build/$(APP)-shared build/*.h
 	test -d build && rmdir -p build || true
diff --git a/examples/vdpa/commands.list b/examples/vdpa/commands.list
new file mode 100644
index 0000000000..0d83b7c0e9
--- /dev/null
+++ b/examples/vdpa/commands.list
@@ -0,0 +1,5 @@
+help                                    # show help
+list                                    # list all available vdpa devices
+create <STRING>socket_path <STRING>bdf  # create a new vdpa port
+stats <STRING>bdf <UINT32>qid           # show device statistics
+quit                                    # exit application
diff --git a/examples/vdpa/main.c b/examples/vdpa/main.c
index 4d3203f3a7..289db26498 100644
--- a/examples/vdpa/main.c
+++ b/examples/vdpa/main.c
@@ -16,11 +16,8 @@
 #include <rte_pci.h>
 #include <rte_string_fns.h>
 
-#include <cmdline_parse.h>
 #include <cmdline_socket.h>
-#include <cmdline_parse_string.h>
-#include <cmdline_parse_num.h>
-#include <cmdline.h>
+#include "commands.h"  /* auto-generated file from commands.list */
 #include "vdpa_blk_compact.h"
 
 #define MAX_PATH_LEN 128
@@ -301,14 +298,9 @@ signal_handler(int signum)
 	}
 }
 
-/* interactive cmds */
+/* interactive cmd functions */
 
-/* *** Help command with introduction. *** */
-struct cmd_help_result {
-	cmdline_fixed_string_t help;
-};
-
-static void cmd_help_parsed(__rte_unused void *parsed_result,
+void cmd_help_parsed(__rte_unused void *parsed_result,
 		struct cmdline *cl,
 		__rte_unused void *data)
 {
@@ -325,25 +317,7 @@ static void cmd_help_parsed(__rte_unused void *parsed_result,
 	);
 }
 
-cmdline_parse_token_string_t cmd_help_help =
-	TOKEN_STRING_INITIALIZER(struct cmd_help_result, help, "help");
-
-cmdline_parse_inst_t cmd_help = {
-	.f = cmd_help_parsed,
-	.data = NULL,
-	.help_str = "show help",
-	.tokens = {
-		(void *)&cmd_help_help,
-		NULL,
-	},
-};
-
-/* *** List all available vdpa devices *** */
-struct cmd_list_result {
-	cmdline_fixed_string_t action;
-};
-
-static void cmd_list_vdpa_devices_parsed(
+void cmd_list_parsed(
 		__rte_unused void *parsed_result,
 		struct cmdline *cl,
 		__rte_unused void *data)
@@ -376,27 +350,7 @@ static void cmd_list_vdpa_devices_parsed(
 	}
 }
 
-cmdline_parse_token_string_t cmd_action_list =
-	TOKEN_STRING_INITIALIZER(struct cmd_list_result, action, "list");
-
-cmdline_parse_inst_t cmd_list_vdpa_devices = {
-	.f = cmd_list_vdpa_devices_parsed,
-	.data = NULL,
-	.help_str = "list all available vdpa devices",
-	.tokens = {
-		(void *)&cmd_action_list,
-		NULL,
-	},
-};
-
-/* *** Create new vdpa port *** */
-struct cmd_create_result {
-	cmdline_fixed_string_t action;
-	cmdline_fixed_string_t socket_path;
-	cmdline_fixed_string_t bdf;
-};
-
-static void cmd_create_vdpa_port_parsed(void *parsed_result,
+void cmd_create_parsed(void *parsed_result,
 		struct cmdline *cl,
 		__rte_unused void *data)
 {
@@ -417,33 +371,7 @@ static void cmd_create_vdpa_port_parsed(void *parsed_result,
 		devcnt++;
 }
 
-cmdline_parse_token_string_t cmd_action_create =
-	TOKEN_STRING_INITIALIZER(struct cmd_create_result, action, "create");
-cmdline_parse_token_string_t cmd_socket_path =
-	TOKEN_STRING_INITIALIZER(struct cmd_create_result, socket_path, NULL);
-cmdline_parse_token_string_t cmd_bdf =
-	TOKEN_STRING_INITIALIZER(struct cmd_create_result, bdf, NULL);
-
-cmdline_parse_inst_t cmd_create_vdpa_port = {
-	.f = cmd_create_vdpa_port_parsed,
-	.data = NULL,
-	.help_str = "create a new vdpa port",
-	.tokens = {
-		(void *)&cmd_action_create,
-		(void *)&cmd_socket_path,
-		(void *)&cmd_bdf,
-		NULL,
-	},
-};
-
-/* *** STATS *** */
-struct cmd_stats_result {
-	cmdline_fixed_string_t stats;
-	cmdline_fixed_string_t bdf;
-	uint16_t qid;
-};
-
-static void cmd_device_stats_parsed(void *parsed_result, struct cmdline *cl,
+void cmd_stats_parsed(void *parsed_result, struct cmdline *cl,
 				    __rte_unused void *data)
 {
 	struct cmd_stats_result *res = parsed_result;
@@ -525,31 +453,7 @@ static void cmd_device_stats_parsed(void *parsed_result, struct cmdline *cl,
 	}
 }
 
-cmdline_parse_token_string_t cmd_device_stats_ =
-	TOKEN_STRING_INITIALIZER(struct cmd_stats_result, stats, "stats");
-cmdline_parse_token_string_t cmd_device_bdf =
-	TOKEN_STRING_INITIALIZER(struct cmd_stats_result, bdf, NULL);
-cmdline_parse_token_num_t cmd_queue_id =
-	TOKEN_NUM_INITIALIZER(struct cmd_stats_result, qid, RTE_UINT32);
-
-cmdline_parse_inst_t cmd_device_stats = {
-	.f = cmd_device_stats_parsed,
-	.data = NULL,
-	.help_str = "stats: show device statistics",
-	.tokens = {
-		(void *)&cmd_device_stats_,
-		(void *)&cmd_device_bdf,
-		(void *)&cmd_queue_id,
-		NULL,
-	},
-};
-
-/* *** QUIT *** */
-struct cmd_quit_result {
-	cmdline_fixed_string_t quit;
-};
-
-static void cmd_quit_parsed(__rte_unused void *parsed_result,
+void cmd_quit_parsed(__rte_unused void *parsed_result,
 		struct cmdline *cl,
 		__rte_unused void *data)
 {
@@ -557,27 +461,6 @@ static void cmd_quit_parsed(__rte_unused void *parsed_result,
 	cmdline_quit(cl);
 }
 
-cmdline_parse_token_string_t cmd_quit_quit =
-	TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit");
-
-cmdline_parse_inst_t cmd_quit = {
-	.f = cmd_quit_parsed,
-	.data = NULL,
-	.help_str = "quit: exit application",
-	.tokens = {
-		(void *)&cmd_quit_quit,
-		NULL,
-	},
-};
-cmdline_parse_ctx_t main_ctx[] = {
-	(cmdline_parse_inst_t *)&cmd_help,
-	(cmdline_parse_inst_t *)&cmd_list_vdpa_devices,
-	(cmdline_parse_inst_t *)&cmd_create_vdpa_port,
-	(cmdline_parse_inst_t *)&cmd_device_stats,
-	(cmdline_parse_inst_t *)&cmd_quit,
-	NULL,
-};
-
 int
 main(int argc, char *argv[])
 {
diff --git a/examples/vdpa/meson.build b/examples/vdpa/meson.build
index bd086050dc..a48028da4d 100644
--- a/examples/vdpa/meson.build
+++ b/examples/vdpa/meson.build
@@ -16,3 +16,10 @@ allow_experimental_apis = true
 sources = files(
         'main.c',
 )
+cmd_h = custom_target('commands_hdr',
+	output: 'commands.h',
+	input: files('commands.list'),
+	capture: true,
+	command: [cmdline_gen_cmd, '--context-name=main_ctx', '@INPUT@']
+)
+sources += cmd_h
-- 
2.39.2

Re: [PATCH v4 0/7] document and simplify use of cmdline

[David Marchand]

Hi Bruce,

On Mon, Oct 16, 2023 at 4:06 PM Bruce Richardson
<bruce.richardson@intel.com> wrote:
>
> The DPDK commandline library is widely used by apps and examples within
> DPDK, but it is not documented in our programmers guide and it requires
> a lot of boilerplate code definitions in order to used. We can improve
> this situation by creating a simple python script to automatically
> generate the boilerplate from a list of commands.
>
> This patchset contains a new documentation chapter on cmdline library,
> going through step-by-step how to add commands and create the necessary
> token lists and parse contexts.
>
> Following that initial doc patch, the set then contains a
> boilerplate-generating script, as well as a set of four patches showing
> its use, by converting four examples to use the script instead of
> having the hard-coded boilerplate. Once the script is used, adding a new
> command becomes as simple as adding the desired command to the .list
> file, and then writing the required function which will be called for
> that command. No other boilerplate coding is necessary.
>
> Script obviously does not cover the full range of capabilities of the
> commandline lib, but does cover the most used parts. The code-saving to
> each of the examples by auto-generating the boilerplate is significant,
> and probably more examples with commandlines can be converted over in
> future.

This is not something blocking for the series, but I had a quick try
with testpmd to see where this series stands.
I hit, right off the bat, some of those limitations, but I think there
are not that difficult to deal with.


- testpmd accepts both "help" and "help <section>" commands.
But the cmdline library does not provide a way (afair) for specifiying
this "optional" aspect.

And it can't be expressed with this series current syntax since
generated symbols would conflict if we ask for two "help" commands.

My quick hack was to introduce a way to select the prefix of the
generated symbols.
There may be a better way, like finding a better discriminant for
naming symbols...
But, on the other hand, the symbols prefix might be something a
developer wants to control (instead of currently hardcoded cmd_
prefix).

@@ -20,6 +20,12 @@ def process_command(tokens, cfile, comment):
     """Generate the structures and definitions for a single command."""
     name = []

+    prefix, sep, cmd_name = tokens[0].partition(':')
+    if cmd_name:
+        tokens[0] = cmd_name
+    else:
+        prefix = 'cmd'
+
     if tokens[0].startswith('<'):
         print('Error: each command must start with at least one
literal string', file=sys.stderr)
         sys.exit(1)

(etc... I am not copying the rest of the diff)

I then used as:

cmd_brief:help # help: Show help
help <STRING>section # help: Show help


- Multi choice fixed strings is something that is often used in
testpmd, like here, in the help <section> command.
Here is my quick hack:

diff --git a/buildtools/dpdk-cmdline-gen.py b/buildtools/dpdk-cmdline-gen.py
index 3b41fb0493..e8c9e079de 100755
--- a/buildtools/dpdk-cmdline-gen.py
+++ b/buildtools/dpdk-cmdline-gen.py
@@ -35,7 +35,11 @@ def process_command(tokens, cfile, comment):
     for t in tokens:
         if t.startswith('<'):
             t_type, t_name = t[1:].split('>')
-            t_val = 'NULL'
+            if len(t_type.split('(')) == 2:
+                t_type, t_val = t_type.split('(')
+                t_val = '"' + t_val.split(')')[0] + '"'
+            else:
+                t_val = 'NULL'
         else:
             t_type = 'STRING'
             t_name = t
@@ -113,7 +117,7 @@ def process_commands(infile, hfile, cfile, ctxname):
             continue
         if '#' not in line:
             line = line + '#'  # ensure split always works, even if
no help text
-        tokens, comment = line.split('#', 1)
+        tokens, comment = line.rsplit('#', 1)
         instances.append(process_command(tokens.strip().split(),
cfile, comment.strip()))

     print(f'static __rte_used cmdline_parse_ctx_t {ctxname}[] = {{')


Which translates as:
cmd_brief:help # help: Show help
help <STRING(all#control#display#config#ports)>section # help: Show help


>
> The "cmdline" example itself, is not converted over, as it should
> probably remain as a simple example of direct library use without the
> script.

+1


-- 
David Marchand

Re: [PATCH v4 0/7] document and simplify use of cmdline

[Bruce Richardson]

On Tue, Oct 17, 2023 at 09:10:51AM +0200, David Marchand wrote:
> Hi Bruce,
> 
> On Mon, Oct 16, 2023 at 4:06 PM Bruce Richardson
> <bruce.richardson@intel.com> wrote:
> >
> > The DPDK commandline library is widely used by apps and examples within
> > DPDK, but it is not documented in our programmers guide and it requires
> > a lot of boilerplate code definitions in order to used. We can improve
> > this situation by creating a simple python script to automatically
> > generate the boilerplate from a list of commands.
> >
> > This patchset contains a new documentation chapter on cmdline library,
> > going through step-by-step how to add commands and create the necessary
> > token lists and parse contexts.
> >
> > Following that initial doc patch, the set then contains a
> > boilerplate-generating script, as well as a set of four patches showing
> > its use, by converting four examples to use the script instead of
> > having the hard-coded boilerplate. Once the script is used, adding a new
> > command becomes as simple as adding the desired command to the .list
> > file, and then writing the required function which will be called for
> > that command. No other boilerplate coding is necessary.
> >
> > Script obviously does not cover the full range of capabilities of the
> > commandline lib, but does cover the most used parts. The code-saving to
> > each of the examples by auto-generating the boilerplate is significant,
> > and probably more examples with commandlines can be converted over in
> > future.
> 
> This is not something blocking for the series, but I had a quick try
> with testpmd to see where this series stands.
> I hit, right off the bat, some of those limitations, but I think there
> are not that difficult to deal with.
> 

Thanks for investigating. Item #2 below is something I'd already been
thinking of, but had not yet implemented. The hope was to get the basics in
first and iterate thereafter, rather than trying to get it all in in one
go. However, if I have the chance, and depending on how long before this
set gets merged, I might have a try at a respin with an additional feature.

> 
> - testpmd accepts both "help" and "help <section>" commands.
> But the cmdline library does not provide a way (afair) for specifiying
> this "optional" aspect.
> 
> And it can't be expressed with this series current syntax since
> generated symbols would conflict if we ask for two "help" commands.
> 
> My quick hack was to introduce a way to select the prefix of the
> generated symbols.
> There may be a better way, like finding a better discriminant for
> naming symbols...
> But, on the other hand, the symbols prefix might be something a
> developer wants to control (instead of currently hardcoded cmd_
> prefix).
> 
> @@ -20,6 +20,12 @@ def process_command(tokens, cfile, comment):
>      """Generate the structures and definitions for a single command."""
>      name = []
> 
> +    prefix, sep, cmd_name = tokens[0].partition(':')
> +    if cmd_name:
> +        tokens[0] = cmd_name
> +    else:
> +        prefix = 'cmd'
> +
>      if tokens[0].startswith('<'):
>          print('Error: each command must start with at least one
> literal string', file=sys.stderr)
>          sys.exit(1)
> 
> (etc... I am not copying the rest of the diff)
> 
> I then used as:
> 
> cmd_brief:help # help: Show help
> help <STRING>section # help: Show help
> 

Interesting. I actually had no plans to even consider moving something like
testpmd over. However, this is an interesting one, though I'm not really
sure I like it that much as a feature :-) I rather like having unique
prefixes for each command. I wasn't actually aware of the testpmd "help
<section>" command at all. I will have to look into it.

> 
> - Multi choice fixed strings is something that is often used in
> testpmd, like here, in the help <section> command.
> Here is my quick hack:
> 
> diff --git a/buildtools/dpdk-cmdline-gen.py b/buildtools/dpdk-cmdline-gen.py
> index 3b41fb0493..e8c9e079de 100755
> --- a/buildtools/dpdk-cmdline-gen.py
> +++ b/buildtools/dpdk-cmdline-gen.py
> @@ -35,7 +35,11 @@ def process_command(tokens, cfile, comment):
>      for t in tokens:
>          if t.startswith('<'):
>              t_type, t_name = t[1:].split('>')
> -            t_val = 'NULL'
> +            if len(t_type.split('(')) == 2:
> +                t_type, t_val = t_type.split('(')
> +                t_val = '"' + t_val.split(')')[0] + '"'
> +            else:
> +                t_val = 'NULL'
>          else:
>              t_type = 'STRING'
>              t_name = t
> @@ -113,7 +117,7 @@ def process_commands(infile, hfile, cfile, ctxname):
>              continue
>          if '#' not in line:
>              line = line + '#'  # ensure split always works, even if
> no help text
> -        tokens, comment = line.split('#', 1)
> +        tokens, comment = line.rsplit('#', 1)
>          instances.append(process_command(tokens.strip().split(),
> cfile, comment.strip()))
> 
>      print(f'static __rte_used cmdline_parse_ctx_t {ctxname}[] = {{')
> 
> 
> Which translates as:
> cmd_brief:help # help: Show help
> help <STRING(all#control#display#config#ports)>section # help: Show help
> 

+1
I was actualy thinking that adding support for multi-choice fixed strings
is something we should add. One thought that I had was that "#" is not a
particularly good choice of separator here. While, as you show, it can be
made to work; I think - since we are defining our own syntax here - that it
would be both simpler for the script, and simpler for the user, to have "|"
as the option separator. It should be familiar for everyone as an option
separator from regexes, unlike "#" which is more familar for comments.

So what about:
help <|all|control|display|config|ports|>section

By starting with the separator, we should avoid having to provide the
STRING type at all.

To my previous point on not liking to have a prefix-specifier, the
alternative to making testpmd work with the script is to tweak very
slightly the "help <section>".

  help   # show general help
  help on <|all|control|display|config|ports|>section

By making the command "help on ports" rather than "help ports" we would
avoid the need for the prefix syntax.

WDYT?

/Bruce

Re: [PATCH v4 0/7] document and simplify use of cmdline

[Bruce Richardson]

On Tue, Oct 17, 2023 at 09:29:34AM +0100, Bruce Richardson wrote:
> On Tue, Oct 17, 2023 at 09:10:51AM +0200, David Marchand wrote:

<snip>

> > 
> > - Multi choice fixed strings is something that is often used in
> > testpmd, like here, in the help <section> command.  Here is my quick
> > hack:
> > 
> > diff --git a/buildtools/dpdk-cmdline-gen.py
> > b/buildtools/dpdk-cmdline-gen.py index 3b41fb0493..e8c9e079de 100755
> > --- a/buildtools/dpdk-cmdline-gen.py +++
> > b/buildtools/dpdk-cmdline-gen.py @@ -35,7 +35,11 @@ def
> > process_command(tokens, cfile, comment): for t in tokens: if
> > t.startswith('<'): t_type, t_name = t[1:].split('>') -            t_val
> > = 'NULL' +            if len(t_type.split('(')) == 2: +
> > t_type, t_val = t_type.split('(') +                t_val = '"' +
> > t_val.split(')')[0] + '"' +            else: +                t_val =
> > 'NULL' else: t_type = 'STRING' t_name = t @@ -113,7 +117,7 @@ def
> > process_commands(infile, hfile, cfile, ctxname): continue if '#' not in
> > line: line = line + '#'  # ensure split always works, even if no help
> > text -        tokens, comment = line.split('#', 1) +        tokens,
> > comment = line.rsplit('#', 1)
> > instances.append(process_command(tokens.strip().split(), cfile,
> > comment.strip()))
> > 
> >      print(f'static __rte_used cmdline_parse_ctx_t {ctxname}[] = {{')
> > 
> > 
> > Which translates as: cmd_brief:help # help: Show help help
> > <STRING(all#control#display#config#ports)>section # help: Show help
> > 
> 
> +1 I was actualy thinking that adding support for multi-choice fixed
> strings is something we should add. One thought that I had was that "#"
> is not a particularly good choice of separator here. While, as you show,
> it can be made to work; I think - since we are defining our own syntax
> here - that it would be both simpler for the script, and simpler for the
> user, to have "|" as the option separator. It should be familiar for
> everyone as an option separator from regexes, unlike "#" which is more
> familar for comments.
> 
> So what about: help <|all|control|display|config|ports|>section
> 
> By starting with the separator, we should avoid having to provide the
> STRING type at all.
> 

This (my suggestion) is now prototyped in V5. I've kept it as a separate
patch so it's easily to review and discuss without affecting the rest of
the set. To test it out, I've converted over the ntb example app which uses
option lists.

/Bruce

Re: [PATCH v4 0/7] document and simplify use of cmdline

[David Marchand]

On Tue, Oct 17, 2023 at 10:29 AM Bruce Richardson
<bruce.richardson@intel.com> wrote:
> >
> > - testpmd accepts both "help" and "help <section>" commands.
> > But the cmdline library does not provide a way (afair) for specifiying
> > this "optional" aspect.
> >
> > And it can't be expressed with this series current syntax since
> > generated symbols would conflict if we ask for two "help" commands.
> >
> > My quick hack was to introduce a way to select the prefix of the
> > generated symbols.
> > There may be a better way, like finding a better discriminant for
> > naming symbols...
> > But, on the other hand, the symbols prefix might be something a
> > developer wants to control (instead of currently hardcoded cmd_
> > prefix).
> >
> > @@ -20,6 +20,12 @@ def process_command(tokens, cfile, comment):
> >      """Generate the structures and definitions for a single command."""
> >      name = []
> >
> > +    prefix, sep, cmd_name = tokens[0].partition(':')
> > +    if cmd_name:
> > +        tokens[0] = cmd_name
> > +    else:
> > +        prefix = 'cmd'
> > +
> >      if tokens[0].startswith('<'):
> >          print('Error: each command must start with at least one
> > literal string', file=sys.stderr)
> >          sys.exit(1)
> >
> > (etc... I am not copying the rest of the diff)
> >
> > I then used as:
> >
> > cmd_brief:help # help: Show help
> > help <STRING>section # help: Show help
> >
>
> Interesting. I actually had no plans to even consider moving something like
> testpmd over. However, this is an interesting one, though I'm not really

Given the extensive use of the cmdline library in testpmd, it is a
good way to identify the limits of this series :-).


> sure I like it that much as a feature :-) I rather like having unique
> prefixes for each command. I wasn't actually aware of the testpmd "help
> <section>" command at all. I will have to look into it.

Let me propose an alternative hack.
I mentionned previously that we could have a better namespace /
discriminant for those symbols, and it seems easier than I thought:

@@ -25,8 +25,10 @@ def process_command(tokens, cfile, comment):
         sys.exit(1)
     for t in tokens:
         if t.startswith('<'):
-            break
-        name.append(t)
+            t_type, t_name = t[1:].split('>')
+            name.append(t_name)
+        else:
+            name.append(t)
     name = '_'.join(name)

     result_struct = []

With this, any command implementation symbol has the full chain of
token names as a prefix which will ensure there is no conflict.
WDYT?

help # help: Show help
help <STRING>section # help: Show help

Results in:

cmd_help_parsed(void *parsed_result, struct cmdline *cl, void *data);
struct cmd_help_result {
static cmdline_parse_token_string_t cmd_help_help_tok =
    TOKEN_STRING_INITIALIZER(struct cmd_help_result, help, "help");
static cmdline_parse_inst_t cmd_help = {
    .f = cmd_help_parsed,
        (void *)&cmd_help_help_tok,

And:

cmd_help_section_parsed(void *parsed_result, struct cmdline *cl, void *data);
struct cmd_help_section_result {
static cmdline_parse_token_string_t cmd_help_section_help_tok =
    TOKEN_STRING_INITIALIZER(struct cmd_help_section_result, help, "help");
static cmdline_parse_token_string_t cmd_help_section_section_tok =
    TOKEN_STRING_INITIALIZER(struct cmd_help_section_result, section, NULL);
static cmdline_parse_inst_t cmd_help_section = {
    .f = cmd_help_section_parsed,
        (void *)&cmd_help_section_help_tok,
        (void *)&cmd_help_section_section_tok,


>
> >
> > - Multi choice fixed strings is something that is often used in
> > testpmd, like here, in the help <section> command.
> > Here is my quick hack:
> >
> > diff --git a/buildtools/dpdk-cmdline-gen.py b/buildtools/dpdk-cmdline-gen.py
> > index 3b41fb0493..e8c9e079de 100755
> > --- a/buildtools/dpdk-cmdline-gen.py
> > +++ b/buildtools/dpdk-cmdline-gen.py
> > @@ -35,7 +35,11 @@ def process_command(tokens, cfile, comment):
> >      for t in tokens:
> >          if t.startswith('<'):
> >              t_type, t_name = t[1:].split('>')
> > -            t_val = 'NULL'
> > +            if len(t_type.split('(')) == 2:
> > +                t_type, t_val = t_type.split('(')
> > +                t_val = '"' + t_val.split(')')[0] + '"'
> > +            else:
> > +                t_val = 'NULL'
> >          else:
> >              t_type = 'STRING'
> >              t_name = t
> > @@ -113,7 +117,7 @@ def process_commands(infile, hfile, cfile, ctxname):
> >              continue
> >          if '#' not in line:
> >              line = line + '#'  # ensure split always works, even if
> > no help text
> > -        tokens, comment = line.split('#', 1)
> > +        tokens, comment = line.rsplit('#', 1)
> >          instances.append(process_command(tokens.strip().split(),
> > cfile, comment.strip()))
> >
> >      print(f'static __rte_used cmdline_parse_ctx_t {ctxname}[] = {{')
> >
> >
> > Which translates as:
> > cmd_brief:help # help: Show help
> > help <STRING(all#control#display#config#ports)>section # help: Show help
> >
>
> +1
> I was actualy thinking that adding support for multi-choice fixed strings
> is something we should add. One thought that I had was that "#" is not a
> particularly good choice of separator here. While, as you show, it can be
> made to work; I think - since we are defining our own syntax here - that it
> would be both simpler for the script, and simpler for the user, to have "|"
> as the option separator. It should be familiar for everyone as an option
> separator from regexes, unlike "#" which is more familar for comments.
>
> So what about:
> help <|all|control|display|config|ports|>section

I don't like using | as it gives the false impression regexp are supported...


>
> By starting with the separator, we should avoid having to provide the
> STRING type at all.

... and as a consequence, I find <|all confusing, it is like an empty
value would be acceptable.


About skipping the token type for such lists, I had considered it, but
I thought other types took an optional list of allowed values...
Now looking at the cmdline types, it is not the case.
Maybe I mixed with some other cli framework I played with in the past...

All of this to say, ok for me to omit the type.


>
> To my previous point on not liking to have a prefix-specifier, the
> alternative to making testpmd work with the script is to tweak very
> slightly the "help <section>".
>
>   help   # show general help
>   help on <|all|control|display|config|ports|>section
>
> By making the command "help on ports" rather than "help ports" we would
> avoid the need for the prefix syntax.

There are other cases where a "chain of command" returns the value of
a parameter.
And the same parameter may be set via "chain of command <UINT>value".


-- 
David Marchand

Re: [PATCH v4 0/7] document and simplify use of cmdline

[Bruce Richardson]

On Tue, Oct 17, 2023 at 06:23:47PM +0200, David Marchand wrote:
> On Tue, Oct 17, 2023 at 10:29 AM Bruce Richardson
> <bruce.richardson@intel.com> wrote:
> > >
> > > - testpmd accepts both "help" and "help <section>" commands.
> > > But the cmdline library does not provide a way (afair) for specifiying
> > > this "optional" aspect.
> > >
> > > And it can't be expressed with this series current syntax since
> > > generated symbols would conflict if we ask for two "help" commands.
> > >
> > > My quick hack was to introduce a way to select the prefix of the
> > > generated symbols.
> > > There may be a better way, like finding a better discriminant for
> > > naming symbols...
> > > But, on the other hand, the symbols prefix might be something a
> > > developer wants to control (instead of currently hardcoded cmd_
> > > prefix).
> > >
> > > @@ -20,6 +20,12 @@ def process_command(tokens, cfile, comment):
> > >      """Generate the structures and definitions for a single command."""
> > >      name = []
> > >
> > > +    prefix, sep, cmd_name = tokens[0].partition(':')
> > > +    if cmd_name:
> > > +        tokens[0] = cmd_name
> > > +    else:
> > > +        prefix = 'cmd'
> > > +
> > >      if tokens[0].startswith('<'):
> > >          print('Error: each command must start with at least one
> > > literal string', file=sys.stderr)
> > >          sys.exit(1)
> > >
> > > (etc... I am not copying the rest of the diff)
> > >
> > > I then used as:
> > >
> > > cmd_brief:help # help: Show help
> > > help <STRING>section # help: Show help
> > >
> >
> > Interesting. I actually had no plans to even consider moving something like
> > testpmd over. However, this is an interesting one, though I'm not really
> 
> Given the extensive use of the cmdline library in testpmd, it is a
> good way to identify the limits of this series :-).
> 

It is indeed. That was never meant to be the target of this series, rather
it was to make writing new apps simpler. However, if we have a path to
getting there, then I suppose we might as well continue.

> 
> > sure I like it that much as a feature :-) I rather like having unique
> > prefixes for each command. I wasn't actually aware of the testpmd "help
> > <section>" command at all. I will have to look into it.
> 
> Let me propose an alternative hack.
> I mentionned previously that we could have a better namespace /
> discriminant for those symbols, and it seems easier than I thought:
> 
> @@ -25,8 +25,10 @@ def process_command(tokens, cfile, comment):
>          sys.exit(1)
>      for t in tokens:
>          if t.startswith('<'):
> -            break
> -        name.append(t)
> +            t_type, t_name = t[1:].split('>')
> +            name.append(t_name)
> +        else:
> +            name.append(t)
>      name = '_'.join(name)
> 
>      result_struct = []
> 
> With this, any command implementation symbol has the full chain of
> token names as a prefix which will ensure there is no conflict.
> WDYT?
> 

I actually think I like this less than the previous proposal. :-(

The main reason for that is that we would need to decide immediately
whether to go with this or not, as it would break compatibility with what
is already proposed - because the function and struct names would get
longer. The other scheme you proposed had the advantage that it could at
least be adopted later. It also had the advantage that it was largely
invisble except for the non-trivial cases that needed it.


> help # help: Show help
> help <STRING>section # help: Show help
> 
> Results in:
> 
> cmd_help_parsed(void *parsed_result, struct cmdline *cl, void *data);
> struct cmd_help_result {
> static cmdline_parse_token_string_t cmd_help_help_tok =
>     TOKEN_STRING_INITIALIZER(struct cmd_help_result, help, "help");
> static cmdline_parse_inst_t cmd_help = {
>     .f = cmd_help_parsed,
>         (void *)&cmd_help_help_tok,
> 
> And:
> 
> cmd_help_section_parsed(void *parsed_result, struct cmdline *cl, void *data);
> struct cmd_help_section_result {
> static cmdline_parse_token_string_t cmd_help_section_help_tok =
>     TOKEN_STRING_INITIALIZER(struct cmd_help_section_result, help, "help");
> static cmdline_parse_token_string_t cmd_help_section_section_tok =
>     TOKEN_STRING_INITIALIZER(struct cmd_help_section_result, section, NULL);
> static cmdline_parse_inst_t cmd_help_section = {
>     .f = cmd_help_section_parsed,
>         (void *)&cmd_help_section_help_tok,
>         (void *)&cmd_help_section_section_tok,
> 
> 
> >
> > >
> > > - Multi choice fixed strings is something that is often used in
> > > testpmd, like here, in the help <section> command.
> > > Here is my quick hack:
> > >
> > > diff --git a/buildtools/dpdk-cmdline-gen.py b/buildtools/dpdk-cmdline-gen.py
> > > index 3b41fb0493..e8c9e079de 100755
> > > --- a/buildtools/dpdk-cmdline-gen.py
> > > +++ b/buildtools/dpdk-cmdline-gen.py
> > > @@ -35,7 +35,11 @@ def process_command(tokens, cfile, comment):
> > >      for t in tokens:
> > >          if t.startswith('<'):
> > >              t_type, t_name = t[1:].split('>')
> > > -            t_val = 'NULL'
> > > +            if len(t_type.split('(')) == 2:
> > > +                t_type, t_val = t_type.split('(')
> > > +                t_val = '"' + t_val.split(')')[0] + '"'
> > > +            else:
> > > +                t_val = 'NULL'
> > >          else:
> > >              t_type = 'STRING'
> > >              t_name = t
> > > @@ -113,7 +117,7 @@ def process_commands(infile, hfile, cfile, ctxname):
> > >              continue
> > >          if '#' not in line:
> > >              line = line + '#'  # ensure split always works, even if
> > > no help text
> > > -        tokens, comment = line.split('#', 1)
> > > +        tokens, comment = line.rsplit('#', 1)
> > >          instances.append(process_command(tokens.strip().split(),
> > > cfile, comment.strip()))
> > >
> > >      print(f'static __rte_used cmdline_parse_ctx_t {ctxname}[] = {{')
> > >
> > >
> > > Which translates as:
> > > cmd_brief:help # help: Show help
> > > help <STRING(all#control#display#config#ports)>section # help: Show help
> > >
> >
> > +1
> > I was actualy thinking that adding support for multi-choice fixed strings
> > is something we should add. One thought that I had was that "#" is not a
> > particularly good choice of separator here. While, as you show, it can be
> > made to work; I think - since we are defining our own syntax here - that it
> > would be both simpler for the script, and simpler for the user, to have "|"
> > as the option separator. It should be familiar for everyone as an option
> > separator from regexes, unlike "#" which is more familar for comments.
> >
> > So what about:
> > help <|all|control|display|config|ports|>section
> 
> I don't like using | as it gives the false impression regexp are supported...
> 
I'm not sure it does particularly. The bit that struck me was that if you
look at the help texts for the commands that take multiple options, they
all give the options separated with "|". Therefore, it seems logical to
also use that in command specifiers too.

> 
> >
> > By starting with the separator, we should avoid having to provide the
> > STRING type at all.
> 
> ... and as a consequence, I find <|all confusing, it is like an empty
> value would be acceptable.
> 

I'm ok to drop the initial and terminating |, if that is what the issue is
- or else I'm misunderstanding things. For example:

help <all|control|display|config|ports>section


> 
> About skipping the token type for such lists, I had considered it, but
> I thought other types took an optional list of allowed values...
> Now looking at the cmdline types, it is not the case.
> Maybe I mixed with some other cli framework I played with in the past...
> 
> All of this to say, ok for me to omit the type.
>

Good. However, it only works if we come up with a suitable separator
character to detect to split on. I don't like "#" as separator as it is
used so many places as a comment character. What about bracketed,
comma-separated lists?

help <(all,control,display,config,ports)>section

> 
> >
> > To my previous point on not liking to have a prefix-specifier, the
> > alternative to making testpmd work with the script is to tweak very
> > slightly the "help <section>".
> >
> >   help   # show general help
> >   help on <|all|control|display|config|ports|>section
> >
> > By making the command "help on ports" rather than "help ports" we would
> > avoid the need for the prefix syntax.
> 
> There are other cases where a "chain of command" returns the value of
> a parameter.
> And the same parameter may be set via "chain of command <UINT>value".
> 
Ok, let me think on it a bit....

/Bruce

Re: [PATCH v4 0/7] document and simplify use of cmdline

[Bruce Richardson]

On Tue, Oct 17, 2023 at 06:23:47PM +0200, David Marchand wrote:
> On Tue, Oct 17, 2023 at 10:29 AM Bruce Richardson
> <bruce.richardson@intel.com> wrote:
> > >
> > > - testpmd accepts both "help" and "help <section>" commands.
> > > But the cmdline library does not provide a way (afair) for specifiying
> > > this "optional" aspect.
> > >
> > > And it can't be expressed with this series current syntax since
> > > generated symbols would conflict if we ask for two "help" commands.
> > >
> > > My quick hack was to introduce a way to select the prefix of the
> > > generated symbols.
> > > There may be a better way, like finding a better discriminant for
> > > naming symbols...
> > > But, on the other hand, the symbols prefix might be something a
> > > developer wants to control (instead of currently hardcoded cmd_
> > > prefix).
> > >
> > > @@ -20,6 +20,12 @@ def process_command(tokens, cfile, comment):
> > >      """Generate the structures and definitions for a single command."""
> > >      name = []
> > >
> > > +    prefix, sep, cmd_name = tokens[0].partition(':')
> > > +    if cmd_name:
> > > +        tokens[0] = cmd_name
> > > +    else:
> > > +        prefix = 'cmd'
> > > +
> > >      if tokens[0].startswith('<'):
> > >          print('Error: each command must start with at least one
> > > literal string', file=sys.stderr)
> > >          sys.exit(1)
> > >
> > > (etc... I am not copying the rest of the diff)
> > >
> > > I then used as:
> > >
> > > cmd_brief:help # help: Show help
> > > help <STRING>section # help: Show help
> > >
> >
> > Interesting. I actually had no plans to even consider moving something like
> > testpmd over. However, this is an interesting one, though I'm not really
> 
> Given the extensive use of the cmdline library in testpmd, it is a
> good way to identify the limits of this series :-).
> 
> 
> > sure I like it that much as a feature :-) I rather like having unique
> > prefixes for each command. I wasn't actually aware of the testpmd "help
> > <section>" command at all. I will have to look into it.
> 
> Let me propose an alternative hack.
> I mentionned previously that we could have a better namespace /
> discriminant for those symbols, and it seems easier than I thought:
> 
> @@ -25,8 +25,10 @@ def process_command(tokens, cfile, comment):
>          sys.exit(1)
>      for t in tokens:
>          if t.startswith('<'):
> -            break
> -        name.append(t)
> +            t_type, t_name = t[1:].split('>')
> +            name.append(t_name)
> +        else:
> +            name.append(t)
>      name = '_'.join(name)
> 
>      result_struct = []
> 
> With this, any command implementation symbol has the full chain of
> token names as a prefix which will ensure there is no conflict.
> WDYT?
> 
Having thought a little more about it, I still don't like having the full
command in all cases, but I can see it being useful for cases of
overlapping prefixes.

How about making it optional - setting a flag in the typename, or in the
parameter name to indicate that it should be included in the overall
command name. For example, if we prefix the variable name with "_" or "__",
it could indicate that we can choose to include this.

show port <UINT16>n  --> void cmd_show_port_parsed(...)
show port <UINT16>_n --> void cmd_show_port_n_parsed(...)

Prefixes on strings beyond initial tokens could just be silently stripped.
[Obviously open to other ideas on what form tagging could take]

/Bruce

Re: [PATCH v4 0/7] document and simplify use of cmdline

[David Marchand]

Hello Bruce,


On Tue, Oct 17, 2023 at 7:08 PM Bruce Richardson
<bruce.richardson@intel.com> wrote:
> > > sure I like it that much as a feature :-) I rather like having unique
> > > prefixes for each command. I wasn't actually aware of the testpmd "help
> > > <section>" command at all. I will have to look into it.
> >
> > Let me propose an alternative hack.
> > I mentionned previously that we could have a better namespace /
> > discriminant for those symbols, and it seems easier than I thought:
> >
> > @@ -25,8 +25,10 @@ def process_command(tokens, cfile, comment):
> >          sys.exit(1)
> >      for t in tokens:
> >          if t.startswith('<'):
> > -            break
> > -        name.append(t)
> > +            t_type, t_name = t[1:].split('>')
> > +            name.append(t_name)
> > +        else:
> > +            name.append(t)
> >      name = '_'.join(name)
> >
> >      result_struct = []
> >
> > With this, any command implementation symbol has the full chain of
> > token names as a prefix which will ensure there is no conflict.
> > WDYT?
> >
> Having thought a little more about it, I still don't like having the full
> command in all cases, but I can see it being useful for cases of
> overlapping prefixes.
>
> How about making it optional - setting a flag in the typename, or in the
> parameter name to indicate that it should be included in the overall
> command name. For example, if we prefix the variable name with "_" or "__",
> it could indicate that we can choose to include this.
>
> show port <UINT16>n  --> void cmd_show_port_parsed(...)
> show port <UINT16>_n --> void cmd_show_port_n_parsed(...)
>

I think I get what you mean, and it seems acceptable.


> Prefixes on strings beyond initial tokens could just be silently stripped.

By initial tokens, do you mean fixed strings token before a <> typed token ?


-- 
David Marchand

Re: [PATCH v4 0/7] document and simplify use of cmdline

[Bruce Richardson]

On Wed, Oct 18, 2023 at 01:21:40PM +0200, David Marchand wrote:
> Hello Bruce,
> 
> 
> On Tue, Oct 17, 2023 at 7:08 PM Bruce Richardson
> <bruce.richardson@intel.com> wrote:
> > > > sure I like it that much as a feature :-) I rather like having unique
> > > > prefixes for each command. I wasn't actually aware of the testpmd "help
> > > > <section>" command at all. I will have to look into it.
> > >
> > > Let me propose an alternative hack.
> > > I mentionned previously that we could have a better namespace /
> > > discriminant for those symbols, and it seems easier than I thought:
> > >
> > > @@ -25,8 +25,10 @@ def process_command(tokens, cfile, comment):
> > >          sys.exit(1)
> > >      for t in tokens:
> > >          if t.startswith('<'):
> > > -            break
> > > -        name.append(t)
> > > +            t_type, t_name = t[1:].split('>')
> > > +            name.append(t_name)
> > > +        else:
> > > +            name.append(t)
> > >      name = '_'.join(name)
> > >
> > >      result_struct = []
> > >
> > > With this, any command implementation symbol has the full chain of
> > > token names as a prefix which will ensure there is no conflict.
> > > WDYT?
> > >
> > Having thought a little more about it, I still don't like having the full
> > command in all cases, but I can see it being useful for cases of
> > overlapping prefixes.
> >
> > How about making it optional - setting a flag in the typename, or in the
> > parameter name to indicate that it should be included in the overall
> > command name. For example, if we prefix the variable name with "_" or "__",
> > it could indicate that we can choose to include this.
> >
> > show port <UINT16>n  --> void cmd_show_port_parsed(...)
> > show port <UINT16>_n --> void cmd_show_port_n_parsed(...)
> >
> 
> I think I get what you mean, and it seems acceptable.
> 

Cool. Any suggestions for a preferred prefix to indicate inclusion in the
cmd name? "_", "__" or something else? I'm trending towards single "_" as
above.

> 
> > Prefixes on strings beyond initial tokens could just be silently stripped.
> 
> By initial tokens, do you mean fixed strings token before a <> typed token ?
>
Yes.

So:

add <UINT16>x             --> cmd_add_parsed
add <UINT16>_x            --> cmd_add_x_parsed
add <UINT16>_x <UINT16>_y --> cmd_add_x_y_parsed
add <UINT16>x <UINT16>_y  --> cmd_add_parsed, strip "_" off y silently

/Bruce

[PATCH v5 0/9] document and simplify use of cmdline

[Bruce Richardson]

The DPDK commandline library is widely used by apps and examples within
DPDK, but it is not documented in our programmers guide and it requires
a lot of boilerplate code definitions in order to used. We can improve
this situation by creating a simple python script to automatically
generate the boilerplate from a list of commands.

This patchset contains a new documentation chapter on cmdline library,
going through step-by-step how to add commands and create the necessary
token lists and parse contexts.

Following that initial doc patch, the set then contains a
boilerplate-generating script, as well as a set of four patches showing
its use, by converting four examples to use the script instead of
having the hard-coded boilerplate. Once the script is used, adding a new
command becomes as simple as adding the desired command to the .list
file, and then writing the required function which will be called for
that command. No other boilerplate coding is necessary.

Final two patches, new in V5, add support for option-lists in values,
and then use that support to convert over a fifth sample app - ntb.

Cmdline script obviously does not cover the full range of capabilities
of the commandline lib, but does cover the most used parts. The
code-saving to each of the examples by auto-generating the boilerplate
is significant, and probably more examples with commandlines can be
converted over in future.

The "cmdline" example itself, is not converted over, as it should
probably remain as a simple example of direct library use without the
script.

V5:
* Added copyright headers to command list files
* Add support for option lists
* Convert examples/ntb to use script

V4:
* Reworked script following feedback from Robin J.
   - formatted code with black
   - used multi-line strings
   - replaced sys.exit(1) with proper error handling
   - per-command function returns lists rather than printing directly
   - other small cleanups
* Added change to CI script so the cmdline script is in PATH
* Converted VDPA example to script, saving another 100 LOC

V3:
* Added lots of documentation
* Added support for help text for each command
* Cleaned up script a little so it passes pycodestyle and most flake8
  checks, when line-length is set to max 100.
* Removed RFC tag, as I consider this patchset stable enough for
  consideration in a release.

V2-RFC:
* Add support for IP addresses in commands
* Move to buildtools directory and make installable
* Convert 3 examples to use script, and eliminate their boilerplate

Bruce Richardson (9):
  doc/prog_guide: new chapter on cmdline library
  buildtools: script to generate cmdline boilerplate
  ci: allow use of DPDK tools when building examples
  examples/simple_mp: auto-generate cmdline boilerplate
  examples/hotplug_mp: auto-generate cmdline boilerplate
  examples/bond: auto-generate cmdline boilerplate
  examples/vdpa: auto-generate cmdline boilerplate
  buildtools/dpdk-cmdline-gen: support option strings
  examples/ntb: auto-generate cmdline boilerplate

 .ci/linux-build.sh                            |   1 +
 app/test/commands.c                           |   2 +
 buildtools/dpdk-cmdline-gen.py                | 197 ++++++++
 buildtools/meson.build                        |   7 +
 doc/guides/prog_guide/cmdline.rst             | 470 ++++++++++++++++++
 doc/guides/prog_guide/index.rst               |   1 +
 examples/bond/Makefile                        |  12 +-
 examples/bond/commands.list                   |   6 +
 examples/bond/main.c                          | 161 +-----
 examples/bond/main.h                          |  10 -
 examples/bond/meson.build                     |   8 +
 examples/multi_process/hotplug_mp/Makefile    |  12 +-
 examples/multi_process/hotplug_mp/commands.c  | 147 +-----
 examples/multi_process/hotplug_mp/commands.h  |  10 -
 .../multi_process/hotplug_mp/commands.list    |   8 +
 examples/multi_process/hotplug_mp/meson.build |   9 +
 examples/multi_process/simple_mp/Makefile     |  12 +-
 examples/multi_process/simple_mp/meson.build  |   9 +
 .../multi_process/simple_mp/mp_commands.c     | 106 +---
 .../multi_process/simple_mp/mp_commands.h     |  14 -
 .../multi_process/simple_mp/mp_commands.list  |   6 +
 examples/ntb/Makefile                         |  12 +-
 examples/ntb/commands.list                    |  11 +
 examples/ntb/meson.build                      |   7 +
 examples/ntb/ntb_fwd.c                        | 200 +-------
 examples/vdpa/Makefile                        |  12 +-
 examples/vdpa/commands.list                   |   8 +
 examples/vdpa/main.c                          | 131 +----
 examples/vdpa/meson.build                     |   7 +
 29 files changed, 863 insertions(+), 733 deletions(-)
 create mode 100755 buildtools/dpdk-cmdline-gen.py
 create mode 100644 doc/guides/prog_guide/cmdline.rst
 create mode 100644 examples/bond/commands.list
 delete mode 100644 examples/bond/main.h
 delete mode 100644 examples/multi_process/hotplug_mp/commands.h
 create mode 100644 examples/multi_process/hotplug_mp/commands.list
 delete mode 100644 examples/multi_process/simple_mp/mp_commands.h
 create mode 100644 examples/multi_process/simple_mp/mp_commands.list
 create mode 100644 examples/ntb/commands.list
 create mode 100644 examples/vdpa/commands.list

--
2.39.2

[PATCH v5 1/9] doc/prog_guide: new chapter on cmdline library

[Bruce Richardson]

The cmdline library was not documented in our programmers guide, so add
a new chapter on it. This chapter covers step-by-step how to use the
library, rather than focusing on the library internals. This complements
the existing cmdline example app document, providing more details on the
process of using the library.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 app/test/commands.c               |   2 +
 doc/guides/prog_guide/cmdline.rst | 337 ++++++++++++++++++++++++++++++
 doc/guides/prog_guide/index.rst   |   1 +
 3 files changed, 340 insertions(+)
 create mode 100644 doc/guides/prog_guide/cmdline.rst

diff --git a/app/test/commands.c b/app/test/commands.c
index 31259e5c21..497d8e9952 100644
--- a/app/test/commands.c
+++ b/app/test/commands.c
@@ -108,6 +108,7 @@ dump_struct_sizes(void)
 #undef DUMP_SIZE
 }
 
+/* Add the dump_* tests cases 8< */
 static void cmd_dump_parsed(void *parsed_result,
 			    __rte_unused struct cmdline *cl,
 			    __rte_unused void *data)
@@ -155,6 +156,7 @@ cmdline_parse_inst_t cmd_dump = {
 		NULL,
 	},
 };
+/* >8 End of add the dump_* tests cases */
 
 /****************/
 
diff --git a/doc/guides/prog_guide/cmdline.rst b/doc/guides/prog_guide/cmdline.rst
new file mode 100644
index 0000000000..40f49a30cc
--- /dev/null
+++ b/doc/guides/prog_guide/cmdline.rst
@@ -0,0 +1,337 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2023 Intel Corporation.
+
+Command-line Library
+====================
+
+Since its earliest versions, DPDK has included a command-line library -
+primarily for internal use by, for example, ``dpdk-testpmd`` and the ``dpdk-test`` binaries,
+but the library is also exported on install and can be used by any end application.
+This chapter covers the basics of the command-line library and how to use it in an application.
+
+Library Features
+----------------
+
+The DPDK command-line library supports the following features:
+
+* Tab-completion available for interactive terminal sessions
+
+* Ability to read and process commands taken from an input file, e.g. startup script
+
+* Parameterized commands able to take multiple parameters with different datatypes:
+
+   * Strings
+   * Signed/unsigned 16/32/64-bit integers
+   * IP Addresses
+   * Ethernet Addresses
+
+* Ability to multiplex multiple commands to a single callback function
+
+Adding Command-line to an Application
+-------------------------------------
+
+Adding a command-line instance to an application involves a number of coding steps.
+
+1. Define the result structure for the command, specifying the command parameters
+
+2. Provide an initializer for each field in the result
+
+3. Define the callback function for the command
+
+4. Provide a parse result structure instance for the command, linking the callback to the command
+
+5. Add the parse result structure to a command-line context
+
+6. Within your main application code, create a new command-line instance passing in the context.
+
+The next few subsections will cover each of these steps in more detail,
+working through an example to add two commands to a command-line instance.
+Those two commands will be:
+
+1. ``quit`` - as the name suggests, to close the application
+
+2. ``show port stats <n>`` - to display on-screen the statistics for a given ethernet port
+
+.. note::
+
+   For further examples of use of the command-line, see
+   :doc:`cmdline example application <../sample_app_ug/cmd_line>`
+
+Defining Command Result Structure
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The first structure to be defined is the structure which will be created on successful parse of a command.
+This structure contains one member field for each token, or word, in the command.
+The simplest case is for a one-word command, like ``quit``.
+For this, we only need to define a structure with a single string parameter to contain that word.
+
+.. code-block:: c
+
+   struct cmd_quit_result {
+              cmdline_fixed_string_t quit;
+   };
+
+For readability, the name of the struct member should match that of the token in the command.
+
+For our second command, we need a structure with four member fields in it,
+as there are four words/tokens in our command.
+The first three are strings, and the final one is a 16-bit numeric value.
+The resulting struct looks like:
+
+.. code-block:: c
+
+   struct cmd_show_port_stats_result {
+      cmdline_fixed_string_t show;
+      cmdline_fixed_string_t port;
+      cmdline_fixed_string_t stats;
+      uint16_t n;
+   };
+
+As before, we choose names to match the tokens in the command.
+Since our numeric parameter is a 16-bit value, we use ``uint16_t`` type for it.
+Any of the standard sized integer types can be used as parameters, depending on the desired result.
+
+Beyond the standard integer types,
+the library also allows variable parameters to be of a number of other types,
+as called out in the feature list above.
+
+* For variable string parameters,
+  the type should be ``cmdline_fixed_string_t`` - the same as for fixed tokens,
+  but these will be initialized differently (as described below).
+
+* For ethernet addresses use type ``struct rte_ether_addr``
+
+* For IP addresses use type ``cmdline_ipaddr_t``
+
+Providing Field Initializers
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Each field of our result structure needs an initializer.
+For fixed string tokens, like "quit", "show" and "port", the initializer will be the string itself.
+
+.. code-block:: c
+
+   static cmdline_parse_token_string_t cmd_quit_quit_tok =
+      TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit");
+
+The convention for naming used here is to include the base name of the overall result structure -
+``cmd_quit`` in this case,
+as well as the name of the field within that structure - ``quit`` in this case, followed by ``_tok``.
+(This is why there is a double ``quit`` in the name above).
+
+This naming convention is seen in our second example,
+which also demonstrates how to define a numeric initializer.
+
+
+.. code-block:: c
+
+   static cmdline_parse_token_string_t cmd_show_port_stats_show_tok =
+      TOKEN_STRING_INITIALIZER(struct cmd_show_port_stats_result, show, "show");
+   static cmdline_parse_token_string_t cmd_show_port_stats_port_tok =
+      TOKEN_STRING_INITIALIZER(struct cmd_show_port_stats_result, port, "port");
+   static cmdline_parse_token_string_t cmd_show_port_stats_stats_tok =
+      TOKEN_STRING_INITIALIZER(struct cmd_show_port_stats_result, stats, "stats");
+   static cmdline_parse_token_num_t cmd_show_port_stats_n_tok =
+      TOKEN_NUM_INITIALIZER(struct cmd_show_port_stats_result, n, RTE_UINT16);
+
+For variable string tokens, the same ``TOKEN_STRING_INITIALIZER`` macro should be used.
+However, the final parameter should be ``NULL`` rather than a hard-coded token string.
+
+For numeric parameters, the final parameter to the ``TOKEN_NUM_INITIALIZER`` macro should be the
+cmdline type matching the variable type defined in the result structure,
+e.g. RTE_UINT8, RTE_UINT32, etc.
+
+For IP addresses, the macro ``TOKEN_IPADDR_INITIALIZER`` should be used.
+
+For ethernet addresses, the macro ``TOKEN_ETHERADDR_INITIALIZER`` should be used.
+
+Defining Callback Function
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+For each command, we need to define a function to be called once the command has been recognised.
+The callback function should have type:
+
+.. code:: c
+
+   void (*f)(void *, struct cmdline *, void *)
+
+where the first parameter is a pointer to the result structure defined above,
+the second parameter is the command-line instance,
+and the final parameter is a user-defined pointer provided when we associate the callback with the command.
+Most callback functions only use the first parameter, or none at all,
+but the additional two parameters provide some extra flexibility,
+to allow the callback to work with non-global state in your application.
+
+For our two example commands, the relevant callback functions would look very similar in definition.
+However, within the function body,
+we assume that the user would need to reference the result structure to extract the port number in
+the second case.
+
+.. code:: c
+
+   void
+   cmd_quit_parsed(void *parsed_result, struct cmdline *cl, void *data)
+   {
+      quit = 1;
+   }
+   void
+   cmd_show_port_stats_parsed(void *parsed_result, struct cmdline *cl, void *data)
+   {
+      struct cmd_show_port_stats_result *res = parsed_result;
+      uint16_t port_id = res->n;
+      ...
+   }
+
+
+Associating Callback and Command
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The ``cmdline_parse_inst_t`` type defines a "parse instance",
+i.e. a sequence of tokens to be matched and then an associated function to be called.
+Also included in the instance type are a field for help text for the command,
+and any additional user-defined parameter to be passed to the callback functions referenced above.
+For example, for our simple "quit" command:
+
+.. code-block:: c
+
+   static cmdline_parse_inst_t cmd_quit = {
+       .f = cmd_quit_parsed,
+       .data = NULL,
+       .help_str = "Close the application",
+       .tokens = {
+           (void *)&cmd_quit_quit_tok,
+           NULL
+       }
+   };
+
+In this case, we firstly identify the callback function to be called,
+then set the user-defined parameter to NULL,
+provide a help message to be given, on request, to the user explaining the command,
+before finally listing out the single token to be matched for this command instance.
+
+For our second, port stats, example,
+as well as making things a little more complicated by having multiple tokens to be matched,
+we can also demonstrate passing in a parameter to the function.
+Let us suppose that our application does not always use all the ports available to it,
+but instead only uses a subset of the ports, stored in an array called ``active_ports``.
+Our stats command, therefore, should only display stats for the currently in-use ports,
+so we pass this ``active_ports`` array.
+(For simplicity of illustration, we shall assume that the array uses a terminating marker,
+e.g. -1 for the end of the port list, so we don't need to pass in a length parameter too.)
+
+.. code-block:: c
+
+   extern int16_t active_ports[];
+   ...
+   static cmdline_parse_inst_t cmd_show_port_stats = {
+       .f = cmd_show_port_stats_parsed,
+       .data = active_ports,
+       .help_str = "Show statistics for active network ports",
+       .tokens = {
+           (void *)&cmd_show_port_stats_show_tok,
+           (void *)&cmd_show_port_stats_port_tok,
+           (void *)&cmd_show_port_stats_stats_tok,
+           (void *)&cmd_show_port_stats_n_tok,
+           NULL
+       }
+   };
+
+
+Adding Command to Command-line Context
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Now that we have configured each individual command and callback,
+we need to merge these into a single array of command-line "contexts".
+This context array will be used to create the actual command-line instance in the application.
+Thankfully, each context entry is the same as each parse instance,
+so our array is defined by simply listing out the previously defined command parse instances.
+
+.. code-block:: c
+
+   static cmdline_parse_ctx_t ctx[] = {
+       &cmd_quit,
+       &cmd_show_port_stats,
+       NULL
+   };
+
+The context list must be terminated by a NULL entry.
+
+Creating a Command-line Instance
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Once we have our ``ctx`` variable defined,
+we now just need to call the API to create the new command-line instance in our application.
+The basic API is ``cmdline_new`` which will create an interactive command-line with all commands available.
+However, if additional features for interactive use - such as tab-completion -
+are desired, it is recommended that ``cmdline_new_stdin`` be used instead.
+
+A pattern that can be used in applications is to use ``cmdline_new`` for processing any startup commands,
+either from file or from the environment (as is done in the "dpdk-test" application),
+and then using ``cmdline_stdin_new`` thereafter to handle the interactive part.
+For example, to handle a startup file and then provide an interactive prompt:
+
+.. code-block:: c
+
+   struct cmdline *cl;
+   int fd = open(startup_file, O_RDONLY);
+
+   if (fd >= 0) {
+       cl = cmdline_new(ctx, "", fd, STDOUT_FILENO);
+       if (cl == NULL) {
+           /* error handling */
+       }
+       cmdline_interact(cl);
+       cmdline_quit(cl);
+       close(fd);
+   }
+
+   cl = cmdline_stdin_new(ctx, "Proxy>> ");
+   if (cl == NULL) {
+       /* error handling */
+   }
+   cmdline_interact(cl);
+   cmdline_stdin_exit(cl);
+
+
+Multiplexing Multiple Commands to a Single Function
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To reduce the amount of boiler-plate code needed when creating a command-line for an application,
+it is possible to merge a number of commands together to have them call a separate function.
+This can be done in a number of different ways:
+
+* A callback function can be used as the target for a number of different commands.
+  Which command was used for entry to the function can be determined by examining the first parameter,
+  ``parsed_result`` in our examples above.
+
+* For simple string commands, multiple options can be concatenated using the "#" character.
+  For example: ``exit#quit``, specified as a token initializer,
+  will match either on the string "exit" or the string "quit".
+
+As a concrete example,
+these two techniques are used in the DPDK unit test application ``dpdk-test``,
+where a single command ``cmdline_parse_t`` instance is used for all the "dump_<item>" test cases.
+
+.. literalinclude:: ../../../app/test/commands.c
+    :language: c
+    :start-after: Add the dump_* tests cases 8<
+    :end-before: >8 End of add the dump_* tests cases
+
+
+Examples of Command-line Use in DPDK
+------------------------------------
+
+To help the user follow the steps provided above,
+the following DPDK files can be consulted for examples of command-line use.
+
+.. note::
+
+   This is not an exhaustive list of examples of command-line use in DPDK.
+   It is simply a list of a few files that may be of use to the application developer.
+   Some of these referenced files contain more complex examples of use that others.
+
+* ``commands.c/.h`` in ``examples/cmdline``
+
+* ``mp_commands.c/.h`` in ``examples/multi_process/simple_mp``
+
+* ``commands.c`` in ``app/test``
diff --git a/doc/guides/prog_guide/index.rst b/doc/guides/prog_guide/index.rst
index e517f0e259..94964357ff 100644
--- a/doc/guides/prog_guide/index.rst
+++ b/doc/guides/prog_guide/index.rst
@@ -13,6 +13,7 @@ Programmer's Guide
     source_org
     env_abstraction_layer
     log_lib
+    cmdline
     service_cores
     trace_lib
     rcu_lib
-- 
2.39.2

[PATCH v5 2/9] buildtools: script to generate cmdline boilerplate

[Bruce Richardson]

Provide a "dpdk-cmdline-gen.py" script for application developers to
quickly generate the boilerplate code necessary for using the cmdline
library.

Example of use:
The script takes an input file with a list of commands the user wants in
the app, where the parameter variables are tagged with the type.
For example:

	$ cat commands.list
	list
	add <UINT16>x <UINT16>y
	echo <STRING>message
	add socket <STRING>path
	quit

When run through the script as "./dpdk-cmdline-gen.py commands.list",
the output will be the contents of a header file with all the
boilerplate necessary for a commandline instance with those commands.

If the flag --stubs is passed, an output header filename must also be
passed, in which case both a header file with the definitions and a C
file with function stubs in it is written to disk. The separation is so
that the header file can be rewritten at any future point to add more
commands, while the C file can be kept as-is and extended by the user
with any additional functions needed.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 buildtools/dpdk-cmdline-gen.py    | 190 ++++++++++++++++++++++++++++++
 buildtools/meson.build            |   7 ++
 doc/guides/prog_guide/cmdline.rst | 131 +++++++++++++++++++-
 3 files changed, 327 insertions(+), 1 deletion(-)
 create mode 100755 buildtools/dpdk-cmdline-gen.py

diff --git a/buildtools/dpdk-cmdline-gen.py b/buildtools/dpdk-cmdline-gen.py
new file mode 100755
index 0000000000..6cb7610de4
--- /dev/null
+++ b/buildtools/dpdk-cmdline-gen.py
@@ -0,0 +1,190 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2023 Intel Corporation
+#
+"""
+Script to automatically generate boilerplate for using DPDK cmdline library.
+"""
+
+import argparse
+import sys
+
+PARSE_FN_PARAMS = "void *parsed_result, struct cmdline *cl, void *data"
+PARSE_FN_BODY = """
+    /* TODO: command action */
+    RTE_SET_USED(parsed_result);
+    RTE_SET_USED(cl);
+    RTE_SET_USED(data);
+"""
+NUMERIC_TYPES = [
+    "UINT8",
+    "UINT16",
+    "UINT32",
+    "UINT64",
+    "INT8",
+    "INT16",
+    "INT32",
+    "INT64",
+]
+
+
+def process_command(lineno, tokens, comment):
+    """Generate the structures and definitions for a single command."""
+    out = []
+    cfile_out = []
+
+    if tokens[0].startswith("<"):
+        raise ValueError(f"Error line {lineno + 1}: command must start with a literal string")
+
+    name_tokens = []
+    for t in tokens:
+        if t.startswith("<"):
+            break
+        name_tokens.append(t)
+    name = "_".join(name_tokens)
+
+    result_struct = []
+    initializers = []
+    token_list = []
+    for t in tokens:
+        if t.startswith("<"):
+            t_type, t_name = t[1:].split(">")
+            t_val = "NULL"
+        else:
+            t_type = "STRING"
+            t_name = t
+            t_val = f'"{t}"'
+
+        if t_type == "STRING":
+            result_struct.append(f"\tcmdline_fixed_string_t {t_name};")
+            initializers.append(
+                f"static cmdline_parse_token_string_t cmd_{name}_{t_name}_tok =\n"
+                + f"\tTOKEN_STRING_INITIALIZER(struct cmd_{name}_result, {t_name}, {t_val});"
+            )
+        elif t_type in NUMERIC_TYPES:
+            result_struct.append(f"\t{t_type.lower()}_t {t_name};")
+            initializers.append(
+                f"static cmdline_parse_token_num_t cmd_{name}_{t_name}_tok =\n"
+                + f"\tTOKEN_NUM_INITIALIZER(struct cmd_{name}_result, {t_name}, RTE_{t_type});"
+            )
+        elif t_type in ["IP", "IP_ADDR", "IPADDR"]:
+            result_struct.append(f"\tcmdline_ipaddr_t {t_name};")
+            initializers.append(
+                f"cmdline_parse_token_ipaddr_t cmd_{name}_{t_name}_tok =\n"
+                + f"\tTOKEN_IPV4_INITIALIZER(struct cmd_{name}_result, {t_name});"
+            )
+        else:
+            raise TypeError(f"Error line {lineno + 1}: unknown token type '{t_type}'")
+        token_list.append(f"cmd_{name}_{t_name}_tok")
+
+    out.append(f'/* Auto-generated handling for command "{" ".join(tokens)}" */')
+    # output function prototype
+    func_sig = f"void\ncmd_{name}_parsed({PARSE_FN_PARAMS})"
+    out.append(f"extern {func_sig};\n")
+    # output result data structure
+    out.append(f"struct cmd_{name}_result {{\n" + "\n".join(result_struct) + "\n};\n")
+    # output the initializer tokens
+    out.append("\n".join(initializers) + "\n")
+    # output the instance structure
+    out.append(
+        f"static cmdline_parse_inst_t cmd_{name} = {{\n"
+        + f"\t.f = cmd_{name}_parsed,\n"
+        + "\t.data = NULL,\n"
+        + f'\t.help_str = "{comment}",\n'
+        + "\t.tokens = {"
+    )
+    for t in token_list:
+        out.append(f"\t\t(void *)&{t},")
+    out.append("\t\tNULL\n" + "\t}\n" + "};\n")
+    # output function template if C file being written
+    cfile_out.append(f"{func_sig}\n{{{PARSE_FN_BODY}}}\n")
+
+    # return the instance structure name
+    return (f"cmd_{name}", out, cfile_out)
+
+
+def process_commands(infile, hfile, cfile, ctxname):
+    """Generate boilerplate output for a list of commands from infile."""
+    instances = []
+
+    hfile.write(
+        f"""/* File autogenerated by {sys.argv[0]} */
+#ifndef GENERATED_COMMANDS_H
+#define GENERATED_COMMANDS_H
+#include <rte_common.h>
+#include <cmdline.h>
+#include <cmdline_parse_string.h>
+#include <cmdline_parse_num.h>
+#include <cmdline_parse_ipaddr.h>
+
+"""
+    )
+
+    for lineno, line in enumerate(infile.readlines()):
+        if line.lstrip().startswith("#"):
+            continue
+        if "#" not in line:
+            line = line + "#"  # ensure split always works, even if no help text
+        tokens, comment = line.split("#", 1)
+        cmd_inst, h_out, c_out = process_command(lineno, tokens.strip().split(), comment.strip())
+        hfile.write("\n".join(h_out))
+        if cfile:
+            cfile.write("\n".join(c_out))
+        instances.append(cmd_inst)
+
+    inst_join_str = ",\n\t&"
+    hfile.write(
+        f"""
+static __rte_used cmdline_parse_ctx_t {ctxname}[] = {{
+\t&{inst_join_str.join(instances)},
+\tNULL
+}};
+
+#endif /* GENERATED_COMMANDS_H */
+"""
+    )
+
+
+def main():
+    """Application main entry point."""
+    ap = argparse.ArgumentParser(description=__doc__)
+    ap.add_argument(
+        "--stubs",
+        action="store_true",
+        help="Produce C file with empty function stubs for each command",
+    )
+    ap.add_argument(
+        "--output-file",
+        "-o",
+        default="-",
+        help="Output header filename [default to stdout]",
+    )
+    ap.add_argument(
+        "--context-name",
+        default="ctx",
+        help="Name given to the cmdline context variable in the output header [default=ctx]",
+    )
+    ap.add_argument("infile", type=argparse.FileType("r"), help="File with list of commands")
+    args = ap.parse_args()
+
+    if not args.stubs:
+        if args.output_file == "-":
+            process_commands(args.infile, sys.stdout, None, args.context_name)
+        else:
+            with open(args.output_file, "w") as hfile:
+                process_commands(args.infile, hfile, None, args.context_name)
+    else:
+        if not args.output_file.endswith(".h"):
+            ap.error(
+                "-o/--output-file: specify an output filename ending with .h when creating stubs"
+            )
+
+        cfilename = args.output_file[:-2] + ".c"
+        with open(args.output_file, "w") as hfile:
+            with open(cfilename, "w") as cfile:
+                cfile.write(f'#include "{args.output_file}"\n\n')
+                process_commands(args.infile, hfile, cfile, args.context_name)
+
+
+if __name__ == "__main__":
+    main()
diff --git a/buildtools/meson.build b/buildtools/meson.build
index 948ac17dd2..72447b60a0 100644
--- a/buildtools/meson.build
+++ b/buildtools/meson.build
@@ -19,6 +19,13 @@ get_cpu_count_cmd = py3 + files('get-cpu-count.py')
 get_numa_count_cmd = py3 + files('get-numa-count.py')
 get_test_suites_cmd = py3 + files('get-test-suites.py')
 has_hugepages_cmd = py3 + files('has-hugepages.py')
+cmdline_gen_cmd = py3 + files('dpdk-cmdline-gen.py')
+
+# install any build tools that end-users might want also
+install_data([
+            'dpdk-cmdline-gen.py',
+        ],
+        install_dir: 'bin')
 
 # select library and object file format
 pmdinfo = py3 + files('gen-pmdinfo-cfile.py') + [meson.current_build_dir()]
diff --git a/doc/guides/prog_guide/cmdline.rst b/doc/guides/prog_guide/cmdline.rst
index 40f49a30cc..0b96b770e2 100644
--- a/doc/guides/prog_guide/cmdline.rst
+++ b/doc/guides/prog_guide/cmdline.rst
@@ -44,7 +44,136 @@ Adding a command-line instance to an application involves a number of coding ste
 
 6. Within your main application code, create a new command-line instance passing in the context.
 
-The next few subsections will cover each of these steps in more detail,
+Many of these steps can be automated using the script ``dpdk-cmdline-gen.py`` installed by DPDK,
+and found in the ``buildtools`` folder in the source tree.
+This section covers adding a command-line using this script to generate the boiler plate,
+while the following section,
+`Worked Example of Adding Command-line to an Application`_ covers the steps to do so manually.
+
+Creating a Command List File
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The ``dpdk-cmdline-gen.py`` script takes as input a list of commands to be used by the application.
+While these can be piped to it via standard input, using a list file is probably best.
+
+The format of the list file must be:
+
+* Comment lines start with '#' as first non-whitespace character
+
+* One command per line
+
+* Variable fields are prefixed by the type-name in angle-brackets, for example:
+
+  * ``<STRING>message``
+
+  * ``<UINT16>port_id``
+
+  * ``<IP>src_ip``
+
+* The help text for a command is given in the form of a comment on the same line as the command
+
+An example list file, with a variety of (unrelated) commands, is shown below::
+
+   # example list file
+   list                     # show all entries
+   add <UINT16>x <UINT16>y  # add x and y
+   echo <STRING>message     # print message to screen
+   add socket <STRING>path  # add unix socket with the given path
+   quit                     # close the application
+
+Running the Generator Script
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To generate the necessary definitions for a command-line, run ``dpdk-cmdline-gen.py`` passing the list file as parameter.
+The script will output the generated C code to standard output,
+the contents of which are in the form of a C header file.
+Optionally, an output filename may be specified via the ``-o/--output-file`` argument.
+
+The generated content includes:
+
+* The result structure definitions for each command
+
+* The token initializers for each structure field
+
+* An "extern" function prototype for the callback for each command
+
+* A parse context for each command, including the per-command comments as help string
+
+* A command-line context array definition, suitable for passing to ``cmdline_new``
+
+If so desired, the script can also output function stubs for the callback functions for each command.
+This behaviour is triggered by passing the ``--stubs`` flag to the script.
+In this case, an output file must be provided with a filename ending in ".h",
+and the callback stubs will be written to an equivalent ".c" file.
+
+.. note::
+
+   The stubs are written to a separate file,
+   to allow continuous use of the script to regenerate the command-line header,
+   without overwriting any code the user has added to the callback functions.
+   This makes it easy to incrementally add new commands to an existing application.
+
+Providing the Function Callbacks
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+As discussed above, the script output is a header file, containing structure definitions,
+but the callback functions themselves obviously have to be provided by the user.
+These callback functions must be provided as non-static functions in a C file,
+and named ``cmd_<cmdname>_parsed``.
+The function prototypes can be seen in the generated output header.
+
+The "cmdname" part of the function name is built up by combining the non-variable initial tokens in the command.
+So, given the commands in our worked example below: ``quit`` and ``show port stats <n>``,
+the callback functions would be:
+
+.. code:: c
+
+   void
+   cmd_quit_parsed(void *parsed_result, struct cmdline *cl, void *data)
+   {
+        ...
+   }
+
+   void
+   cmd_show_port_stats_parsed(void *parsed_result, struct cmdline *cl, void *data)
+   {
+        ...
+   }
+
+These functions must be provided by the developer, but, as stated above,
+stub functions may be generated by the script automatically using the ``--stubs`` parameter.
+
+The same "cmdname" stem is used in the naming of the generated structures too.
+To get at the results structure for each command above,
+the ``parsed_result`` parameter should be cast to ``struct cmd_quit_result``
+or ``struct cmd_show_port_stats_result`` respectively.
+
+Integrating with the Application
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To integrate the script output with the application,
+we must ``#include`` the generated header into our applications C file,
+and then have the command-line created via either ``cmdline_new`` or ``cmdline_stdin_new``.
+The first parameter to the function call should be the context array in the generated header file,
+``ctx`` by default. (Modifiable via script parameter).
+
+The callback functions may be in this same file, or in a separate one -
+they just need to be available to the linker at build-time.
+
+Limitations of the Script Approach
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The script approach works for most commands that a user may wish to add to an application.
+However, it does not support the full range of functions possible with the DPDK command-line library.
+For example,
+it is not possible using the script to multiplex multiple commands into a single callback function.
+To use this functionality, the user should follow the instructions in the next section
+`Worked Example of Adding Command-line to an Application`_ to manually configure a command-line instance.
+
+Worked Example of Adding Command-line to an Application
+-------------------------------------------------------
+
+The next few subsections will cover each of the steps listed in `Adding Command-line to an Application`_ in more detail,
 working through an example to add two commands to a command-line instance.
 Those two commands will be:
 
-- 
2.39.2

Re: [PATCH v5 2/9] buildtools: script to generate cmdline boilerplate

[Robin Jarry]

Bruce Richardson, Oct 17, 2023 at 14:13:
> Provide a "dpdk-cmdline-gen.py" script for application developers to
> quickly generate the boilerplate code necessary for using the cmdline
> library.
>
> Example of use:
> The script takes an input file with a list of commands the user wants in
> the app, where the parameter variables are tagged with the type.
> For example:
>
> 	$ cat commands.list
> 	list
> 	add <UINT16>x <UINT16>y
> 	echo <STRING>message
> 	add socket <STRING>path
> 	quit
>
> When run through the script as "./dpdk-cmdline-gen.py commands.list",
> the output will be the contents of a header file with all the
> boilerplate necessary for a commandline instance with those commands.
>
> If the flag --stubs is passed, an output header filename must also be
> passed, in which case both a header file with the definitions and a C
> file with function stubs in it is written to disk. The separation is so
> that the header file can be rewritten at any future point to add more
> commands, while the C file can be kept as-is and extended by the user
> with any additional functions needed.
>
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> ---
>  buildtools/dpdk-cmdline-gen.py    | 190 ++++++++++++++++++++++++++++++
>  buildtools/meson.build            |   7 ++
>  doc/guides/prog_guide/cmdline.rst | 131 +++++++++++++++++++-
>  3 files changed, 327 insertions(+), 1 deletion(-)
>  create mode 100755 buildtools/dpdk-cmdline-gen.py

Hi Bruce,

thanks for the respin! I have some small remarks inline.

> diff --git a/buildtools/dpdk-cmdline-gen.py b/buildtools/dpdk-cmdline-gen.py
> new file mode 100755
> index 0000000000..6cb7610de4
> --- /dev/null
> +++ b/buildtools/dpdk-cmdline-gen.py
> @@ -0,0 +1,190 @@
> +#!/usr/bin/env python3
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright(c) 2023 Intel Corporation
> +#
> +"""
> +Script to automatically generate boilerplate for using DPDK cmdline library.
> +"""
> +
> +import argparse
> +import sys
> +
> +PARSE_FN_PARAMS = "void *parsed_result, struct cmdline *cl, void *data"
> +PARSE_FN_BODY = """
> +    /* TODO: command action */
> +    RTE_SET_USED(parsed_result);
> +    RTE_SET_USED(cl);
> +    RTE_SET_USED(data);
> +"""
> +NUMERIC_TYPES = [
> +    "UINT8",
> +    "UINT16",
> +    "UINT32",
> +    "UINT64",
> +    "INT8",
> +    "INT16",
> +    "INT32",
> +    "INT64",
> +]
> +
> +
> +def process_command(lineno, tokens, comment):
> +    """Generate the structures and definitions for a single command."""
> +    out = []
> +    cfile_out = []
> +
> +    if tokens[0].startswith("<"):
> +        raise ValueError(f"Error line {lineno + 1}: command must start with a literal string")
> +
> +    name_tokens = []
> +    for t in tokens:
> +        if t.startswith("<"):
> +            break
> +        name_tokens.append(t)
> +    name = "_".join(name_tokens)
> +
> +    result_struct = []
> +    initializers = []
> +    token_list = []
> +    for t in tokens:
> +        if t.startswith("<"):
> +            t_type, t_name = t[1:].split(">")
> +            t_val = "NULL"
> +        else:
> +            t_type = "STRING"
> +            t_name = t
> +            t_val = f'"{t}"'
> +
> +        if t_type == "STRING":
> +            result_struct.append(f"\tcmdline_fixed_string_t {t_name};")
> +            initializers.append(
> +                f"static cmdline_parse_token_string_t cmd_{name}_{t_name}_tok =\n"
> +                + f"\tTOKEN_STRING_INITIALIZER(struct cmd_{name}_result, {t_name}, {t_val});"

Since you are now using multiline strings in process_commands(), why not 
use them everywhere?

It would make the code more readable in my opinion and would avoid 
inline f-string concatenation.

> +            )
> +        elif t_type in NUMERIC_TYPES:
> +            result_struct.append(f"\t{t_type.lower()}_t {t_name};")
> +            initializers.append(
> +                f"static cmdline_parse_token_num_t cmd_{name}_{t_name}_tok =\n"
> +                + f"\tTOKEN_NUM_INITIALIZER(struct cmd_{name}_result, {t_name}, RTE_{t_type});"
> +            )
> +        elif t_type in ["IP", "IP_ADDR", "IPADDR"]:
> +            result_struct.append(f"\tcmdline_ipaddr_t {t_name};")
> +            initializers.append(
> +                f"cmdline_parse_token_ipaddr_t cmd_{name}_{t_name}_tok =\n"
> +                + f"\tTOKEN_IPV4_INITIALIZER(struct cmd_{name}_result, {t_name});"
> +            )
> +        else:
> +            raise TypeError(f"Error line {lineno + 1}: unknown token type '{t_type}'")
> +        token_list.append(f"cmd_{name}_{t_name}_tok")
> +
> +    out.append(f'/* Auto-generated handling for command "{" ".join(tokens)}" */')
> +    # output function prototype
> +    func_sig = f"void\ncmd_{name}_parsed({PARSE_FN_PARAMS})"
> +    out.append(f"extern {func_sig};\n")
> +    # output result data structure
> +    out.append(f"struct cmd_{name}_result {{\n" + "\n".join(result_struct) + "\n};\n")
> +    # output the initializer tokens
> +    out.append("\n".join(initializers) + "\n")
> +    # output the instance structure
> +    out.append(
> +        f"static cmdline_parse_inst_t cmd_{name} = {{\n"
> +        + f"\t.f = cmd_{name}_parsed,\n"
> +        + "\t.data = NULL,\n"
> +        + f'\t.help_str = "{comment}",\n'
> +        + "\t.tokens = {"

Especially here :)

> +    )
> +    for t in token_list:
> +        out.append(f"\t\t(void *)&{t},")
> +    out.append("\t\tNULL\n" + "\t}\n" + "};\n")
> +    # output function template if C file being written
> +    cfile_out.append(f"{func_sig}\n{{{PARSE_FN_BODY}}}\n")
> +
> +    # return the instance structure name
> +    return (f"cmd_{name}", out, cfile_out)
> +
> +
> +def process_commands(infile, hfile, cfile, ctxname):
> +    """Generate boilerplate output for a list of commands from infile."""
> +    instances = []
> +
> +    hfile.write(
> +        f"""/* File autogenerated by {sys.argv[0]} */
> +#ifndef GENERATED_COMMANDS_H
> +#define GENERATED_COMMANDS_H
> +#include <rte_common.h>
> +#include <cmdline.h>
> +#include <cmdline_parse_string.h>
> +#include <cmdline_parse_num.h>
> +#include <cmdline_parse_ipaddr.h>
> +
> +"""
> +    )
> +
> +    for lineno, line in enumerate(infile.readlines()):
> +        if line.lstrip().startswith("#"):
> +            continue
> +        if "#" not in line:
> +            line = line + "#"  # ensure split always works, even if no help text
> +        tokens, comment = line.split("#", 1)
> +        cmd_inst, h_out, c_out = process_command(lineno, tokens.strip().split(), comment.strip())
> +        hfile.write("\n".join(h_out))
> +        if cfile:
> +            cfile.write("\n".join(c_out))
> +        instances.append(cmd_inst)
> +
> +    inst_join_str = ",\n\t&"
> +    hfile.write(
> +        f"""
> +static __rte_used cmdline_parse_ctx_t {ctxname}[] = {{
> +\t&{inst_join_str.join(instances)},
> +\tNULL
> +}};
> +
> +#endif /* GENERATED_COMMANDS_H */
> +"""

By the way, you can put literal tabs in the multiline strings. That way 
the indentation will look the same than in the generated C code.

        f"""
static __rte_used cmdline_parse_ctx_t {ctxname}[] = {{
	&{inst_join_str.join(instances)},
	NULL
}};

#endif /* GENERATED_COMMANDS_H */
"""

> +    )
> +
> +
> +def main():
> +    """Application main entry point."""
> +    ap = argparse.ArgumentParser(description=__doc__)
> +    ap.add_argument(
> +        "--stubs",
> +        action="store_true",
> +        help="Produce C file with empty function stubs for each command",
> +    )
> +    ap.add_argument(
> +        "--output-file",
> +        "-o",
> +        default="-",
> +        help="Output header filename [default to stdout]",
> +    )
> +    ap.add_argument(
> +        "--context-name",
> +        default="ctx",
> +        help="Name given to the cmdline context variable in the output header [default=ctx]",
> +    )
> +    ap.add_argument("infile", type=argparse.FileType("r"), help="File with list of commands")
> +    args = ap.parse_args()
> +
> +    if not args.stubs:
> +        if args.output_file == "-":
> +            process_commands(args.infile, sys.stdout, None, args.context_name)
> +        else:
> +            with open(args.output_file, "w") as hfile:
> +                process_commands(args.infile, hfile, None, args.context_name)
> +    else:
> +        if not args.output_file.endswith(".h"):
> +            ap.error(
> +                "-o/--output-file: specify an output filename ending with .h when creating stubs"
> +            )
> +
> +        cfilename = args.output_file[:-2] + ".c"
> +        with open(args.output_file, "w") as hfile:
> +            with open(cfilename, "w") as cfile:
> +                cfile.write(f'#include "{args.output_file}"\n\n')
> +                process_commands(args.infile, hfile, cfile, args.context_name)
> +
> +
> +if __name__ == "__main__":
> +    main()


-- 
Robin Jarry
Principal Software Engineer
Red Hat, Telco/NFV

Re: [PATCH v5 2/9] buildtools: script to generate cmdline boilerplate

[Bruce Richardson]

On Wed, Oct 25, 2023 at 03:04:05PM +0200, Robin Jarry wrote:
> Bruce Richardson, Oct 17, 2023 at 14:13:
> > Provide a "dpdk-cmdline-gen.py" script for application developers to
> > quickly generate the boilerplate code necessary for using the cmdline
> > library.
> > 
> > Example of use:
> > The script takes an input file with a list of commands the user wants in
> > the app, where the parameter variables are tagged with the type.
> > For example:
> > 
> > 	$ cat commands.list
> > 	list
> > 	add <UINT16>x <UINT16>y
> > 	echo <STRING>message
> > 	add socket <STRING>path
> > 	quit
> > 
> > When run through the script as "./dpdk-cmdline-gen.py commands.list",
> > the output will be the contents of a header file with all the
> > boilerplate necessary for a commandline instance with those commands.
> > 
> > If the flag --stubs is passed, an output header filename must also be
> > passed, in which case both a header file with the definitions and a C
> > file with function stubs in it is written to disk. The separation is so
> > that the header file can be rewritten at any future point to add more
> > commands, while the C file can be kept as-is and extended by the user
> > with any additional functions needed.
> > 
> > Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> > ---
> >  buildtools/dpdk-cmdline-gen.py    | 190 ++++++++++++++++++++++++++++++
> >  buildtools/meson.build            |   7 ++
> >  doc/guides/prog_guide/cmdline.rst | 131 +++++++++++++++++++-
> >  3 files changed, 327 insertions(+), 1 deletion(-)
> >  create mode 100755 buildtools/dpdk-cmdline-gen.py
> 
> Hi Bruce,
> 
> thanks for the respin! I have some small remarks inline.
> 
> > diff --git a/buildtools/dpdk-cmdline-gen.py b/buildtools/dpdk-cmdline-gen.py
> > new file mode 100755
> > index 0000000000..6cb7610de4
> > --- /dev/null
> > +++ b/buildtools/dpdk-cmdline-gen.py
> > @@ -0,0 +1,190 @@
> > +#!/usr/bin/env python3
> > +# SPDX-License-Identifier: BSD-3-Clause
> > +# Copyright(c) 2023 Intel Corporation
> > +#
> > +"""
> > +Script to automatically generate boilerplate for using DPDK cmdline library.
> > +"""
> > +
> > +import argparse
> > +import sys
> > +
> > +PARSE_FN_PARAMS = "void *parsed_result, struct cmdline *cl, void *data"
> > +PARSE_FN_BODY = """
> > +    /* TODO: command action */
> > +    RTE_SET_USED(parsed_result);
> > +    RTE_SET_USED(cl);
> > +    RTE_SET_USED(data);
> > +"""
> > +NUMERIC_TYPES = [
> > +    "UINT8",
> > +    "UINT16",
> > +    "UINT32",
> > +    "UINT64",
> > +    "INT8",
> > +    "INT16",
> > +    "INT32",
> > +    "INT64",
> > +]
> > +
> > +
> > +def process_command(lineno, tokens, comment):
> > +    """Generate the structures and definitions for a single command."""
> > +    out = []
> > +    cfile_out = []
> > +
> > +    if tokens[0].startswith("<"):
> > +        raise ValueError(f"Error line {lineno + 1}: command must start with a literal string")
> > +
> > +    name_tokens = []
> > +    for t in tokens:
> > +        if t.startswith("<"):
> > +            break
> > +        name_tokens.append(t)
> > +    name = "_".join(name_tokens)
> > +
> > +    result_struct = []
> > +    initializers = []
> > +    token_list = []
> > +    for t in tokens:
> > +        if t.startswith("<"):
> > +            t_type, t_name = t[1:].split(">")
> > +            t_val = "NULL"
> > +        else:
> > +            t_type = "STRING"
> > +            t_name = t
> > +            t_val = f'"{t}"'
> > +
> > +        if t_type == "STRING":
> > +            result_struct.append(f"\tcmdline_fixed_string_t {t_name};")
> > +            initializers.append(
> > +                f"static cmdline_parse_token_string_t cmd_{name}_{t_name}_tok =\n"
> > +                + f"\tTOKEN_STRING_INITIALIZER(struct cmd_{name}_result, {t_name}, {t_val});"
> 
> Since you are now using multiline strings in process_commands(), why not use
> them everywhere?
> 
> It would make the code more readable in my opinion and would avoid inline
> f-string concatenation.
> 

I'm a bit unsure about this case. I notice I can at least remove the "+"
symbol and have implicit string concat, but I really don't like the way the
indentation gets adjusted when we use multi-line strings, since the indent
has to match the C-code indent rather than the python indentation levels.

Therefore, I'm going to leave these pairs of lines as they are.

> > +            )
> > +        elif t_type in NUMERIC_TYPES:
> > +            result_struct.append(f"\t{t_type.lower()}_t {t_name};")
> > +            initializers.append(
> > +                f"static cmdline_parse_token_num_t cmd_{name}_{t_name}_tok =\n"
> > +                + f"\tTOKEN_NUM_INITIALIZER(struct cmd_{name}_result, {t_name}, RTE_{t_type});"
> > +            )
> > +        elif t_type in ["IP", "IP_ADDR", "IPADDR"]:
> > +            result_struct.append(f"\tcmdline_ipaddr_t {t_name};")
> > +            initializers.append(
> > +                f"cmdline_parse_token_ipaddr_t cmd_{name}_{t_name}_tok =\n"
> > +                + f"\tTOKEN_IPV4_INITIALIZER(struct cmd_{name}_result, {t_name});"
> > +            )
> > +        else:
> > +            raise TypeError(f"Error line {lineno + 1}: unknown token type '{t_type}'")
> > +        token_list.append(f"cmd_{name}_{t_name}_tok")
> > +
> > +    out.append(f'/* Auto-generated handling for command "{" ".join(tokens)}" */')
> > +    # output function prototype
> > +    func_sig = f"void\ncmd_{name}_parsed({PARSE_FN_PARAMS})"
> > +    out.append(f"extern {func_sig};\n")
> > +    # output result data structure
> > +    out.append(f"struct cmd_{name}_result {{\n" + "\n".join(result_struct) + "\n};\n")
> > +    # output the initializer tokens
> > +    out.append("\n".join(initializers) + "\n")
> > +    # output the instance structure
> > +    out.append(
> > +        f"static cmdline_parse_inst_t cmd_{name} = {{\n"
> > +        + f"\t.f = cmd_{name}_parsed,\n"
> > +        + "\t.data = NULL,\n"
> > +        + f'\t.help_str = "{comment}",\n'
> > +        + "\t.tokens = {"
> 
> Especially here :)
> 

I'll see about converting this and what it looks like. Maybe see if
textwrap.dedent will allow sensible indentation with it.

> > +    )
> > +    for t in token_list:
> > +        out.append(f"\t\t(void *)&{t},")
> > +    out.append("\t\tNULL\n" + "\t}\n" + "};\n")
> > +    # output function template if C file being written
> > +    cfile_out.append(f"{func_sig}\n{{{PARSE_FN_BODY}}}\n")
> > +
> > +    # return the instance structure name
> > +    return (f"cmd_{name}", out, cfile_out)
> > +
> > +
> > +def process_commands(infile, hfile, cfile, ctxname):
> > +    """Generate boilerplate output for a list of commands from infile."""
> > +    instances = []
> > +
> > +    hfile.write(
> > +        f"""/* File autogenerated by {sys.argv[0]} */
> > +#ifndef GENERATED_COMMANDS_H
> > +#define GENERATED_COMMANDS_H
> > +#include <rte_common.h>
> > +#include <cmdline.h>
> > +#include <cmdline_parse_string.h>
> > +#include <cmdline_parse_num.h>
> > +#include <cmdline_parse_ipaddr.h>
> > +
> > +"""
> > +    )
> > +
> > +    for lineno, line in enumerate(infile.readlines()):
> > +        if line.lstrip().startswith("#"):
> > +            continue
> > +        if "#" not in line:
> > +            line = line + "#"  # ensure split always works, even if no help text
> > +        tokens, comment = line.split("#", 1)
> > +        cmd_inst, h_out, c_out = process_command(lineno, tokens.strip().split(), comment.strip())
> > +        hfile.write("\n".join(h_out))
> > +        if cfile:
> > +            cfile.write("\n".join(c_out))
> > +        instances.append(cmd_inst)
> > +
> > +    inst_join_str = ",\n\t&"
> > +    hfile.write(
> > +        f"""
> > +static __rte_used cmdline_parse_ctx_t {ctxname}[] = {{
> > +\t&{inst_join_str.join(instances)},
> > +\tNULL
> > +}};
> > +
> > +#endif /* GENERATED_COMMANDS_H */
> > +"""
> 
> By the way, you can put literal tabs in the multiline strings. That way the
> indentation will look the same than in the generated C code.
> 
>        f"""
> static __rte_used cmdline_parse_ctx_t {ctxname}[] = {{
> 	&{inst_join_str.join(instances)},
> 	NULL
> }};
> 
> #endif /* GENERATED_COMMANDS_H */
> """
> 

Trouble with that is that for some editors when using python, the tabs are
automatically expanded with spaces. For me using eclipse right now, I
physically can't insert leading tabs into the file! I need to open in vim
or some other editor to do it (or maybe use copy-paste from a C file), and
while that's not a big deal for me, it will be a problem for anyone else
editing this in future who has similar settings.

/Bruce

[PATCH v5 3/9] ci: allow use of DPDK tools when building examples

[Bruce Richardson]

To allow use of the DPDK python scripts (installed in $(prefix)/bin)
from within the makefiles of our examples, we need to export the PATH
variable with the location of our installed scripts from within our CI
scripts. This matches what is already done for other paths e.g. the
PKG_CONFIG_PATH variable.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 .ci/linux-build.sh | 1 +
 1 file changed, 1 insertion(+)

diff --git a/.ci/linux-build.sh b/.ci/linux-build.sh
index e0b62bac90..3db9d9de6e 100755
--- a/.ci/linux-build.sh
+++ b/.ci/linux-build.sh
@@ -174,6 +174,7 @@ fi
 if [ "$BUILD_EXAMPLES" = "true" ]; then
     [ -d install ] || DESTDIR=$(pwd)/install meson install -C build
     export LD_LIBRARY_PATH=$(dirname $(find $(pwd)/install -name librte_eal.so)):$LD_LIBRARY_PATH
+    export PATH=$(dirname $(find $(pwd)/install -name dpdk-devbind.py)):$PATH
     export PKG_CONFIG_PATH=$(dirname $(find $(pwd)/install -name libdpdk.pc)):$PKG_CONFIG_PATH
     export PKGCONF="pkg-config --define-prefix"
     find build/examples -maxdepth 1 -type f -name "dpdk-*" |
-- 
2.39.2

Re: [PATCH v5 3/9] ci: allow use of DPDK tools when building examples

[Aaron Conole]

Bruce Richardson <bruce.richardson@intel.com> writes:

> To allow use of the DPDK python scripts (installed in $(prefix)/bin)
> from within the makefiles of our examples, we need to export the PATH
> variable with the location of our installed scripts from within our CI
> scripts. This matches what is already done for other paths e.g. the
> PKG_CONFIG_PATH variable.
>
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> ---

Reviewed-by: Aaron Conole <aconole@redhat.com>

[PATCH v5 4/9] examples/simple_mp: auto-generate cmdline boilerplate

[Bruce Richardson]

Use the dpdk-cmdline-gen script to autogenerate all the boilerplate
structs and defines for the commandline part of the app.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 examples/multi_process/simple_mp/Makefile     |  12 +-
 examples/multi_process/simple_mp/meson.build  |   9 ++
 .../multi_process/simple_mp/mp_commands.c     | 106 ++----------------
 .../multi_process/simple_mp/mp_commands.h     |  14 ---
 .../multi_process/simple_mp/mp_commands.list  |   3 +
 5 files changed, 30 insertions(+), 114 deletions(-)
 delete mode 100644 examples/multi_process/simple_mp/mp_commands.h
 create mode 100644 examples/multi_process/simple_mp/mp_commands.list

diff --git a/examples/multi_process/simple_mp/Makefile b/examples/multi_process/simple_mp/Makefile
index 1d0a260e64..890b6b7e62 100644
--- a/examples/multi_process/simple_mp/Makefile
+++ b/examples/multi_process/simple_mp/Makefile
@@ -6,6 +6,7 @@ APP = simple_mp
 
 # all source are stored in SRCS-y
 SRCS-y := main.c mp_commands.c
+SRC-DEPS := build/mp_commands.h
 
 PKGCONF ?= pkg-config
 
@@ -22,10 +23,13 @@ static: build/$(APP)-static
 	ln -sf $(APP)-static build/$(APP)
 
 PC_FILE := $(shell $(PKGCONF) --path libdpdk 2>/dev/null)
-CFLAGS += -O3 $(shell $(PKGCONF) --cflags libdpdk)
+CFLAGS += -O3 $(shell $(PKGCONF) --cflags libdpdk) -I build/
 LDFLAGS_SHARED = $(shell $(PKGCONF) --libs libdpdk)
 LDFLAGS_STATIC = $(shell $(PKGCONF) --static --libs libdpdk)
 
+build/mp_commands.h: mp_commands.list Makefile
+	dpdk-cmdline-gen.py -o $@ --context-name=simple_mp_ctx $<
+
 ifeq ($(MAKECMDGOALS),static)
 # check for broken pkg-config
 ifeq ($(shell echo $(LDFLAGS_STATIC) | grep 'whole-archive.*l:lib.*no-whole-archive'),)
@@ -36,10 +40,10 @@ endif
 
 CFLAGS += -DALLOW_EXPERIMENTAL_API
 
-build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
+build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build $(SRC-DEPS)
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
 
-build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build
+build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build $(SRC-DEPS)
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_STATIC)
 
 build:
@@ -47,5 +51,5 @@ build:
 
 .PHONY: clean
 clean:
-	rm -f build/$(APP) build/$(APP)-static build/$(APP)-shared
+	rm -f build/$(APP) build/$(APP)-static build/$(APP)-shared build/mp_commands.h
 	test -d build && rmdir -p build || true
diff --git a/examples/multi_process/simple_mp/meson.build b/examples/multi_process/simple_mp/meson.build
index 359af4384d..e99b7a3f6f 100644
--- a/examples/multi_process/simple_mp/meson.build
+++ b/examples/multi_process/simple_mp/meson.build
@@ -7,7 +7,16 @@
 # DPDK instance, use 'make'
 
 allow_experimental_apis = true
+
+cmd_h = custom_target('commands_hdr',
+	output: 'mp_commands.h',
+	input: files('mp_commands.list'),
+	capture: true,
+	command: [cmdline_gen_cmd, '--context-name=simple_mp_ctx', '@INPUT@']
+)
+
 sources = files(
         'mp_commands.c',
         'main.c',
 )
+sources += cmd_h
diff --git a/examples/multi_process/simple_mp/mp_commands.c b/examples/multi_process/simple_mp/mp_commands.c
index a5f91b00be..df9fa94208 100644
--- a/examples/multi_process/simple_mp/mp_commands.c
+++ b/examples/multi_process/simple_mp/mp_commands.c
@@ -1,44 +1,18 @@
 /* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2010-2014 Intel Corporation
+ * Copyright(c) 2010-2023 Intel Corporation
  */
-#include <stdint.h>
-#include <string.h>
-#include <stdlib.h>
-#include <stdarg.h>
-#include <inttypes.h>
-#include <stdio.h>
-#include <termios.h>
-#include <errno.h>
-#include <sys/queue.h>
-
-#include <rte_common.h>
-#include <rte_memory.h>
-#include <rte_eal.h>
-#include <rte_branch_prediction.h>
-#include <rte_launch.h>
-#include <rte_log.h>
-#include <rte_per_lcore.h>
-#include <rte_lcore.h>
 #include <rte_ring.h>
-#include <rte_debug.h>
 #include <rte_mempool.h>
 #include <rte_string_fns.h>
 
-#include <cmdline_rdline.h>
-#include <cmdline_parse.h>
-#include <cmdline_parse_string.h>
-#include <cmdline_socket.h>
-#include <cmdline.h>
 #include "mp_commands.h"
 
-/**********************************************************/
-
-struct cmd_send_result {
-	cmdline_fixed_string_t action;
-	cmdline_fixed_string_t message;
-};
+extern struct rte_ring *send_ring, *recv_ring;
+extern struct rte_mempool *message_pool;
+extern volatile int quit;
 
-static void cmd_send_parsed(void *parsed_result,
+void
+cmd_send_parsed(void *parsed_result,
 		__rte_unused struct cmdline *cl,
 		__rte_unused void *data)
 {
@@ -54,29 +28,8 @@ static void cmd_send_parsed(void *parsed_result,
 	}
 }
 
-cmdline_parse_token_string_t cmd_send_action =
-	TOKEN_STRING_INITIALIZER(struct cmd_send_result, action, "send");
-cmdline_parse_token_string_t cmd_send_message =
-	TOKEN_STRING_INITIALIZER(struct cmd_send_result, message, NULL);
-
-cmdline_parse_inst_t cmd_send = {
-	.f = cmd_send_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "send a string to another process",
-	.tokens = {        /* token list, NULL terminated */
-			(void *)&cmd_send_action,
-			(void *)&cmd_send_message,
-			NULL,
-	},
-};
-
-/**********************************************************/
-
-struct cmd_quit_result {
-	cmdline_fixed_string_t quit;
-};
-
-static void cmd_quit_parsed(__rte_unused void *parsed_result,
+void
+cmd_quit_parsed(__rte_unused void *parsed_result,
 			    struct cmdline *cl,
 			    __rte_unused void *data)
 {
@@ -84,26 +37,8 @@ static void cmd_quit_parsed(__rte_unused void *parsed_result,
 	cmdline_quit(cl);
 }
 
-cmdline_parse_token_string_t cmd_quit_quit =
-	TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit");
-
-cmdline_parse_inst_t cmd_quit = {
-	.f = cmd_quit_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "close the application",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_quit_quit,
-		NULL,
-	},
-};
-
-/**********************************************************/
-
-struct cmd_help_result {
-	cmdline_fixed_string_t help;
-};
-
-static void cmd_help_parsed(__rte_unused void *parsed_result,
+void
+cmd_help_parsed(__rte_unused void *parsed_result,
 			    struct cmdline *cl,
 			    __rte_unused void *data)
 {
@@ -112,24 +47,3 @@ static void cmd_help_parsed(__rte_unused void *parsed_result,
 			"send commands to the simple app. Commands supported are:\n\n"
 			"- send [string]\n" "- help\n" "- quit\n\n");
 }
-
-cmdline_parse_token_string_t cmd_help_help =
-	TOKEN_STRING_INITIALIZER(struct cmd_help_result, help, "help");
-
-cmdline_parse_inst_t cmd_help = {
-	.f = cmd_help_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "show help",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_help_help,
-		NULL,
-	},
-};
-
-/****** CONTEXT (list of instruction) */
-cmdline_parse_ctx_t simple_mp_ctx[] = {
-		(cmdline_parse_inst_t *)&cmd_send,
-		(cmdline_parse_inst_t *)&cmd_quit,
-		(cmdline_parse_inst_t *)&cmd_help,
-	NULL,
-};
diff --git a/examples/multi_process/simple_mp/mp_commands.h b/examples/multi_process/simple_mp/mp_commands.h
deleted file mode 100644
index 5d67413e7c..0000000000
--- a/examples/multi_process/simple_mp/mp_commands.h
+++ /dev/null
@@ -1,14 +0,0 @@
-/* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2010-2014 Intel Corporation
- */
-
-#ifndef _SIMPLE_MP_COMMANDS_H_
-#define _SIMPLE_MP_COMMANDS_H_
-
-extern struct rte_ring *send_ring;
-extern struct rte_mempool *message_pool;
-extern volatile int quit;
-
-extern cmdline_parse_ctx_t simple_mp_ctx[];
-
-#endif /* _SIMPLE_MP_COMMANDS_H_ */
diff --git a/examples/multi_process/simple_mp/mp_commands.list b/examples/multi_process/simple_mp/mp_commands.list
new file mode 100644
index 0000000000..c8890cb071
--- /dev/null
+++ b/examples/multi_process/simple_mp/mp_commands.list
@@ -0,0 +1,3 @@
+send <STRING>message  # send a string to another process
+help                  # show help
+quit                  # close the application
-- 
2.39.2

[PATCH v5 5/9] examples/hotplug_mp: auto-generate cmdline boilerplate

[Bruce Richardson]

Use the dpdk-cmdline-gen script to autogenerate all the boilerplate
structs and defines for the commandline part of the app.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 examples/multi_process/hotplug_mp/Makefile    |  12 +-
 examples/multi_process/hotplug_mp/commands.c  | 147 ++----------------
 examples/multi_process/hotplug_mp/commands.h  |  10 --
 .../multi_process/hotplug_mp/commands.list    |   8 +
 examples/multi_process/hotplug_mp/meson.build |   9 ++
 5 files changed, 38 insertions(+), 148 deletions(-)
 delete mode 100644 examples/multi_process/hotplug_mp/commands.h
 create mode 100644 examples/multi_process/hotplug_mp/commands.list

diff --git a/examples/multi_process/hotplug_mp/Makefile b/examples/multi_process/hotplug_mp/Makefile
index 6b20d6e49a..81ee85cd6b 100644
--- a/examples/multi_process/hotplug_mp/Makefile
+++ b/examples/multi_process/hotplug_mp/Makefile
@@ -6,6 +6,7 @@ APP = hotplug_mp
 
 # all source are stored in SRCS-y
 SRCS-y := main.c commands.c
+SRC-DEPS := build/commands.h
 
 PKGCONF ?= pkg-config
 
@@ -22,10 +23,13 @@ static: build/$(APP)-static
 	ln -sf $(APP)-static build/$(APP)
 
 PC_FILE := $(shell $(PKGCONF) --path libdpdk 2>/dev/null)
-CFLAGS += -O3 $(shell $(PKGCONF) --cflags libdpdk)
+CFLAGS += -O3 $(shell $(PKGCONF) --cflags libdpdk) -I build/
 LDFLAGS_SHARED = $(shell $(PKGCONF) --libs libdpdk)
 LDFLAGS_STATIC = $(shell $(PKGCONF) --static --libs libdpdk)
 
+build/commands.h: commands.list Makefile
+	dpdk-cmdline-gen.py -o $@ --context-name=main_ctx $<
+
 ifeq ($(MAKECMDGOALS),static)
 # check for broken pkg-config
 ifeq ($(shell echo $(LDFLAGS_STATIC) | grep 'whole-archive.*l:lib.*no-whole-archive'),)
@@ -36,10 +40,10 @@ endif
 
 CFLAGS += -DALLOW_EXPERIMENTAL_API
 
-build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
+build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build $(SRC-DEPS)
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
 
-build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build
+build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build  $(SRC-DEPS)
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_STATIC)
 
 build:
@@ -47,5 +51,5 @@ build:
 
 .PHONY: clean
 clean:
-	rm -f build/$(APP) build/$(APP)-static build/$(APP)-shared
+	rm -f build/$(APP) build/$(APP)-static build/$(APP)-shared build/commands.h
 	test -d build && rmdir -p build || true
diff --git a/examples/multi_process/hotplug_mp/commands.c b/examples/multi_process/hotplug_mp/commands.c
index 88f44e00a0..900eb9f774 100644
--- a/examples/multi_process/hotplug_mp/commands.c
+++ b/examples/multi_process/hotplug_mp/commands.c
@@ -1,24 +1,12 @@
 /* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2018 Intel Corporation.
+ * Copyright(c) 2018-2023 Intel Corporation.
  */
 
-#include <cmdline_rdline.h>
-#include <cmdline_parse.h>
-#include <cmdline_parse_ipaddr.h>
-#include <cmdline_parse_num.h>
-#include <cmdline_parse_string.h>
-#include <cmdline.h>
-
 #include <rte_bus.h>
 #include <rte_ethdev.h>
+#include "commands.h"
 
-/**********************************************************/
-
-struct cmd_help_result {
-	cmdline_fixed_string_t help;
-};
-
-static void cmd_help_parsed(__rte_unused void *parsed_result,
+void cmd_help_parsed(__rte_unused void *parsed_result,
 			    struct cmdline *cl,
 			    __rte_unused void *data)
 {
@@ -29,52 +17,16 @@ static void cmd_help_parsed(__rte_unused void *parsed_result,
 		       "- list\n\n");
 }
 
-cmdline_parse_token_string_t cmd_help_help =
-	TOKEN_STRING_INITIALIZER(struct cmd_help_result, help, "help");
-
-cmdline_parse_inst_t cmd_help = {
-	.f = cmd_help_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "show help",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_help_help,
-		NULL,
-	},
-};
-
-/**********************************************************/
-
-struct cmd_quit_result {
-	cmdline_fixed_string_t quit;
-};
-
-static void cmd_quit_parsed(__rte_unused void *parsed_result,
+void
+cmd_quit_parsed(__rte_unused void *parsed_result,
 			    struct cmdline *cl,
 			    __rte_unused void *data)
 {
 	cmdline_quit(cl);
 }
 
-cmdline_parse_token_string_t cmd_quit_quit =
-	TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit");
-
-cmdline_parse_inst_t cmd_quit = {
-	.f = cmd_quit_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "quit",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_quit_quit,
-		NULL,
-	},
-};
-
-/**********************************************************/
-
-struct cmd_list_result {
-	cmdline_fixed_string_t list;
-};
-
-static void cmd_list_parsed(__rte_unused void *parsed_result,
+void
+cmd_list_parsed(__rte_unused void *parsed_result,
 			    struct cmdline *cl,
 			    __rte_unused void *data)
 {
@@ -92,31 +44,12 @@ static void cmd_list_parsed(__rte_unused void *parsed_result,
 	}
 }
 
-cmdline_parse_token_string_t cmd_list_list =
-	TOKEN_STRING_INITIALIZER(struct cmd_list_result, list, "list");
-
-cmdline_parse_inst_t cmd_list = {
-	.f = cmd_list_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "list all devices",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_list_list,
-		NULL,
-	},
-};
-
-/**********************************************************/
-
-struct cmd_dev_attach_result {
-	cmdline_fixed_string_t attach;
-	cmdline_fixed_string_t devargs;
-};
-
-static void cmd_dev_attach_parsed(void *parsed_result,
+void
+cmd_attach_parsed(void *parsed_result,
 				  struct cmdline *cl,
 				  __rte_unused void *data)
 {
-	struct cmd_dev_attach_result *res = parsed_result;
+	struct cmd_attach_result *res = parsed_result;
 	struct rte_devargs da;
 
 	memset(&da, 0, sizeof(da));
@@ -134,35 +67,12 @@ static void cmd_dev_attach_parsed(void *parsed_result,
 	rte_devargs_reset(&da);
 }
 
-cmdline_parse_token_string_t cmd_dev_attach_attach =
-	TOKEN_STRING_INITIALIZER(struct cmd_dev_attach_result, attach,
-				 "attach");
-cmdline_parse_token_string_t cmd_dev_attach_devargs =
-	TOKEN_STRING_INITIALIZER(struct cmd_dev_attach_result, devargs, NULL);
-
-cmdline_parse_inst_t cmd_attach_device = {
-	.f = cmd_dev_attach_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "attach a device",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_dev_attach_attach,
-		(void *)&cmd_dev_attach_devargs,
-		NULL,
-	},
-};
-
-/**********************************************************/
-
-struct cmd_dev_detach_result {
-	cmdline_fixed_string_t detach;
-	cmdline_fixed_string_t devargs;
-};
-
-static void cmd_dev_detach_parsed(void *parsed_result,
+void
+cmd_detach_parsed(void *parsed_result,
 				   struct cmdline *cl,
 				   __rte_unused void *data)
 {
-	struct cmd_dev_detach_result *res = parsed_result;
+	struct cmd_detach_result *res = parsed_result;
 	struct rte_devargs da;
 
 	memset(&da, 0, sizeof(da));
@@ -181,34 +91,3 @@ static void cmd_dev_detach_parsed(void *parsed_result,
 			da.name);
 	rte_devargs_reset(&da);
 }
-
-cmdline_parse_token_string_t cmd_dev_detach_detach =
-	TOKEN_STRING_INITIALIZER(struct cmd_dev_detach_result, detach,
-				 "detach");
-
-cmdline_parse_token_string_t cmd_dev_detach_devargs =
-	TOKEN_STRING_INITIALIZER(struct cmd_dev_detach_result, devargs, NULL);
-
-cmdline_parse_inst_t cmd_detach_device = {
-	.f = cmd_dev_detach_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "detach a device",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_dev_detach_detach,
-		(void *)&cmd_dev_detach_devargs,
-		NULL,
-	},
-};
-
-/**********************************************************/
-/**********************************************************/
-/****** CONTEXT (list of instruction) */
-
-cmdline_parse_ctx_t main_ctx[] = {
-	(cmdline_parse_inst_t *)&cmd_help,
-	(cmdline_parse_inst_t *)&cmd_quit,
-	(cmdline_parse_inst_t *)&cmd_list,
-	(cmdline_parse_inst_t *)&cmd_attach_device,
-	(cmdline_parse_inst_t *)&cmd_detach_device,
-	NULL,
-};
diff --git a/examples/multi_process/hotplug_mp/commands.h b/examples/multi_process/hotplug_mp/commands.h
deleted file mode 100644
index afcf177dba..0000000000
--- a/examples/multi_process/hotplug_mp/commands.h
+++ /dev/null
@@ -1,10 +0,0 @@
-/* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2018 Intel Corporation
- */
-
-#ifndef _COMMANDS_H_
-#define _COMMANDS_H_
-
-extern cmdline_parse_ctx_t main_ctx[];
-
-#endif /* _COMMANDS_H_ */
diff --git a/examples/multi_process/hotplug_mp/commands.list b/examples/multi_process/hotplug_mp/commands.list
new file mode 100644
index 0000000000..8064df77c0
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/commands.list
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2023 Intel Corporation
+#
+attach <STRING>devargs     # attach a device
+detach <STRING>devargs     # detach a device
+list                       # list all devices
+help                       # show help
+quit                       # quit
diff --git a/examples/multi_process/hotplug_mp/meson.build b/examples/multi_process/hotplug_mp/meson.build
index a1ad98ca2e..7a0e9ca47a 100644
--- a/examples/multi_process/hotplug_mp/meson.build
+++ b/examples/multi_process/hotplug_mp/meson.build
@@ -7,7 +7,16 @@
 # DPDK instance, use 'make'
 
 allow_experimental_apis = true
+
+cmd_h = custom_target('commands_hdr',
+	output: 'commands.h',
+	input: files('commands.list'),
+	capture: true,
+	command: [cmdline_gen_cmd, '--context-name=main_ctx', '@INPUT@']
+)
+
 sources = files(
         'commands.c',
         'main.c',
 )
+sources += cmd_h
-- 
2.39.2

[PATCH v5 6/9] examples/bond: auto-generate cmdline boilerplate

[Bruce Richardson]

Use the dpdk-cmdline-gen script to autogenerate all the boilerplate
structs and defines for the commandline part of the app.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>

---
Note: the original help text on some of the commands in this example
  were not useful "this command do not handle any arguments". Therefore,
  when converting over to the command script, the per-command help
  info has been updated with reference to the code rather than a literal
  transfer of the existing help text, as was done with the previous 2
  example apps.
---
 examples/bond/Makefile                        |  12 +-
 examples/bond/commands.list                   |   6 +
 examples/bond/main.c                          | 161 ++----------------
 examples/bond/main.h                          |  10 --
 examples/bond/meson.build                     |   8 +
 .../multi_process/simple_mp/mp_commands.list  |   3 +
 6 files changed, 40 insertions(+), 160 deletions(-)
 create mode 100644 examples/bond/commands.list
 delete mode 100644 examples/bond/main.h

diff --git a/examples/bond/Makefile b/examples/bond/Makefile
index ad711a5bee..d87c7a32ba 100644
--- a/examples/bond/Makefile
+++ b/examples/bond/Makefile
@@ -6,6 +6,7 @@ APP = bond_app
 
 # all source are stored in SRCS-y
 SRCS-y := main.c
+SRC-DEPS := build/commands.h
 
 PKGCONF ?= pkg-config
 
@@ -24,10 +25,13 @@ static: build/$(APP)-static
 LDFLAGS += -lrte_net_bond
 
 PC_FILE := $(shell $(PKGCONF) --path libdpdk 2>/dev/null)
-CFLAGS += -O3 $(shell $(PKGCONF) --cflags libdpdk)
+CFLAGS += -O3 $(shell $(PKGCONF) --cflags libdpdk) -I build/
 LDFLAGS_SHARED = $(shell $(PKGCONF) --libs libdpdk)
 LDFLAGS_STATIC = $(shell $(PKGCONF) --static --libs libdpdk)
 
+build/commands.h: commands.list Makefile
+	dpdk-cmdline-gen.py -o $@ --context-name=main_ctx $<
+
 ifeq ($(MAKECMDGOALS),static)
 # check for broken pkg-config
 ifeq ($(shell echo $(LDFLAGS_STATIC) | grep 'whole-archive.*l:lib.*no-whole-archive'),)
@@ -38,10 +42,10 @@ endif
 
 CFLAGS += -DALLOW_EXPERIMENTAL_API
 
-build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
+build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build $(SRC-DEPS)
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
 
-build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build
+build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build  $(SRC-DEPS)
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_STATIC)
 
 build:
@@ -49,5 +53,5 @@ build:
 
 .PHONY: clean
 clean:
-	rm -f build/$(APP) build/$(APP)-static build/$(APP)-shared
+	rm -f build/$(APP) build/$(APP)-static build/$(APP)-shared build/*.h
 	test -d build && rmdir -p build || true
diff --git a/examples/bond/commands.list b/examples/bond/commands.list
new file mode 100644
index 0000000000..a10bf75708
--- /dev/null
+++ b/examples/bond/commands.list
@@ -0,0 +1,6 @@
+send <IP>ip   # sends one ARPrequest through bonding for IP
+start         # starts listening if not started at startup
+stop          # stops listening
+show          # shows some bond info, e.g. active members
+help          # show help
+quit          # close application
diff --git a/examples/bond/main.c b/examples/bond/main.c
index 90f422ec11..8528abf675 100644
--- a/examples/bond/main.c
+++ b/examples/bond/main.c
@@ -45,16 +45,8 @@
 #include <rte_cpuflags.h>
 #include <rte_eth_bond.h>
 
-#include <cmdline_rdline.h>
-#include <cmdline_parse.h>
-#include <cmdline_parse_num.h>
-#include <cmdline_parse_string.h>
-#include <cmdline_parse_ipaddr.h>
-#include <cmdline_parse_etheraddr.h>
 #include <cmdline_socket.h>
-#include <cmdline.h>
-
-#include "main.h"
+#include "commands.h"
 
 #define RTE_LOGTYPE_DCB RTE_LOGTYPE_USER1
 
@@ -462,11 +454,7 @@ static int lcore_main(__rte_unused void *arg1)
 	return 0;
 }
 
-struct cmd_obj_send_result {
-	cmdline_fixed_string_t action;
-	cmdline_ipaddr_t ip;
-};
-static inline void get_string(struct cmd_obj_send_result *res, char *buf, uint8_t size)
+static inline void get_string(struct cmd_send_result *res, char *buf, uint8_t size)
 {
 	snprintf(buf, size, NIPQUAD_FMT,
 		((unsigned)((unsigned char *)&(res->ip.addr.ipv4))[0]),
@@ -475,12 +463,11 @@ static inline void get_string(struct cmd_obj_send_result *res, char *buf, uint8_
 		((unsigned)((unsigned char *)&(res->ip.addr.ipv4))[3])
 		);
 }
-static void cmd_obj_send_parsed(void *parsed_result,
-		__rte_unused struct cmdline *cl,
-			       __rte_unused void *data)
+void
+cmd_send_parsed(void *parsed_result, __rte_unused struct cmdline *cl, __rte_unused void *data)
 {
 
-	struct cmd_obj_send_result *res = parsed_result;
+	struct cmd_send_result *res = parsed_result;
 	char ip_str[INET6_ADDRSTRLEN];
 
 	struct rte_ether_addr bond_mac_addr;
@@ -544,29 +531,8 @@ static void cmd_obj_send_parsed(void *parsed_result,
 	cmdline_printf(cl, "\n");
 }
 
-cmdline_parse_token_string_t cmd_obj_action_send =
-	TOKEN_STRING_INITIALIZER(struct cmd_obj_send_result, action, "send");
-cmdline_parse_token_ipaddr_t cmd_obj_ip =
-	TOKEN_IPV4_INITIALIZER(struct cmd_obj_send_result, ip);
-
-cmdline_parse_inst_t cmd_obj_send = {
-	.f = cmd_obj_send_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "send client_ip",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_obj_action_send,
-		(void *)&cmd_obj_ip,
-		NULL,
-	},
-};
-
-struct cmd_start_result {
-	cmdline_fixed_string_t start;
-};
-
-static void cmd_start_parsed(__rte_unused void *parsed_result,
-			       struct cmdline *cl,
-			       __rte_unused void *data)
+void
+cmd_start_parsed(__rte_unused void *parsed_result, struct cmdline *cl, __rte_unused void *data)
 {
 	int worker_core_id = rte_lcore_id();
 
@@ -605,26 +571,8 @@ static void cmd_start_parsed(__rte_unused void *parsed_result,
 		);
 }
 
-cmdline_parse_token_string_t cmd_start_start =
-	TOKEN_STRING_INITIALIZER(struct cmd_start_result, start, "start");
-
-cmdline_parse_inst_t cmd_start = {
-	.f = cmd_start_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "starts listening if not started at startup",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_start_start,
-		NULL,
-	},
-};
-
-struct cmd_help_result {
-	cmdline_fixed_string_t help;
-};
-
-static void cmd_help_parsed(__rte_unused void *parsed_result,
-			    struct cmdline *cl,
-			    __rte_unused void *data)
+void
+cmd_help_parsed(__rte_unused void *parsed_result, struct cmdline *cl, __rte_unused void *data)
 {
 	cmdline_printf(cl,
 			"ALB - link bonding mode 6 example\n"
@@ -637,26 +585,8 @@ static void cmd_help_parsed(__rte_unused void *parsed_result,
 		       );
 }
 
-cmdline_parse_token_string_t cmd_help_help =
-	TOKEN_STRING_INITIALIZER(struct cmd_help_result, help, "help");
-
-cmdline_parse_inst_t cmd_help = {
-	.f = cmd_help_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "show help",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_help_help,
-		NULL,
-	},
-};
-
-struct cmd_stop_result {
-	cmdline_fixed_string_t stop;
-};
-
-static void cmd_stop_parsed(__rte_unused void *parsed_result,
-			    struct cmdline *cl,
-			    __rte_unused void *data)
+void
+cmd_stop_parsed(__rte_unused void *parsed_result, struct cmdline *cl, __rte_unused void *data)
 {
 	rte_spinlock_lock(&global_flag_stru_p->lock);
 	if (global_flag_stru_p->LcoreMainIsRunning == 0)	{
@@ -678,26 +608,8 @@ static void cmd_stop_parsed(__rte_unused void *parsed_result,
 	rte_spinlock_unlock(&global_flag_stru_p->lock);
 }
 
-cmdline_parse_token_string_t cmd_stop_stop =
-	TOKEN_STRING_INITIALIZER(struct cmd_stop_result, stop, "stop");
-
-cmdline_parse_inst_t cmd_stop = {
-	.f = cmd_stop_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "this command do not handle any arguments",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_stop_stop,
-		NULL,
-	},
-};
-
-struct cmd_quit_result {
-	cmdline_fixed_string_t quit;
-};
-
-static void cmd_quit_parsed(__rte_unused void *parsed_result,
-			    struct cmdline *cl,
-			    __rte_unused void *data)
+void
+cmd_quit_parsed(__rte_unused void *parsed_result, struct cmdline *cl, __rte_unused void *data)
 {
 	rte_spinlock_lock(&global_flag_stru_p->lock);
 	if (global_flag_stru_p->LcoreMainIsRunning == 0)	{
@@ -721,26 +633,8 @@ static void cmd_quit_parsed(__rte_unused void *parsed_result,
 	cmdline_quit(cl);
 }
 
-cmdline_parse_token_string_t cmd_quit_quit =
-	TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit");
-
-cmdline_parse_inst_t cmd_quit = {
-	.f = cmd_quit_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "this command do not handle any arguments",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_quit_quit,
-		NULL,
-	},
-};
-
-struct cmd_show_result {
-	cmdline_fixed_string_t show;
-};
-
-static void cmd_show_parsed(__rte_unused void *parsed_result,
-			    struct cmdline *cl,
-			    __rte_unused void *data)
+void
+cmd_show_parsed(__rte_unused void *parsed_result, struct cmdline *cl, __rte_unused void *data)
 {
 	uint16_t members[16] = {0};
 	uint8_t len = 16;
@@ -772,31 +666,6 @@ static void cmd_show_parsed(__rte_unused void *parsed_result,
 	rte_spinlock_unlock(&global_flag_stru_p->lock);
 }
 
-cmdline_parse_token_string_t cmd_show_show =
-	TOKEN_STRING_INITIALIZER(struct cmd_show_result, show, "show");
-
-cmdline_parse_inst_t cmd_show = {
-	.f = cmd_show_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "this command do not handle any arguments",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_show_show,
-		NULL,
-	},
-};
-
-/****** CONTEXT (list of instruction) */
-
-cmdline_parse_ctx_t main_ctx[] = {
-	(cmdline_parse_inst_t *)&cmd_start,
-	(cmdline_parse_inst_t *)&cmd_obj_send,
-	(cmdline_parse_inst_t *)&cmd_stop,
-	(cmdline_parse_inst_t *)&cmd_show,
-	(cmdline_parse_inst_t *)&cmd_quit,
-	(cmdline_parse_inst_t *)&cmd_help,
-	NULL,
-};
-
 /* prompt function, called from main on MAIN lcore */
 static void prompt(__rte_unused void *arg1)
 {
diff --git a/examples/bond/main.h b/examples/bond/main.h
deleted file mode 100644
index f91ed9c885..0000000000
--- a/examples/bond/main.h
+++ /dev/null
@@ -1,10 +0,0 @@
-/* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2010-2015 Intel Corporation
- */
-
-#ifndef _MAIN_H_
-#define _MAIN_H_
-
-int main(int argc, char *argv[]);
-
-#endif /* ifndef _MAIN_H_ */
diff --git a/examples/bond/meson.build b/examples/bond/meson.build
index ed22b7887d..bfbec04ecd 100644
--- a/examples/bond/meson.build
+++ b/examples/bond/meson.build
@@ -11,3 +11,11 @@ allow_experimental_apis = true
 sources = files(
         'main.c',
 )
+
+cmd_h = custom_target('commands_hdr',
+	output: 'commands.h',
+	input: files('commands.list'),
+	capture: true,
+	command: [cmdline_gen_cmd, '--context-name=main_ctx', '@INPUT@']
+)
+sources += cmd_h
diff --git a/examples/multi_process/simple_mp/mp_commands.list b/examples/multi_process/simple_mp/mp_commands.list
index c8890cb071..afce5eb1c1 100644
--- a/examples/multi_process/simple_mp/mp_commands.list
+++ b/examples/multi_process/simple_mp/mp_commands.list
@@ -1,3 +1,6 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2023 Intel Corporation
+#
 send <STRING>message  # send a string to another process
 help                  # show help
 quit                  # close the application
-- 
2.39.2

[PATCH v5 7/9] examples/vdpa: auto-generate cmdline boilerplate

[Bruce Richardson]

Use the dpdk-cmdline-gen script to autogenerate all the boilerplate
structs and defines for the commandline part of the app.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 examples/vdpa/Makefile      |  12 ++--
 examples/vdpa/commands.list |   8 +++
 examples/vdpa/main.c        | 131 ++----------------------------------
 examples/vdpa/meson.build   |   7 ++
 4 files changed, 30 insertions(+), 128 deletions(-)
 create mode 100644 examples/vdpa/commands.list

diff --git a/examples/vdpa/Makefile b/examples/vdpa/Makefile
index d974db4f40..aa60a000cf 100644
--- a/examples/vdpa/Makefile
+++ b/examples/vdpa/Makefile
@@ -6,6 +6,7 @@ APP = vdpa
 
 # all source are stored in SRCS-y
 SRCS-y := main.c
+SRC-DEPS := build/commands.h
 CFLAGS += -DALLOW_EXPERIMENTAL_API
 
 PKGCONF ?= pkg-config
@@ -23,10 +24,13 @@ static: build/$(APP)-static
 	ln -sf $(APP)-static build/$(APP)
 
 PC_FILE := $(shell $(PKGCONF) --path libdpdk 2>/dev/null)
-CFLAGS += -O3 $(shell $(PKGCONF) --cflags libdpdk)
+CFLAGS += -O3 $(shell $(PKGCONF) --cflags libdpdk) -I build/
 LDFLAGS_SHARED = $(shell $(PKGCONF) --libs libdpdk)
 LDFLAGS_STATIC = $(shell $(PKGCONF) --static --libs libdpdk)
 
+build/commands.h: commands.list Makefile
+	dpdk-cmdline-gen.py -o $@ --context-name=main_ctx $<
+
 ifeq ($(MAKECMDGOALS),static)
 # check for broken pkg-config
 ifeq ($(shell echo $(LDFLAGS_STATIC) | grep 'whole-archive.*l:lib.*no-whole-archive'),)
@@ -35,10 +39,10 @@ $(error "Cannot generate statically-linked binaries with this version of pkg-con
 endif
 endif
 
-build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
+build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build $(SRC-DEPS)
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
 
-build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build
+build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build $(SRC-DEPS)
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_STATIC)
 
 build:
@@ -46,5 +50,5 @@ build:
 
 .PHONY: clean
 clean:
-	rm -f build/$(APP) build/$(APP)-static build/$(APP)-shared
+	rm -f build/$(APP) build/$(APP)-static build/$(APP)-shared build/*.h
 	test -d build && rmdir -p build || true
diff --git a/examples/vdpa/commands.list b/examples/vdpa/commands.list
new file mode 100644
index 0000000000..1eb8486c45
--- /dev/null
+++ b/examples/vdpa/commands.list
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2023 Intel Corporation
+#
+help                                    # show help
+list                                    # list all available vdpa devices
+create <STRING>socket_path <STRING>bdf  # create a new vdpa port
+stats <STRING>bdf <UINT32>qid           # show device statistics
+quit                                    # exit application
diff --git a/examples/vdpa/main.c b/examples/vdpa/main.c
index 4d3203f3a7..289db26498 100644
--- a/examples/vdpa/main.c
+++ b/examples/vdpa/main.c
@@ -16,11 +16,8 @@
 #include <rte_pci.h>
 #include <rte_string_fns.h>
 
-#include <cmdline_parse.h>
 #include <cmdline_socket.h>
-#include <cmdline_parse_string.h>
-#include <cmdline_parse_num.h>
-#include <cmdline.h>
+#include "commands.h"  /* auto-generated file from commands.list */
 #include "vdpa_blk_compact.h"
 
 #define MAX_PATH_LEN 128
@@ -301,14 +298,9 @@ signal_handler(int signum)
 	}
 }
 
-/* interactive cmds */
+/* interactive cmd functions */
 
-/* *** Help command with introduction. *** */
-struct cmd_help_result {
-	cmdline_fixed_string_t help;
-};
-
-static void cmd_help_parsed(__rte_unused void *parsed_result,
+void cmd_help_parsed(__rte_unused void *parsed_result,
 		struct cmdline *cl,
 		__rte_unused void *data)
 {
@@ -325,25 +317,7 @@ static void cmd_help_parsed(__rte_unused void *parsed_result,
 	);
 }
 
-cmdline_parse_token_string_t cmd_help_help =
-	TOKEN_STRING_INITIALIZER(struct cmd_help_result, help, "help");
-
-cmdline_parse_inst_t cmd_help = {
-	.f = cmd_help_parsed,
-	.data = NULL,
-	.help_str = "show help",
-	.tokens = {
-		(void *)&cmd_help_help,
-		NULL,
-	},
-};
-
-/* *** List all available vdpa devices *** */
-struct cmd_list_result {
-	cmdline_fixed_string_t action;
-};
-
-static void cmd_list_vdpa_devices_parsed(
+void cmd_list_parsed(
 		__rte_unused void *parsed_result,
 		struct cmdline *cl,
 		__rte_unused void *data)
@@ -376,27 +350,7 @@ static void cmd_list_vdpa_devices_parsed(
 	}
 }
 
-cmdline_parse_token_string_t cmd_action_list =
-	TOKEN_STRING_INITIALIZER(struct cmd_list_result, action, "list");
-
-cmdline_parse_inst_t cmd_list_vdpa_devices = {
-	.f = cmd_list_vdpa_devices_parsed,
-	.data = NULL,
-	.help_str = "list all available vdpa devices",
-	.tokens = {
-		(void *)&cmd_action_list,
-		NULL,
-	},
-};
-
-/* *** Create new vdpa port *** */
-struct cmd_create_result {
-	cmdline_fixed_string_t action;
-	cmdline_fixed_string_t socket_path;
-	cmdline_fixed_string_t bdf;
-};
-
-static void cmd_create_vdpa_port_parsed(void *parsed_result,
+void cmd_create_parsed(void *parsed_result,
 		struct cmdline *cl,
 		__rte_unused void *data)
 {
@@ -417,33 +371,7 @@ static void cmd_create_vdpa_port_parsed(void *parsed_result,
 		devcnt++;
 }
 
-cmdline_parse_token_string_t cmd_action_create =
-	TOKEN_STRING_INITIALIZER(struct cmd_create_result, action, "create");
-cmdline_parse_token_string_t cmd_socket_path =
-	TOKEN_STRING_INITIALIZER(struct cmd_create_result, socket_path, NULL);
-cmdline_parse_token_string_t cmd_bdf =
-	TOKEN_STRING_INITIALIZER(struct cmd_create_result, bdf, NULL);
-
-cmdline_parse_inst_t cmd_create_vdpa_port = {
-	.f = cmd_create_vdpa_port_parsed,
-	.data = NULL,
-	.help_str = "create a new vdpa port",
-	.tokens = {
-		(void *)&cmd_action_create,
-		(void *)&cmd_socket_path,
-		(void *)&cmd_bdf,
-		NULL,
-	},
-};
-
-/* *** STATS *** */
-struct cmd_stats_result {
-	cmdline_fixed_string_t stats;
-	cmdline_fixed_string_t bdf;
-	uint16_t qid;
-};
-
-static void cmd_device_stats_parsed(void *parsed_result, struct cmdline *cl,
+void cmd_stats_parsed(void *parsed_result, struct cmdline *cl,
 				    __rte_unused void *data)
 {
 	struct cmd_stats_result *res = parsed_result;
@@ -525,31 +453,7 @@ static void cmd_device_stats_parsed(void *parsed_result, struct cmdline *cl,
 	}
 }
 
-cmdline_parse_token_string_t cmd_device_stats_ =
-	TOKEN_STRING_INITIALIZER(struct cmd_stats_result, stats, "stats");
-cmdline_parse_token_string_t cmd_device_bdf =
-	TOKEN_STRING_INITIALIZER(struct cmd_stats_result, bdf, NULL);
-cmdline_parse_token_num_t cmd_queue_id =
-	TOKEN_NUM_INITIALIZER(struct cmd_stats_result, qid, RTE_UINT32);
-
-cmdline_parse_inst_t cmd_device_stats = {
-	.f = cmd_device_stats_parsed,
-	.data = NULL,
-	.help_str = "stats: show device statistics",
-	.tokens = {
-		(void *)&cmd_device_stats_,
-		(void *)&cmd_device_bdf,
-		(void *)&cmd_queue_id,
-		NULL,
-	},
-};
-
-/* *** QUIT *** */
-struct cmd_quit_result {
-	cmdline_fixed_string_t quit;
-};
-
-static void cmd_quit_parsed(__rte_unused void *parsed_result,
+void cmd_quit_parsed(__rte_unused void *parsed_result,
 		struct cmdline *cl,
 		__rte_unused void *data)
 {
@@ -557,27 +461,6 @@ static void cmd_quit_parsed(__rte_unused void *parsed_result,
 	cmdline_quit(cl);
 }
 
-cmdline_parse_token_string_t cmd_quit_quit =
-	TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit");
-
-cmdline_parse_inst_t cmd_quit = {
-	.f = cmd_quit_parsed,
-	.data = NULL,
-	.help_str = "quit: exit application",
-	.tokens = {
-		(void *)&cmd_quit_quit,
-		NULL,
-	},
-};
-cmdline_parse_ctx_t main_ctx[] = {
-	(cmdline_parse_inst_t *)&cmd_help,
-	(cmdline_parse_inst_t *)&cmd_list_vdpa_devices,
-	(cmdline_parse_inst_t *)&cmd_create_vdpa_port,
-	(cmdline_parse_inst_t *)&cmd_device_stats,
-	(cmdline_parse_inst_t *)&cmd_quit,
-	NULL,
-};
-
 int
 main(int argc, char *argv[])
 {
diff --git a/examples/vdpa/meson.build b/examples/vdpa/meson.build
index bd086050dc..a48028da4d 100644
--- a/examples/vdpa/meson.build
+++ b/examples/vdpa/meson.build
@@ -16,3 +16,10 @@ allow_experimental_apis = true
 sources = files(
         'main.c',
 )
+cmd_h = custom_target('commands_hdr',
+	output: 'commands.h',
+	input: files('commands.list'),
+	capture: true,
+	command: [cmdline_gen_cmd, '--context-name=main_ctx', '@INPUT@']
+)
+sources += cmd_h
-- 
2.39.2

[PATCH v5 8/9] buildtools/dpdk-cmdline-gen: support option strings

[Bruce Richardson]

Add support to the commandline generator for option strings, where there
are only a limited number of acceptable values to be passed as a
parameter.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 buildtools/dpdk-cmdline-gen.py    | 7 +++++++
 doc/guides/prog_guide/cmdline.rst | 6 +++++-
 2 files changed, 12 insertions(+), 1 deletion(-)

diff --git a/buildtools/dpdk-cmdline-gen.py b/buildtools/dpdk-cmdline-gen.py
index 6cb7610de4..fe74194121 100755
--- a/buildtools/dpdk-cmdline-gen.py
+++ b/buildtools/dpdk-cmdline-gen.py
@@ -73,6 +73,13 @@ def process_command(lineno, tokens, comment):
                 f"cmdline_parse_token_ipaddr_t cmd_{name}_{t_name}_tok =\n"
                 + f"\tTOKEN_IPV4_INITIALIZER(struct cmd_{name}_result, {t_name});"
             )
+        elif t_type.startswith("|") and t_type.endswith("|"):
+            result_struct.append(f"\tcmdline_fixed_string_t {t_name};")
+            t_val = f'"{t_type[1:-1].replace("|","#")}"'
+            initializers.append(
+                f"static cmdline_parse_token_string_t cmd_{name}_{t_name}_tok =\n"
+                + f"\tTOKEN_STRING_INITIALIZER(struct cmd_{name}_result, {t_name}, {t_val});"
+            )
         else:
             raise TypeError(f"Error line {lineno + 1}: unknown token type '{t_type}'")
         token_list.append(f"cmd_{name}_{t_name}_tok")
diff --git a/doc/guides/prog_guide/cmdline.rst b/doc/guides/prog_guide/cmdline.rst
index 0b96b770e2..1268eca911 100644
--- a/doc/guides/prog_guide/cmdline.rst
+++ b/doc/guides/prog_guide/cmdline.rst
@@ -62,7 +62,8 @@ The format of the list file must be:
 
 * One command per line
 
-* Variable fields are prefixed by the type-name in angle-brackets, for example:
+* Variable fields are prefixed by the type-name, or "|"-delimited option-list, in angle-brackets.
+  For example:
 
   * ``<STRING>message``
 
@@ -70,6 +71,8 @@ The format of the list file must be:
 
   * ``<IP>src_ip``
 
+  * ``<|rx|tx|rxtx|>mode``
+
 * The help text for a command is given in the form of a comment on the same line as the command
 
 An example list file, with a variety of (unrelated) commands, is shown below::
@@ -79,6 +82,7 @@ An example list file, with a variety of (unrelated) commands, is shown below::
    add <UINT16>x <UINT16>y  # add x and y
    echo <STRING>message     # print message to screen
    add socket <STRING>path  # add unix socket with the given path
+   set mode <|rx|tx|>rxtx   # set Rx-only or Tx-only mode
    quit                     # close the application
 
 Running the Generator Script
-- 
2.39.2

[PATCH v5 9/9] examples/ntb: auto-generate cmdline boilerplate

[Bruce Richardson]

Use the dpdk-cmdline-gen script to autogenerate all the boilerplate
structs and defines for the commandline part of the app.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 examples/ntb/Makefile      |  12 ++-
 examples/ntb/commands.list |  11 ++
 examples/ntb/meson.build   |   7 ++
 examples/ntb/ntb_fwd.c     | 200 ++++---------------------------------
 4 files changed, 47 insertions(+), 183 deletions(-)
 create mode 100644 examples/ntb/commands.list

diff --git a/examples/ntb/Makefile b/examples/ntb/Makefile
index d9b6e53090..f5b3405c47 100644
--- a/examples/ntb/Makefile
+++ b/examples/ntb/Makefile
@@ -6,6 +6,7 @@ APP = ntb_fwd
 
 # all source are stored in SRCS-y
 SRCS-y := ntb_fwd.c
+SRC-DEPS := build/commands.h
 
 PKGCONF ?= pkg-config
 
@@ -25,10 +26,13 @@ CFLAGS += -D_FILE_OFFSET_BITS=64
 LDFLAGS += -pthread
 
 PC_FILE := $(shell $(PKGCONF) --path libdpdk 2>/dev/null)
-CFLAGS += -O3 $(shell $(PKGCONF) --cflags libdpdk)
+CFLAGS += -O3 $(shell $(PKGCONF) --cflags libdpdk) -I build/
 LDFLAGS_SHARED = $(shell $(PKGCONF) --libs libdpdk)
 LDFLAGS_STATIC = $(shell $(PKGCONF) --static --libs libdpdk)
 
+build/commands.h: commands.list Makefile
+	dpdk-cmdline-gen.py -o $@ --context-name=main_ctx $<
+
 ifeq ($(MAKECMDGOALS),static)
 # check for broken pkg-config
 ifeq ($(shell echo $(LDFLAGS_STATIC) | grep 'whole-archive.*l:lib.*no-whole-archive'),)
@@ -39,10 +43,10 @@ endif
 
 CFLAGS += -DALLOW_EXPERIMENTAL_API
 
-build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
+build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build $(SRC-DEPS)
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
 
-build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build
+build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build $(SRC-DEPS)
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_STATIC)
 
 build:
@@ -50,5 +54,5 @@ build:
 
 .PHONY: clean
 clean:
-	rm -f build/$(APP) build/$(APP)-static build/$(APP)-shared
+	rm -f build/$(APP) build/$(APP)-static build/$(APP)-shared build/*.h
 	test -d build && rmdir -p build || true
diff --git a/examples/ntb/commands.list b/examples/ntb/commands.list
new file mode 100644
index 0000000000..cdac3519fc
--- /dev/null
+++ b/examples/ntb/commands.list
@@ -0,0 +1,11 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2023 Intel Corporation
+#
+help                   # show help
+quit                   # exit application
+send <STRING>filepath  # send <file_path>
+start                  # start pkt fwd between ntb and ethdev
+stop                   # stop packet forwarding
+show port stats        # show statistics for all ports
+clear port stats       # clear all port statistics
+set fwd <|file-trans|iofwd|txonly|rxonly|>mode  # set forwarding mode as file-trans|rxonly|txonly|iofwd
diff --git a/examples/ntb/meson.build b/examples/ntb/meson.build
index 18eaffdf21..6c23081c41 100644
--- a/examples/ntb/meson.build
+++ b/examples/ntb/meson.build
@@ -17,3 +17,10 @@ cflags += ['-D_FILE_OFFSET_BITS=64']
 sources = files(
         'ntb_fwd.c',
 )
+cmd_h = custom_target('commands_hdr',
+	output: 'commands.h',
+	input: files('commands.list'),
+	capture: true,
+	command: [cmdline_gen_cmd, '--context-name=main_ctx', '@INPUT@']
+)
+sources += cmd_h
diff --git a/examples/ntb/ntb_fwd.c b/examples/ntb/ntb_fwd.c
index 585aad9d70..95a6148c82 100644
--- a/examples/ntb/ntb_fwd.c
+++ b/examples/ntb/ntb_fwd.c
@@ -21,6 +21,7 @@
 #include <rte_cycles.h>
 #include <rte_pmd_ntb.h>
 #include <rte_mbuf_pool_ops.h>
+#include "commands.h"
 
 /* Per-port statistics struct */
 struct ntb_port_statistics {
@@ -103,12 +104,7 @@ static struct rte_eth_conf eth_port_conf = {
 	},
 };
 
-/* *** Help command with introduction. *** */
-struct cmd_help_result {
-	cmdline_fixed_string_t help;
-};
-
-static void
+void
 cmd_help_parsed(__rte_unused void *parsed_result,
 		struct cmdline *cl,
 		__rte_unused void *data)
@@ -134,26 +130,7 @@ cmd_help_parsed(__rte_unused void *parsed_result,
 	);
 
 }
-
-cmdline_parse_token_string_t cmd_help_help =
-	TOKEN_STRING_INITIALIZER(struct cmd_help_result, help, "help");
-
-cmdline_parse_inst_t cmd_help = {
-	.f = cmd_help_parsed,
-	.data = NULL,
-	.help_str = "show help",
-	.tokens = {
-		(void *)&cmd_help_help,
-		NULL,
-	},
-};
-
-/* *** QUIT *** */
-struct cmd_quit_result {
-	cmdline_fixed_string_t quit;
-};
-
-static void
+void
 cmd_quit_parsed(__rte_unused void *parsed_result,
 		struct cmdline *cl,
 		__rte_unused void *data)
@@ -188,31 +165,12 @@ cmd_quit_parsed(__rte_unused void *parsed_result,
 	cmdline_quit(cl);
 }
 
-cmdline_parse_token_string_t cmd_quit_quit =
-		TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit");
-
-cmdline_parse_inst_t cmd_quit = {
-	.f = cmd_quit_parsed,
-	.data = NULL,
-	.help_str = "exit application",
-	.tokens = {
-		(void *)&cmd_quit_quit,
-		NULL,
-	},
-};
-
-/* *** SEND FILE PARAMETERS *** */
-struct cmd_sendfile_result {
-	cmdline_fixed_string_t send_string;
-	char filepath[];
-};
-
-static void
-cmd_sendfile_parsed(void *parsed_result,
+void
+cmd_send_parsed(void *parsed_result,
 		    __rte_unused struct cmdline *cl,
 		    __rte_unused void *data)
 {
-	struct cmd_sendfile_result *res = parsed_result;
+	struct cmd_send_result *res = parsed_result;
 	struct rte_rawdev_buf *pkts_send[NTB_MAX_PKT_BURST];
 	struct rte_mbuf *mbuf_send[NTB_MAX_PKT_BURST];
 	uint64_t size, count, i, j, nb_burst;
@@ -327,24 +285,6 @@ cmd_sendfile_parsed(void *parsed_result,
 	fclose(file);
 }
 
-cmdline_parse_token_string_t cmd_send_file_send =
-	TOKEN_STRING_INITIALIZER(struct cmd_sendfile_result, send_string,
-				 "send");
-cmdline_parse_token_string_t cmd_send_file_filepath =
-	TOKEN_STRING_INITIALIZER(struct cmd_sendfile_result, filepath, NULL);
-
-
-cmdline_parse_inst_t cmd_send_file = {
-	.f = cmd_sendfile_parsed,
-	.data = NULL,
-	.help_str = "send <file_path>",
-	.tokens = {
-		(void *)&cmd_send_file_send,
-		(void *)&cmd_send_file_filepath,
-		NULL,
-	},
-};
-
 #define RECV_FILE_LEN 30
 static int
 start_polling_recv_file(void *param)
@@ -788,12 +728,7 @@ start_pkt_fwd(void)
 	}
 }
 
-/* *** START FWD PARAMETERS *** */
-struct cmd_start_result {
-	cmdline_fixed_string_t start;
-};
-
-static void
+void
 cmd_start_parsed(__rte_unused void *parsed_result,
 			    __rte_unused struct cmdline *cl,
 			    __rte_unused void *data)
@@ -801,25 +736,7 @@ cmd_start_parsed(__rte_unused void *parsed_result,
 	start_pkt_fwd();
 }
 
-cmdline_parse_token_string_t cmd_start_start =
-		TOKEN_STRING_INITIALIZER(struct cmd_start_result, start, "start");
-
-cmdline_parse_inst_t cmd_start = {
-	.f = cmd_start_parsed,
-	.data = NULL,
-	.help_str = "start pkt fwd between ntb and ethdev",
-	.tokens = {
-		(void *)&cmd_start_start,
-		NULL,
-	},
-};
-
-/* *** STOP *** */
-struct cmd_stop_result {
-	cmdline_fixed_string_t stop;
-};
-
-static void
+void
 cmd_stop_parsed(__rte_unused void *parsed_result,
 		__rte_unused struct cmdline *cl,
 		__rte_unused void *data)
@@ -844,19 +761,6 @@ cmd_stop_parsed(__rte_unused void *parsed_result,
 	printf("\nDone.\n");
 }
 
-cmdline_parse_token_string_t cmd_stop_stop =
-		TOKEN_STRING_INITIALIZER(struct cmd_stop_result, stop, "stop");
-
-cmdline_parse_inst_t cmd_stop = {
-	.f = cmd_stop_parsed,
-	.data = NULL,
-	.help_str = "stop: Stop packet forwarding",
-	.tokens = {
-		(void *)&cmd_stop_stop,
-		NULL,
-	},
-};
-
 static void
 ntb_stats_clear(void)
 {
@@ -975,58 +879,28 @@ ntb_stats_display(void)
 	free(ids);
 }
 
-/* *** SHOW/CLEAR PORT STATS *** */
-struct cmd_stats_result {
-	cmdline_fixed_string_t show;
-	cmdline_fixed_string_t port;
-	cmdline_fixed_string_t stats;
-};
-
-static void
-cmd_stats_parsed(void *parsed_result,
+void
+cmd_show_port_stats_parsed(__rte_unused void *parsed_result,
 		 __rte_unused struct cmdline *cl,
 		 __rte_unused void *data)
 {
-	struct cmd_stats_result *res = parsed_result;
-	if (!strcmp(res->show, "clear"))
-		ntb_stats_clear();
-	else
-		ntb_stats_display();
+	ntb_stats_display();
 }
 
-cmdline_parse_token_string_t cmd_stats_show =
-	TOKEN_STRING_INITIALIZER(struct cmd_stats_result, show, "show#clear");
-cmdline_parse_token_string_t cmd_stats_port =
-	TOKEN_STRING_INITIALIZER(struct cmd_stats_result, port, "port");
-cmdline_parse_token_string_t cmd_stats_stats =
-	TOKEN_STRING_INITIALIZER(struct cmd_stats_result, stats, "stats");
-
-
-cmdline_parse_inst_t cmd_stats = {
-	.f = cmd_stats_parsed,
-	.data = NULL,
-	.help_str = "show|clear port stats",
-	.tokens = {
-		(void *)&cmd_stats_show,
-		(void *)&cmd_stats_port,
-		(void *)&cmd_stats_stats,
-		NULL,
-	},
-};
-
-/* *** SET FORWARDING MODE *** */
-struct cmd_set_fwd_mode_result {
-	cmdline_fixed_string_t set;
-	cmdline_fixed_string_t fwd;
-	cmdline_fixed_string_t mode;
-};
+void
+cmd_clear_port_stats_parsed(__rte_unused void *parsed_result,
+		 __rte_unused struct cmdline *cl,
+		 __rte_unused void *data)
+{
+	ntb_stats_clear();
+}
 
-static void
-cmd_set_fwd_mode_parsed(__rte_unused void *parsed_result,
+void
+cmd_set_fwd_parsed(void *parsed_result,
 			__rte_unused struct cmdline *cl,
 			__rte_unused void *data)
 {
-	struct cmd_set_fwd_mode_result *res = parsed_result;
+	struct cmd_set_fwd_result *res = parsed_result;
 	int i;
 
 	if (in_test) {
@@ -1043,38 +917,6 @@ cmd_set_fwd_mode_parsed(__rte_unused void *parsed_result,
 	printf("Invalid %s packet forwarding mode.\n", res->mode);
 }
 
-cmdline_parse_token_string_t cmd_setfwd_set =
-	TOKEN_STRING_INITIALIZER(struct cmd_set_fwd_mode_result, set, "set");
-cmdline_parse_token_string_t cmd_setfwd_fwd =
-	TOKEN_STRING_INITIALIZER(struct cmd_set_fwd_mode_result, fwd, "fwd");
-cmdline_parse_token_string_t cmd_setfwd_mode =
-	TOKEN_STRING_INITIALIZER(struct cmd_set_fwd_mode_result, mode,
-				"file-trans#iofwd#txonly#rxonly");
-
-cmdline_parse_inst_t cmd_set_fwd_mode = {
-	.f = cmd_set_fwd_mode_parsed,
-	.data = NULL,
-	.help_str = "set forwarding mode as file-trans|rxonly|txonly|iofwd",
-	.tokens = {
-		(void *)&cmd_setfwd_set,
-		(void *)&cmd_setfwd_fwd,
-		(void *)&cmd_setfwd_mode,
-		NULL,
-	},
-};
-
-/* list of instructions */
-cmdline_parse_ctx_t main_ctx[] = {
-	(cmdline_parse_inst_t *)&cmd_help,
-	(cmdline_parse_inst_t *)&cmd_send_file,
-	(cmdline_parse_inst_t *)&cmd_start,
-	(cmdline_parse_inst_t *)&cmd_stop,
-	(cmdline_parse_inst_t *)&cmd_stats,
-	(cmdline_parse_inst_t *)&cmd_set_fwd_mode,
-	(cmdline_parse_inst_t *)&cmd_quit,
-	NULL,
-};
-
 /* prompt function, called from main on MAIN lcore */
 static void
 prompt(void)
-- 
2.39.2

[PATCH v6 0/9] document and simplify use of cmdline

[Bruce Richardson]

The DPDK commandline library is widely used by apps and examples within
DPDK, but it is not documented in our programmers guide and it requires
a lot of boilerplate code definitions in order to used. We can improve
this situation by creating a simple python script to automatically
generate the boilerplate from a list of commands.

This patchset contains a new documentation chapter on cmdline library,
going through step-by-step how to add commands and create the necessary
token lists and parse contexts.

Following that initial doc patch, the set then contains a
boilerplate-generating script, as well as a set of four patches showing
its use, by converting four examples to use the script instead of
having the hard-coded boilerplate. Once the script is used, adding a new
command becomes as simple as adding the desired command to the .list
file, and then writing the required function which will be called for
that command. No other boilerplate coding is necessary.

Final two patches, from V5 onwards, add support for option-lists in values,
and then use that support to convert over a fifth sample app - ntb.

Cmdline script obviously does not cover the full range of capabilities
of the commandline lib, but does cover the most used parts. The
code-saving to each of the examples by auto-generating the boilerplate
is significant, and probably more examples with commandlines can be
converted over in future.

The "cmdline" example itself, is not converted over, as it should
probably remain as a simple example of direct library use without the
script.

v6:
* rework syntax of the option lists following feedback from David.
  Now use "(x,y,z)" instead of "|x|y|z|", to avoid giving impression
  of regex support

V5:
* Added copyright headers to command list files
* Add support for option lists
* Convert examples/ntb to use script

V4:
* Reworked script following feedback from Robin J.
   - formatted code with black
   - used multi-line strings
   - replaced sys.exit(1) with proper error handling
   - per-command function returns lists rather than printing directly
   - other small cleanups
* Added change to CI script so the cmdline script is in PATH
* Converted VDPA example to script, saving another 100 LOC

V3:
* Added lots of documentation
* Added support for help text for each command
* Cleaned up script a little so it passes pycodestyle and most flake8
  checks, when line-length is set to max 100.
* Removed RFC tag, as I consider this patchset stable enough for
  consideration in a release.

V2-RFC:
* Add support for IP addresses in commands
* Move to buildtools directory and make installable
* Convert 3 examples to use script, and eliminate their boilerplate
 *** SUBJECT HERE ***

*** BLURB HERE ***

Bruce Richardson (9):
  doc/prog_guide: new chapter on cmdline library
  buildtools: script to generate cmdline boilerplate
  ci: allow use of DPDK tools when building examples
  examples/simple_mp: auto-generate cmdline boilerplate
  examples/hotplug_mp: auto-generate cmdline boilerplate
  examples/bond: auto-generate cmdline boilerplate
  examples/vdpa: auto-generate cmdline boilerplate
  buildtools/dpdk-cmdline-gen: support option strings
  examples/ntb: auto-generate cmdline boilerplate

 .ci/linux-build.sh                            |   1 +
 app/test/commands.c                           |   2 +
 buildtools/dpdk-cmdline-gen.py                | 197 ++++++++
 buildtools/meson.build                        |   7 +
 doc/guides/prog_guide/cmdline.rst             | 473 ++++++++++++++++++
 doc/guides/prog_guide/index.rst               |   1 +
 examples/bond/Makefile                        |  12 +-
 examples/bond/commands.list                   |   6 +
 examples/bond/main.c                          | 161 +-----
 examples/bond/main.h                          |  10 -
 examples/bond/meson.build                     |   8 +
 examples/multi_process/hotplug_mp/Makefile    |  12 +-
 examples/multi_process/hotplug_mp/commands.c  | 147 +-----
 examples/multi_process/hotplug_mp/commands.h  |  10 -
 .../multi_process/hotplug_mp/commands.list    |   8 +
 examples/multi_process/hotplug_mp/meson.build |   9 +
 examples/multi_process/simple_mp/Makefile     |  12 +-
 examples/multi_process/simple_mp/meson.build  |   9 +
 .../multi_process/simple_mp/mp_commands.c     | 106 +---
 .../multi_process/simple_mp/mp_commands.h     |  14 -
 .../multi_process/simple_mp/mp_commands.list  |   6 +
 examples/ntb/Makefile                         |  12 +-
 examples/ntb/commands.list                    |  11 +
 examples/ntb/meson.build                      |   7 +
 examples/ntb/ntb_fwd.c                        | 200 +-------
 examples/vdpa/Makefile                        |  12 +-
 examples/vdpa/commands.list                   |   8 +
 examples/vdpa/main.c                          | 131 +----
 examples/vdpa/meson.build                     |   7 +
 29 files changed, 866 insertions(+), 733 deletions(-)
 create mode 100755 buildtools/dpdk-cmdline-gen.py
 create mode 100644 doc/guides/prog_guide/cmdline.rst
 create mode 100644 examples/bond/commands.list
 delete mode 100644 examples/bond/main.h
 delete mode 100644 examples/multi_process/hotplug_mp/commands.h
 create mode 100644 examples/multi_process/hotplug_mp/commands.list
 delete mode 100644 examples/multi_process/simple_mp/mp_commands.h
 create mode 100644 examples/multi_process/simple_mp/mp_commands.list
 create mode 100644 examples/ntb/commands.list
 create mode 100644 examples/vdpa/commands.list

--
2.39.2

[PATCH v6 1/9] doc/prog_guide: new chapter on cmdline library

[Bruce Richardson]

The cmdline library was not documented in our programmers guide, so add
a new chapter on it. This chapter covers step-by-step how to use the
library, rather than focusing on the library internals. This complements
the existing cmdline example app document, providing more details on the
process of using the library.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 app/test/commands.c               |   2 +
 doc/guides/prog_guide/cmdline.rst | 337 ++++++++++++++++++++++++++++++
 doc/guides/prog_guide/index.rst   |   1 +
 3 files changed, 340 insertions(+)
 create mode 100644 doc/guides/prog_guide/cmdline.rst

diff --git a/app/test/commands.c b/app/test/commands.c
index 31259e5c21..497d8e9952 100644
--- a/app/test/commands.c
+++ b/app/test/commands.c
@@ -108,6 +108,7 @@ dump_struct_sizes(void)
 #undef DUMP_SIZE
 }
 
+/* Add the dump_* tests cases 8< */
 static void cmd_dump_parsed(void *parsed_result,
 			    __rte_unused struct cmdline *cl,
 			    __rte_unused void *data)
@@ -155,6 +156,7 @@ cmdline_parse_inst_t cmd_dump = {
 		NULL,
 	},
 };
+/* >8 End of add the dump_* tests cases */
 
 /****************/
 
diff --git a/doc/guides/prog_guide/cmdline.rst b/doc/guides/prog_guide/cmdline.rst
new file mode 100644
index 0000000000..40f49a30cc
--- /dev/null
+++ b/doc/guides/prog_guide/cmdline.rst
@@ -0,0 +1,337 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2023 Intel Corporation.
+
+Command-line Library
+====================
+
+Since its earliest versions, DPDK has included a command-line library -
+primarily for internal use by, for example, ``dpdk-testpmd`` and the ``dpdk-test`` binaries,
+but the library is also exported on install and can be used by any end application.
+This chapter covers the basics of the command-line library and how to use it in an application.
+
+Library Features
+----------------
+
+The DPDK command-line library supports the following features:
+
+* Tab-completion available for interactive terminal sessions
+
+* Ability to read and process commands taken from an input file, e.g. startup script
+
+* Parameterized commands able to take multiple parameters with different datatypes:
+
+   * Strings
+   * Signed/unsigned 16/32/64-bit integers
+   * IP Addresses
+   * Ethernet Addresses
+
+* Ability to multiplex multiple commands to a single callback function
+
+Adding Command-line to an Application
+-------------------------------------
+
+Adding a command-line instance to an application involves a number of coding steps.
+
+1. Define the result structure for the command, specifying the command parameters
+
+2. Provide an initializer for each field in the result
+
+3. Define the callback function for the command
+
+4. Provide a parse result structure instance for the command, linking the callback to the command
+
+5. Add the parse result structure to a command-line context
+
+6. Within your main application code, create a new command-line instance passing in the context.
+
+The next few subsections will cover each of these steps in more detail,
+working through an example to add two commands to a command-line instance.
+Those two commands will be:
+
+1. ``quit`` - as the name suggests, to close the application
+
+2. ``show port stats <n>`` - to display on-screen the statistics for a given ethernet port
+
+.. note::
+
+   For further examples of use of the command-line, see
+   :doc:`cmdline example application <../sample_app_ug/cmd_line>`
+
+Defining Command Result Structure
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The first structure to be defined is the structure which will be created on successful parse of a command.
+This structure contains one member field for each token, or word, in the command.
+The simplest case is for a one-word command, like ``quit``.
+For this, we only need to define a structure with a single string parameter to contain that word.
+
+.. code-block:: c
+
+   struct cmd_quit_result {
+              cmdline_fixed_string_t quit;
+   };
+
+For readability, the name of the struct member should match that of the token in the command.
+
+For our second command, we need a structure with four member fields in it,
+as there are four words/tokens in our command.
+The first three are strings, and the final one is a 16-bit numeric value.
+The resulting struct looks like:
+
+.. code-block:: c
+
+   struct cmd_show_port_stats_result {
+      cmdline_fixed_string_t show;
+      cmdline_fixed_string_t port;
+      cmdline_fixed_string_t stats;
+      uint16_t n;
+   };
+
+As before, we choose names to match the tokens in the command.
+Since our numeric parameter is a 16-bit value, we use ``uint16_t`` type for it.
+Any of the standard sized integer types can be used as parameters, depending on the desired result.
+
+Beyond the standard integer types,
+the library also allows variable parameters to be of a number of other types,
+as called out in the feature list above.
+
+* For variable string parameters,
+  the type should be ``cmdline_fixed_string_t`` - the same as for fixed tokens,
+  but these will be initialized differently (as described below).
+
+* For ethernet addresses use type ``struct rte_ether_addr``
+
+* For IP addresses use type ``cmdline_ipaddr_t``
+
+Providing Field Initializers
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Each field of our result structure needs an initializer.
+For fixed string tokens, like "quit", "show" and "port", the initializer will be the string itself.
+
+.. code-block:: c
+
+   static cmdline_parse_token_string_t cmd_quit_quit_tok =
+      TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit");
+
+The convention for naming used here is to include the base name of the overall result structure -
+``cmd_quit`` in this case,
+as well as the name of the field within that structure - ``quit`` in this case, followed by ``_tok``.
+(This is why there is a double ``quit`` in the name above).
+
+This naming convention is seen in our second example,
+which also demonstrates how to define a numeric initializer.
+
+
+.. code-block:: c
+
+   static cmdline_parse_token_string_t cmd_show_port_stats_show_tok =
+      TOKEN_STRING_INITIALIZER(struct cmd_show_port_stats_result, show, "show");
+   static cmdline_parse_token_string_t cmd_show_port_stats_port_tok =
+      TOKEN_STRING_INITIALIZER(struct cmd_show_port_stats_result, port, "port");
+   static cmdline_parse_token_string_t cmd_show_port_stats_stats_tok =
+      TOKEN_STRING_INITIALIZER(struct cmd_show_port_stats_result, stats, "stats");
+   static cmdline_parse_token_num_t cmd_show_port_stats_n_tok =
+      TOKEN_NUM_INITIALIZER(struct cmd_show_port_stats_result, n, RTE_UINT16);
+
+For variable string tokens, the same ``TOKEN_STRING_INITIALIZER`` macro should be used.
+However, the final parameter should be ``NULL`` rather than a hard-coded token string.
+
+For numeric parameters, the final parameter to the ``TOKEN_NUM_INITIALIZER`` macro should be the
+cmdline type matching the variable type defined in the result structure,
+e.g. RTE_UINT8, RTE_UINT32, etc.
+
+For IP addresses, the macro ``TOKEN_IPADDR_INITIALIZER`` should be used.
+
+For ethernet addresses, the macro ``TOKEN_ETHERADDR_INITIALIZER`` should be used.
+
+Defining Callback Function
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+For each command, we need to define a function to be called once the command has been recognised.
+The callback function should have type:
+
+.. code:: c
+
+   void (*f)(void *, struct cmdline *, void *)
+
+where the first parameter is a pointer to the result structure defined above,
+the second parameter is the command-line instance,
+and the final parameter is a user-defined pointer provided when we associate the callback with the command.
+Most callback functions only use the first parameter, or none at all,
+but the additional two parameters provide some extra flexibility,
+to allow the callback to work with non-global state in your application.
+
+For our two example commands, the relevant callback functions would look very similar in definition.
+However, within the function body,
+we assume that the user would need to reference the result structure to extract the port number in
+the second case.
+
+.. code:: c
+
+   void
+   cmd_quit_parsed(void *parsed_result, struct cmdline *cl, void *data)
+   {
+      quit = 1;
+   }
+   void
+   cmd_show_port_stats_parsed(void *parsed_result, struct cmdline *cl, void *data)
+   {
+      struct cmd_show_port_stats_result *res = parsed_result;
+      uint16_t port_id = res->n;
+      ...
+   }
+
+
+Associating Callback and Command
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The ``cmdline_parse_inst_t`` type defines a "parse instance",
+i.e. a sequence of tokens to be matched and then an associated function to be called.
+Also included in the instance type are a field for help text for the command,
+and any additional user-defined parameter to be passed to the callback functions referenced above.
+For example, for our simple "quit" command:
+
+.. code-block:: c
+
+   static cmdline_parse_inst_t cmd_quit = {
+       .f = cmd_quit_parsed,
+       .data = NULL,
+       .help_str = "Close the application",
+       .tokens = {
+           (void *)&cmd_quit_quit_tok,
+           NULL
+       }
+   };
+
+In this case, we firstly identify the callback function to be called,
+then set the user-defined parameter to NULL,
+provide a help message to be given, on request, to the user explaining the command,
+before finally listing out the single token to be matched for this command instance.
+
+For our second, port stats, example,
+as well as making things a little more complicated by having multiple tokens to be matched,
+we can also demonstrate passing in a parameter to the function.
+Let us suppose that our application does not always use all the ports available to it,
+but instead only uses a subset of the ports, stored in an array called ``active_ports``.
+Our stats command, therefore, should only display stats for the currently in-use ports,
+so we pass this ``active_ports`` array.
+(For simplicity of illustration, we shall assume that the array uses a terminating marker,
+e.g. -1 for the end of the port list, so we don't need to pass in a length parameter too.)
+
+.. code-block:: c
+
+   extern int16_t active_ports[];
+   ...
+   static cmdline_parse_inst_t cmd_show_port_stats = {
+       .f = cmd_show_port_stats_parsed,
+       .data = active_ports,
+       .help_str = "Show statistics for active network ports",
+       .tokens = {
+           (void *)&cmd_show_port_stats_show_tok,
+           (void *)&cmd_show_port_stats_port_tok,
+           (void *)&cmd_show_port_stats_stats_tok,
+           (void *)&cmd_show_port_stats_n_tok,
+           NULL
+       }
+   };
+
+
+Adding Command to Command-line Context
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Now that we have configured each individual command and callback,
+we need to merge these into a single array of command-line "contexts".
+This context array will be used to create the actual command-line instance in the application.
+Thankfully, each context entry is the same as each parse instance,
+so our array is defined by simply listing out the previously defined command parse instances.
+
+.. code-block:: c
+
+   static cmdline_parse_ctx_t ctx[] = {
+       &cmd_quit,
+       &cmd_show_port_stats,
+       NULL
+   };
+
+The context list must be terminated by a NULL entry.
+
+Creating a Command-line Instance
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Once we have our ``ctx`` variable defined,
+we now just need to call the API to create the new command-line instance in our application.
+The basic API is ``cmdline_new`` which will create an interactive command-line with all commands available.
+However, if additional features for interactive use - such as tab-completion -
+are desired, it is recommended that ``cmdline_new_stdin`` be used instead.
+
+A pattern that can be used in applications is to use ``cmdline_new`` for processing any startup commands,
+either from file or from the environment (as is done in the "dpdk-test" application),
+and then using ``cmdline_stdin_new`` thereafter to handle the interactive part.
+For example, to handle a startup file and then provide an interactive prompt:
+
+.. code-block:: c
+
+   struct cmdline *cl;
+   int fd = open(startup_file, O_RDONLY);
+
+   if (fd >= 0) {
+       cl = cmdline_new(ctx, "", fd, STDOUT_FILENO);
+       if (cl == NULL) {
+           /* error handling */
+       }
+       cmdline_interact(cl);
+       cmdline_quit(cl);
+       close(fd);
+   }
+
+   cl = cmdline_stdin_new(ctx, "Proxy>> ");
+   if (cl == NULL) {
+       /* error handling */
+   }
+   cmdline_interact(cl);
+   cmdline_stdin_exit(cl);
+
+
+Multiplexing Multiple Commands to a Single Function
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To reduce the amount of boiler-plate code needed when creating a command-line for an application,
+it is possible to merge a number of commands together to have them call a separate function.
+This can be done in a number of different ways:
+
+* A callback function can be used as the target for a number of different commands.
+  Which command was used for entry to the function can be determined by examining the first parameter,
+  ``parsed_result`` in our examples above.
+
+* For simple string commands, multiple options can be concatenated using the "#" character.
+  For example: ``exit#quit``, specified as a token initializer,
+  will match either on the string "exit" or the string "quit".
+
+As a concrete example,
+these two techniques are used in the DPDK unit test application ``dpdk-test``,
+where a single command ``cmdline_parse_t`` instance is used for all the "dump_<item>" test cases.
+
+.. literalinclude:: ../../../app/test/commands.c
+    :language: c
+    :start-after: Add the dump_* tests cases 8<
+    :end-before: >8 End of add the dump_* tests cases
+
+
+Examples of Command-line Use in DPDK
+------------------------------------
+
+To help the user follow the steps provided above,
+the following DPDK files can be consulted for examples of command-line use.
+
+.. note::
+
+   This is not an exhaustive list of examples of command-line use in DPDK.
+   It is simply a list of a few files that may be of use to the application developer.
+   Some of these referenced files contain more complex examples of use that others.
+
+* ``commands.c/.h`` in ``examples/cmdline``
+
+* ``mp_commands.c/.h`` in ``examples/multi_process/simple_mp``
+
+* ``commands.c`` in ``app/test``
diff --git a/doc/guides/prog_guide/index.rst b/doc/guides/prog_guide/index.rst
index e517f0e259..94964357ff 100644
--- a/doc/guides/prog_guide/index.rst
+++ b/doc/guides/prog_guide/index.rst
@@ -13,6 +13,7 @@ Programmer's Guide
     source_org
     env_abstraction_layer
     log_lib
+    cmdline
     service_cores
     trace_lib
     rcu_lib
-- 
2.39.2

[PATCH v6 2/9] buildtools: script to generate cmdline boilerplate

[Bruce Richardson]

Provide a "dpdk-cmdline-gen.py" script for application developers to
quickly generate the boilerplate code necessary for using the cmdline
library.

Example of use:
The script takes an input file with a list of commands the user wants in
the app, where the parameter variables are tagged with the type.
For example:

	$ cat commands.list
	list
	add <UINT16>x <UINT16>y
	echo <STRING>message
	add socket <STRING>path
	quit

When run through the script as "./dpdk-cmdline-gen.py commands.list",
the output will be the contents of a header file with all the
boilerplate necessary for a commandline instance with those commands.

If the flag --stubs is passed, an output header filename must also be
passed, in which case both a header file with the definitions and a C
file with function stubs in it is written to disk. The separation is so
that the header file can be rewritten at any future point to add more
commands, while the C file can be kept as-is and extended by the user
with any additional functions needed.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 buildtools/dpdk-cmdline-gen.py    | 190 ++++++++++++++++++++++++++++++
 buildtools/meson.build            |   7 ++
 doc/guides/prog_guide/cmdline.rst | 131 +++++++++++++++++++-
 3 files changed, 327 insertions(+), 1 deletion(-)
 create mode 100755 buildtools/dpdk-cmdline-gen.py

diff --git a/buildtools/dpdk-cmdline-gen.py b/buildtools/dpdk-cmdline-gen.py
new file mode 100755
index 0000000000..6cb7610de4
--- /dev/null
+++ b/buildtools/dpdk-cmdline-gen.py
@@ -0,0 +1,190 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2023 Intel Corporation
+#
+"""
+Script to automatically generate boilerplate for using DPDK cmdline library.
+"""
+
+import argparse
+import sys
+
+PARSE_FN_PARAMS = "void *parsed_result, struct cmdline *cl, void *data"
+PARSE_FN_BODY = """
+    /* TODO: command action */
+    RTE_SET_USED(parsed_result);
+    RTE_SET_USED(cl);
+    RTE_SET_USED(data);
+"""
+NUMERIC_TYPES = [
+    "UINT8",
+    "UINT16",
+    "UINT32",
+    "UINT64",
+    "INT8",
+    "INT16",
+    "INT32",
+    "INT64",
+]
+
+
+def process_command(lineno, tokens, comment):
+    """Generate the structures and definitions for a single command."""
+    out = []
+    cfile_out = []
+
+    if tokens[0].startswith("<"):
+        raise ValueError(f"Error line {lineno + 1}: command must start with a literal string")
+
+    name_tokens = []
+    for t in tokens:
+        if t.startswith("<"):
+            break
+        name_tokens.append(t)
+    name = "_".join(name_tokens)
+
+    result_struct = []
+    initializers = []
+    token_list = []
+    for t in tokens:
+        if t.startswith("<"):
+            t_type, t_name = t[1:].split(">")
+            t_val = "NULL"
+        else:
+            t_type = "STRING"
+            t_name = t
+            t_val = f'"{t}"'
+
+        if t_type == "STRING":
+            result_struct.append(f"\tcmdline_fixed_string_t {t_name};")
+            initializers.append(
+                f"static cmdline_parse_token_string_t cmd_{name}_{t_name}_tok =\n"
+                + f"\tTOKEN_STRING_INITIALIZER(struct cmd_{name}_result, {t_name}, {t_val});"
+            )
+        elif t_type in NUMERIC_TYPES:
+            result_struct.append(f"\t{t_type.lower()}_t {t_name};")
+            initializers.append(
+                f"static cmdline_parse_token_num_t cmd_{name}_{t_name}_tok =\n"
+                + f"\tTOKEN_NUM_INITIALIZER(struct cmd_{name}_result, {t_name}, RTE_{t_type});"
+            )
+        elif t_type in ["IP", "IP_ADDR", "IPADDR"]:
+            result_struct.append(f"\tcmdline_ipaddr_t {t_name};")
+            initializers.append(
+                f"cmdline_parse_token_ipaddr_t cmd_{name}_{t_name}_tok =\n"
+                + f"\tTOKEN_IPV4_INITIALIZER(struct cmd_{name}_result, {t_name});"
+            )
+        else:
+            raise TypeError(f"Error line {lineno + 1}: unknown token type '{t_type}'")
+        token_list.append(f"cmd_{name}_{t_name}_tok")
+
+    out.append(f'/* Auto-generated handling for command "{" ".join(tokens)}" */')
+    # output function prototype
+    func_sig = f"void\ncmd_{name}_parsed({PARSE_FN_PARAMS})"
+    out.append(f"extern {func_sig};\n")
+    # output result data structure
+    out.append(f"struct cmd_{name}_result {{\n" + "\n".join(result_struct) + "\n};\n")
+    # output the initializer tokens
+    out.append("\n".join(initializers) + "\n")
+    # output the instance structure
+    out.append(
+        f"static cmdline_parse_inst_t cmd_{name} = {{\n"
+        + f"\t.f = cmd_{name}_parsed,\n"
+        + "\t.data = NULL,\n"
+        + f'\t.help_str = "{comment}",\n'
+        + "\t.tokens = {"
+    )
+    for t in token_list:
+        out.append(f"\t\t(void *)&{t},")
+    out.append("\t\tNULL\n" + "\t}\n" + "};\n")
+    # output function template if C file being written
+    cfile_out.append(f"{func_sig}\n{{{PARSE_FN_BODY}}}\n")
+
+    # return the instance structure name
+    return (f"cmd_{name}", out, cfile_out)
+
+
+def process_commands(infile, hfile, cfile, ctxname):
+    """Generate boilerplate output for a list of commands from infile."""
+    instances = []
+
+    hfile.write(
+        f"""/* File autogenerated by {sys.argv[0]} */
+#ifndef GENERATED_COMMANDS_H
+#define GENERATED_COMMANDS_H
+#include <rte_common.h>
+#include <cmdline.h>
+#include <cmdline_parse_string.h>
+#include <cmdline_parse_num.h>
+#include <cmdline_parse_ipaddr.h>
+
+"""
+    )
+
+    for lineno, line in enumerate(infile.readlines()):
+        if line.lstrip().startswith("#"):
+            continue
+        if "#" not in line:
+            line = line + "#"  # ensure split always works, even if no help text
+        tokens, comment = line.split("#", 1)
+        cmd_inst, h_out, c_out = process_command(lineno, tokens.strip().split(), comment.strip())
+        hfile.write("\n".join(h_out))
+        if cfile:
+            cfile.write("\n".join(c_out))
+        instances.append(cmd_inst)
+
+    inst_join_str = ",\n\t&"
+    hfile.write(
+        f"""
+static __rte_used cmdline_parse_ctx_t {ctxname}[] = {{
+\t&{inst_join_str.join(instances)},
+\tNULL
+}};
+
+#endif /* GENERATED_COMMANDS_H */
+"""
+    )
+
+
+def main():
+    """Application main entry point."""
+    ap = argparse.ArgumentParser(description=__doc__)
+    ap.add_argument(
+        "--stubs",
+        action="store_true",
+        help="Produce C file with empty function stubs for each command",
+    )
+    ap.add_argument(
+        "--output-file",
+        "-o",
+        default="-",
+        help="Output header filename [default to stdout]",
+    )
+    ap.add_argument(
+        "--context-name",
+        default="ctx",
+        help="Name given to the cmdline context variable in the output header [default=ctx]",
+    )
+    ap.add_argument("infile", type=argparse.FileType("r"), help="File with list of commands")
+    args = ap.parse_args()
+
+    if not args.stubs:
+        if args.output_file == "-":
+            process_commands(args.infile, sys.stdout, None, args.context_name)
+        else:
+            with open(args.output_file, "w") as hfile:
+                process_commands(args.infile, hfile, None, args.context_name)
+    else:
+        if not args.output_file.endswith(".h"):
+            ap.error(
+                "-o/--output-file: specify an output filename ending with .h when creating stubs"
+            )
+
+        cfilename = args.output_file[:-2] + ".c"
+        with open(args.output_file, "w") as hfile:
+            with open(cfilename, "w") as cfile:
+                cfile.write(f'#include "{args.output_file}"\n\n')
+                process_commands(args.infile, hfile, cfile, args.context_name)
+
+
+if __name__ == "__main__":
+    main()
diff --git a/buildtools/meson.build b/buildtools/meson.build
index 948ac17dd2..72447b60a0 100644
--- a/buildtools/meson.build
+++ b/buildtools/meson.build
@@ -19,6 +19,13 @@ get_cpu_count_cmd = py3 + files('get-cpu-count.py')
 get_numa_count_cmd = py3 + files('get-numa-count.py')
 get_test_suites_cmd = py3 + files('get-test-suites.py')
 has_hugepages_cmd = py3 + files('has-hugepages.py')
+cmdline_gen_cmd = py3 + files('dpdk-cmdline-gen.py')
+
+# install any build tools that end-users might want also
+install_data([
+            'dpdk-cmdline-gen.py',
+        ],
+        install_dir: 'bin')
 
 # select library and object file format
 pmdinfo = py3 + files('gen-pmdinfo-cfile.py') + [meson.current_build_dir()]
diff --git a/doc/guides/prog_guide/cmdline.rst b/doc/guides/prog_guide/cmdline.rst
index 40f49a30cc..0b96b770e2 100644
--- a/doc/guides/prog_guide/cmdline.rst
+++ b/doc/guides/prog_guide/cmdline.rst
@@ -44,7 +44,136 @@ Adding a command-line instance to an application involves a number of coding ste
 
 6. Within your main application code, create a new command-line instance passing in the context.
 
-The next few subsections will cover each of these steps in more detail,
+Many of these steps can be automated using the script ``dpdk-cmdline-gen.py`` installed by DPDK,
+and found in the ``buildtools`` folder in the source tree.
+This section covers adding a command-line using this script to generate the boiler plate,
+while the following section,
+`Worked Example of Adding Command-line to an Application`_ covers the steps to do so manually.
+
+Creating a Command List File
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The ``dpdk-cmdline-gen.py`` script takes as input a list of commands to be used by the application.
+While these can be piped to it via standard input, using a list file is probably best.
+
+The format of the list file must be:
+
+* Comment lines start with '#' as first non-whitespace character
+
+* One command per line
+
+* Variable fields are prefixed by the type-name in angle-brackets, for example:
+
+  * ``<STRING>message``
+
+  * ``<UINT16>port_id``
+
+  * ``<IP>src_ip``
+
+* The help text for a command is given in the form of a comment on the same line as the command
+
+An example list file, with a variety of (unrelated) commands, is shown below::
+
+   # example list file
+   list                     # show all entries
+   add <UINT16>x <UINT16>y  # add x and y
+   echo <STRING>message     # print message to screen
+   add socket <STRING>path  # add unix socket with the given path
+   quit                     # close the application
+
+Running the Generator Script
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To generate the necessary definitions for a command-line, run ``dpdk-cmdline-gen.py`` passing the list file as parameter.
+The script will output the generated C code to standard output,
+the contents of which are in the form of a C header file.
+Optionally, an output filename may be specified via the ``-o/--output-file`` argument.
+
+The generated content includes:
+
+* The result structure definitions for each command
+
+* The token initializers for each structure field
+
+* An "extern" function prototype for the callback for each command
+
+* A parse context for each command, including the per-command comments as help string
+
+* A command-line context array definition, suitable for passing to ``cmdline_new``
+
+If so desired, the script can also output function stubs for the callback functions for each command.
+This behaviour is triggered by passing the ``--stubs`` flag to the script.
+In this case, an output file must be provided with a filename ending in ".h",
+and the callback stubs will be written to an equivalent ".c" file.
+
+.. note::
+
+   The stubs are written to a separate file,
+   to allow continuous use of the script to regenerate the command-line header,
+   without overwriting any code the user has added to the callback functions.
+   This makes it easy to incrementally add new commands to an existing application.
+
+Providing the Function Callbacks
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+As discussed above, the script output is a header file, containing structure definitions,
+but the callback functions themselves obviously have to be provided by the user.
+These callback functions must be provided as non-static functions in a C file,
+and named ``cmd_<cmdname>_parsed``.
+The function prototypes can be seen in the generated output header.
+
+The "cmdname" part of the function name is built up by combining the non-variable initial tokens in the command.
+So, given the commands in our worked example below: ``quit`` and ``show port stats <n>``,
+the callback functions would be:
+
+.. code:: c
+
+   void
+   cmd_quit_parsed(void *parsed_result, struct cmdline *cl, void *data)
+   {
+        ...
+   }
+
+   void
+   cmd_show_port_stats_parsed(void *parsed_result, struct cmdline *cl, void *data)
+   {
+        ...
+   }
+
+These functions must be provided by the developer, but, as stated above,
+stub functions may be generated by the script automatically using the ``--stubs`` parameter.
+
+The same "cmdname" stem is used in the naming of the generated structures too.
+To get at the results structure for each command above,
+the ``parsed_result`` parameter should be cast to ``struct cmd_quit_result``
+or ``struct cmd_show_port_stats_result`` respectively.
+
+Integrating with the Application
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To integrate the script output with the application,
+we must ``#include`` the generated header into our applications C file,
+and then have the command-line created via either ``cmdline_new`` or ``cmdline_stdin_new``.
+The first parameter to the function call should be the context array in the generated header file,
+``ctx`` by default. (Modifiable via script parameter).
+
+The callback functions may be in this same file, or in a separate one -
+they just need to be available to the linker at build-time.
+
+Limitations of the Script Approach
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The script approach works for most commands that a user may wish to add to an application.
+However, it does not support the full range of functions possible with the DPDK command-line library.
+For example,
+it is not possible using the script to multiplex multiple commands into a single callback function.
+To use this functionality, the user should follow the instructions in the next section
+`Worked Example of Adding Command-line to an Application`_ to manually configure a command-line instance.
+
+Worked Example of Adding Command-line to an Application
+-------------------------------------------------------
+
+The next few subsections will cover each of the steps listed in `Adding Command-line to an Application`_ in more detail,
 working through an example to add two commands to a command-line instance.
 Those two commands will be:
 
-- 
2.39.2

[PATCH v6 3/9] ci: allow use of DPDK tools when building examples

[Bruce Richardson]

To allow use of the DPDK python scripts (installed in $(prefix)/bin)
from within the makefiles of our examples, we need to export the PATH
variable with the location of our installed scripts from within our CI
scripts. This matches what is already done for other paths e.g. the
PKG_CONFIG_PATH variable.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
Reviewed-by: Aaron Conole <aconole@redhat.com>
---
 .ci/linux-build.sh | 1 +
 1 file changed, 1 insertion(+)

diff --git a/.ci/linux-build.sh b/.ci/linux-build.sh
index e0b62bac90..3db9d9de6e 100755
--- a/.ci/linux-build.sh
+++ b/.ci/linux-build.sh
@@ -174,6 +174,7 @@ fi
 if [ "$BUILD_EXAMPLES" = "true" ]; then
     [ -d install ] || DESTDIR=$(pwd)/install meson install -C build
     export LD_LIBRARY_PATH=$(dirname $(find $(pwd)/install -name librte_eal.so)):$LD_LIBRARY_PATH
+    export PATH=$(dirname $(find $(pwd)/install -name dpdk-devbind.py)):$PATH
     export PKG_CONFIG_PATH=$(dirname $(find $(pwd)/install -name libdpdk.pc)):$PKG_CONFIG_PATH
     export PKGCONF="pkg-config --define-prefix"
     find build/examples -maxdepth 1 -type f -name "dpdk-*" |
-- 
2.39.2

[PATCH v6 4/9] examples/simple_mp: auto-generate cmdline boilerplate

[Bruce Richardson]

Use the dpdk-cmdline-gen script to autogenerate all the boilerplate
structs and defines for the commandline part of the app.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 examples/multi_process/simple_mp/Makefile     |  12 +-
 examples/multi_process/simple_mp/meson.build  |   9 ++
 .../multi_process/simple_mp/mp_commands.c     | 106 ++----------------
 .../multi_process/simple_mp/mp_commands.h     |  14 ---
 .../multi_process/simple_mp/mp_commands.list  |   3 +
 5 files changed, 30 insertions(+), 114 deletions(-)
 delete mode 100644 examples/multi_process/simple_mp/mp_commands.h
 create mode 100644 examples/multi_process/simple_mp/mp_commands.list

diff --git a/examples/multi_process/simple_mp/Makefile b/examples/multi_process/simple_mp/Makefile
index 1d0a260e64..890b6b7e62 100644
--- a/examples/multi_process/simple_mp/Makefile
+++ b/examples/multi_process/simple_mp/Makefile
@@ -6,6 +6,7 @@ APP = simple_mp
 
 # all source are stored in SRCS-y
 SRCS-y := main.c mp_commands.c
+SRC-DEPS := build/mp_commands.h
 
 PKGCONF ?= pkg-config
 
@@ -22,10 +23,13 @@ static: build/$(APP)-static
 	ln -sf $(APP)-static build/$(APP)
 
 PC_FILE := $(shell $(PKGCONF) --path libdpdk 2>/dev/null)
-CFLAGS += -O3 $(shell $(PKGCONF) --cflags libdpdk)
+CFLAGS += -O3 $(shell $(PKGCONF) --cflags libdpdk) -I build/
 LDFLAGS_SHARED = $(shell $(PKGCONF) --libs libdpdk)
 LDFLAGS_STATIC = $(shell $(PKGCONF) --static --libs libdpdk)
 
+build/mp_commands.h: mp_commands.list Makefile
+	dpdk-cmdline-gen.py -o $@ --context-name=simple_mp_ctx $<
+
 ifeq ($(MAKECMDGOALS),static)
 # check for broken pkg-config
 ifeq ($(shell echo $(LDFLAGS_STATIC) | grep 'whole-archive.*l:lib.*no-whole-archive'),)
@@ -36,10 +40,10 @@ endif
 
 CFLAGS += -DALLOW_EXPERIMENTAL_API
 
-build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
+build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build $(SRC-DEPS)
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
 
-build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build
+build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build $(SRC-DEPS)
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_STATIC)
 
 build:
@@ -47,5 +51,5 @@ build:
 
 .PHONY: clean
 clean:
-	rm -f build/$(APP) build/$(APP)-static build/$(APP)-shared
+	rm -f build/$(APP) build/$(APP)-static build/$(APP)-shared build/mp_commands.h
 	test -d build && rmdir -p build || true
diff --git a/examples/multi_process/simple_mp/meson.build b/examples/multi_process/simple_mp/meson.build
index 359af4384d..e99b7a3f6f 100644
--- a/examples/multi_process/simple_mp/meson.build
+++ b/examples/multi_process/simple_mp/meson.build
@@ -7,7 +7,16 @@
 # DPDK instance, use 'make'
 
 allow_experimental_apis = true
+
+cmd_h = custom_target('commands_hdr',
+	output: 'mp_commands.h',
+	input: files('mp_commands.list'),
+	capture: true,
+	command: [cmdline_gen_cmd, '--context-name=simple_mp_ctx', '@INPUT@']
+)
+
 sources = files(
         'mp_commands.c',
         'main.c',
 )
+sources += cmd_h
diff --git a/examples/multi_process/simple_mp/mp_commands.c b/examples/multi_process/simple_mp/mp_commands.c
index a5f91b00be..df9fa94208 100644
--- a/examples/multi_process/simple_mp/mp_commands.c
+++ b/examples/multi_process/simple_mp/mp_commands.c
@@ -1,44 +1,18 @@
 /* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2010-2014 Intel Corporation
+ * Copyright(c) 2010-2023 Intel Corporation
  */
-#include <stdint.h>
-#include <string.h>
-#include <stdlib.h>
-#include <stdarg.h>
-#include <inttypes.h>
-#include <stdio.h>
-#include <termios.h>
-#include <errno.h>
-#include <sys/queue.h>
-
-#include <rte_common.h>
-#include <rte_memory.h>
-#include <rte_eal.h>
-#include <rte_branch_prediction.h>
-#include <rte_launch.h>
-#include <rte_log.h>
-#include <rte_per_lcore.h>
-#include <rte_lcore.h>
 #include <rte_ring.h>
-#include <rte_debug.h>
 #include <rte_mempool.h>
 #include <rte_string_fns.h>
 
-#include <cmdline_rdline.h>
-#include <cmdline_parse.h>
-#include <cmdline_parse_string.h>
-#include <cmdline_socket.h>
-#include <cmdline.h>
 #include "mp_commands.h"
 
-/**********************************************************/
-
-struct cmd_send_result {
-	cmdline_fixed_string_t action;
-	cmdline_fixed_string_t message;
-};
+extern struct rte_ring *send_ring, *recv_ring;
+extern struct rte_mempool *message_pool;
+extern volatile int quit;
 
-static void cmd_send_parsed(void *parsed_result,
+void
+cmd_send_parsed(void *parsed_result,
 		__rte_unused struct cmdline *cl,
 		__rte_unused void *data)
 {
@@ -54,29 +28,8 @@ static void cmd_send_parsed(void *parsed_result,
 	}
 }
 
-cmdline_parse_token_string_t cmd_send_action =
-	TOKEN_STRING_INITIALIZER(struct cmd_send_result, action, "send");
-cmdline_parse_token_string_t cmd_send_message =
-	TOKEN_STRING_INITIALIZER(struct cmd_send_result, message, NULL);
-
-cmdline_parse_inst_t cmd_send = {
-	.f = cmd_send_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "send a string to another process",
-	.tokens = {        /* token list, NULL terminated */
-			(void *)&cmd_send_action,
-			(void *)&cmd_send_message,
-			NULL,
-	},
-};
-
-/**********************************************************/
-
-struct cmd_quit_result {
-	cmdline_fixed_string_t quit;
-};
-
-static void cmd_quit_parsed(__rte_unused void *parsed_result,
+void
+cmd_quit_parsed(__rte_unused void *parsed_result,
 			    struct cmdline *cl,
 			    __rte_unused void *data)
 {
@@ -84,26 +37,8 @@ static void cmd_quit_parsed(__rte_unused void *parsed_result,
 	cmdline_quit(cl);
 }
 
-cmdline_parse_token_string_t cmd_quit_quit =
-	TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit");
-
-cmdline_parse_inst_t cmd_quit = {
-	.f = cmd_quit_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "close the application",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_quit_quit,
-		NULL,
-	},
-};
-
-/**********************************************************/
-
-struct cmd_help_result {
-	cmdline_fixed_string_t help;
-};
-
-static void cmd_help_parsed(__rte_unused void *parsed_result,
+void
+cmd_help_parsed(__rte_unused void *parsed_result,
 			    struct cmdline *cl,
 			    __rte_unused void *data)
 {
@@ -112,24 +47,3 @@ static void cmd_help_parsed(__rte_unused void *parsed_result,
 			"send commands to the simple app. Commands supported are:\n\n"
 			"- send [string]\n" "- help\n" "- quit\n\n");
 }
-
-cmdline_parse_token_string_t cmd_help_help =
-	TOKEN_STRING_INITIALIZER(struct cmd_help_result, help, "help");
-
-cmdline_parse_inst_t cmd_help = {
-	.f = cmd_help_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "show help",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_help_help,
-		NULL,
-	},
-};
-
-/****** CONTEXT (list of instruction) */
-cmdline_parse_ctx_t simple_mp_ctx[] = {
-		(cmdline_parse_inst_t *)&cmd_send,
-		(cmdline_parse_inst_t *)&cmd_quit,
-		(cmdline_parse_inst_t *)&cmd_help,
-	NULL,
-};
diff --git a/examples/multi_process/simple_mp/mp_commands.h b/examples/multi_process/simple_mp/mp_commands.h
deleted file mode 100644
index 5d67413e7c..0000000000
--- a/examples/multi_process/simple_mp/mp_commands.h
+++ /dev/null
@@ -1,14 +0,0 @@
-/* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2010-2014 Intel Corporation
- */
-
-#ifndef _SIMPLE_MP_COMMANDS_H_
-#define _SIMPLE_MP_COMMANDS_H_
-
-extern struct rte_ring *send_ring;
-extern struct rte_mempool *message_pool;
-extern volatile int quit;
-
-extern cmdline_parse_ctx_t simple_mp_ctx[];
-
-#endif /* _SIMPLE_MP_COMMANDS_H_ */
diff --git a/examples/multi_process/simple_mp/mp_commands.list b/examples/multi_process/simple_mp/mp_commands.list
new file mode 100644
index 0000000000..c8890cb071
--- /dev/null
+++ b/examples/multi_process/simple_mp/mp_commands.list
@@ -0,0 +1,3 @@
+send <STRING>message  # send a string to another process
+help                  # show help
+quit                  # close the application
-- 
2.39.2

[PATCH v6 5/9] examples/hotplug_mp: auto-generate cmdline boilerplate

[Bruce Richardson]

Use the dpdk-cmdline-gen script to autogenerate all the boilerplate
structs and defines for the commandline part of the app.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 examples/multi_process/hotplug_mp/Makefile    |  12 +-
 examples/multi_process/hotplug_mp/commands.c  | 147 ++----------------
 examples/multi_process/hotplug_mp/commands.h  |  10 --
 .../multi_process/hotplug_mp/commands.list    |   8 +
 examples/multi_process/hotplug_mp/meson.build |   9 ++
 5 files changed, 38 insertions(+), 148 deletions(-)
 delete mode 100644 examples/multi_process/hotplug_mp/commands.h
 create mode 100644 examples/multi_process/hotplug_mp/commands.list

diff --git a/examples/multi_process/hotplug_mp/Makefile b/examples/multi_process/hotplug_mp/Makefile
index 6b20d6e49a..81ee85cd6b 100644
--- a/examples/multi_process/hotplug_mp/Makefile
+++ b/examples/multi_process/hotplug_mp/Makefile
@@ -6,6 +6,7 @@ APP = hotplug_mp
 
 # all source are stored in SRCS-y
 SRCS-y := main.c commands.c
+SRC-DEPS := build/commands.h
 
 PKGCONF ?= pkg-config
 
@@ -22,10 +23,13 @@ static: build/$(APP)-static
 	ln -sf $(APP)-static build/$(APP)
 
 PC_FILE := $(shell $(PKGCONF) --path libdpdk 2>/dev/null)
-CFLAGS += -O3 $(shell $(PKGCONF) --cflags libdpdk)
+CFLAGS += -O3 $(shell $(PKGCONF) --cflags libdpdk) -I build/
 LDFLAGS_SHARED = $(shell $(PKGCONF) --libs libdpdk)
 LDFLAGS_STATIC = $(shell $(PKGCONF) --static --libs libdpdk)
 
+build/commands.h: commands.list Makefile
+	dpdk-cmdline-gen.py -o $@ --context-name=main_ctx $<
+
 ifeq ($(MAKECMDGOALS),static)
 # check for broken pkg-config
 ifeq ($(shell echo $(LDFLAGS_STATIC) | grep 'whole-archive.*l:lib.*no-whole-archive'),)
@@ -36,10 +40,10 @@ endif
 
 CFLAGS += -DALLOW_EXPERIMENTAL_API
 
-build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
+build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build $(SRC-DEPS)
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
 
-build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build
+build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build  $(SRC-DEPS)
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_STATIC)
 
 build:
@@ -47,5 +51,5 @@ build:
 
 .PHONY: clean
 clean:
-	rm -f build/$(APP) build/$(APP)-static build/$(APP)-shared
+	rm -f build/$(APP) build/$(APP)-static build/$(APP)-shared build/commands.h
 	test -d build && rmdir -p build || true
diff --git a/examples/multi_process/hotplug_mp/commands.c b/examples/multi_process/hotplug_mp/commands.c
index 88f44e00a0..900eb9f774 100644
--- a/examples/multi_process/hotplug_mp/commands.c
+++ b/examples/multi_process/hotplug_mp/commands.c
@@ -1,24 +1,12 @@
 /* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2018 Intel Corporation.
+ * Copyright(c) 2018-2023 Intel Corporation.
  */
 
-#include <cmdline_rdline.h>
-#include <cmdline_parse.h>
-#include <cmdline_parse_ipaddr.h>
-#include <cmdline_parse_num.h>
-#include <cmdline_parse_string.h>
-#include <cmdline.h>
-
 #include <rte_bus.h>
 #include <rte_ethdev.h>
+#include "commands.h"
 
-/**********************************************************/
-
-struct cmd_help_result {
-	cmdline_fixed_string_t help;
-};
-
-static void cmd_help_parsed(__rte_unused void *parsed_result,
+void cmd_help_parsed(__rte_unused void *parsed_result,
 			    struct cmdline *cl,
 			    __rte_unused void *data)
 {
@@ -29,52 +17,16 @@ static void cmd_help_parsed(__rte_unused void *parsed_result,
 		       "- list\n\n");
 }
 
-cmdline_parse_token_string_t cmd_help_help =
-	TOKEN_STRING_INITIALIZER(struct cmd_help_result, help, "help");
-
-cmdline_parse_inst_t cmd_help = {
-	.f = cmd_help_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "show help",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_help_help,
-		NULL,
-	},
-};
-
-/**********************************************************/
-
-struct cmd_quit_result {
-	cmdline_fixed_string_t quit;
-};
-
-static void cmd_quit_parsed(__rte_unused void *parsed_result,
+void
+cmd_quit_parsed(__rte_unused void *parsed_result,
 			    struct cmdline *cl,
 			    __rte_unused void *data)
 {
 	cmdline_quit(cl);
 }
 
-cmdline_parse_token_string_t cmd_quit_quit =
-	TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit");
-
-cmdline_parse_inst_t cmd_quit = {
-	.f = cmd_quit_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "quit",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_quit_quit,
-		NULL,
-	},
-};
-
-/**********************************************************/
-
-struct cmd_list_result {
-	cmdline_fixed_string_t list;
-};
-
-static void cmd_list_parsed(__rte_unused void *parsed_result,
+void
+cmd_list_parsed(__rte_unused void *parsed_result,
 			    struct cmdline *cl,
 			    __rte_unused void *data)
 {
@@ -92,31 +44,12 @@ static void cmd_list_parsed(__rte_unused void *parsed_result,
 	}
 }
 
-cmdline_parse_token_string_t cmd_list_list =
-	TOKEN_STRING_INITIALIZER(struct cmd_list_result, list, "list");
-
-cmdline_parse_inst_t cmd_list = {
-	.f = cmd_list_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "list all devices",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_list_list,
-		NULL,
-	},
-};
-
-/**********************************************************/
-
-struct cmd_dev_attach_result {
-	cmdline_fixed_string_t attach;
-	cmdline_fixed_string_t devargs;
-};
-
-static void cmd_dev_attach_parsed(void *parsed_result,
+void
+cmd_attach_parsed(void *parsed_result,
 				  struct cmdline *cl,
 				  __rte_unused void *data)
 {
-	struct cmd_dev_attach_result *res = parsed_result;
+	struct cmd_attach_result *res = parsed_result;
 	struct rte_devargs da;
 
 	memset(&da, 0, sizeof(da));
@@ -134,35 +67,12 @@ static void cmd_dev_attach_parsed(void *parsed_result,
 	rte_devargs_reset(&da);
 }
 
-cmdline_parse_token_string_t cmd_dev_attach_attach =
-	TOKEN_STRING_INITIALIZER(struct cmd_dev_attach_result, attach,
-				 "attach");
-cmdline_parse_token_string_t cmd_dev_attach_devargs =
-	TOKEN_STRING_INITIALIZER(struct cmd_dev_attach_result, devargs, NULL);
-
-cmdline_parse_inst_t cmd_attach_device = {
-	.f = cmd_dev_attach_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "attach a device",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_dev_attach_attach,
-		(void *)&cmd_dev_attach_devargs,
-		NULL,
-	},
-};
-
-/**********************************************************/
-
-struct cmd_dev_detach_result {
-	cmdline_fixed_string_t detach;
-	cmdline_fixed_string_t devargs;
-};
-
-static void cmd_dev_detach_parsed(void *parsed_result,
+void
+cmd_detach_parsed(void *parsed_result,
 				   struct cmdline *cl,
 				   __rte_unused void *data)
 {
-	struct cmd_dev_detach_result *res = parsed_result;
+	struct cmd_detach_result *res = parsed_result;
 	struct rte_devargs da;
 
 	memset(&da, 0, sizeof(da));
@@ -181,34 +91,3 @@ static void cmd_dev_detach_parsed(void *parsed_result,
 			da.name);
 	rte_devargs_reset(&da);
 }
-
-cmdline_parse_token_string_t cmd_dev_detach_detach =
-	TOKEN_STRING_INITIALIZER(struct cmd_dev_detach_result, detach,
-				 "detach");
-
-cmdline_parse_token_string_t cmd_dev_detach_devargs =
-	TOKEN_STRING_INITIALIZER(struct cmd_dev_detach_result, devargs, NULL);
-
-cmdline_parse_inst_t cmd_detach_device = {
-	.f = cmd_dev_detach_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "detach a device",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_dev_detach_detach,
-		(void *)&cmd_dev_detach_devargs,
-		NULL,
-	},
-};
-
-/**********************************************************/
-/**********************************************************/
-/****** CONTEXT (list of instruction) */
-
-cmdline_parse_ctx_t main_ctx[] = {
-	(cmdline_parse_inst_t *)&cmd_help,
-	(cmdline_parse_inst_t *)&cmd_quit,
-	(cmdline_parse_inst_t *)&cmd_list,
-	(cmdline_parse_inst_t *)&cmd_attach_device,
-	(cmdline_parse_inst_t *)&cmd_detach_device,
-	NULL,
-};
diff --git a/examples/multi_process/hotplug_mp/commands.h b/examples/multi_process/hotplug_mp/commands.h
deleted file mode 100644
index afcf177dba..0000000000
--- a/examples/multi_process/hotplug_mp/commands.h
+++ /dev/null
@@ -1,10 +0,0 @@
-/* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2018 Intel Corporation
- */
-
-#ifndef _COMMANDS_H_
-#define _COMMANDS_H_
-
-extern cmdline_parse_ctx_t main_ctx[];
-
-#endif /* _COMMANDS_H_ */
diff --git a/examples/multi_process/hotplug_mp/commands.list b/examples/multi_process/hotplug_mp/commands.list
new file mode 100644
index 0000000000..8064df77c0
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/commands.list
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2023 Intel Corporation
+#
+attach <STRING>devargs     # attach a device
+detach <STRING>devargs     # detach a device
+list                       # list all devices
+help                       # show help
+quit                       # quit
diff --git a/examples/multi_process/hotplug_mp/meson.build b/examples/multi_process/hotplug_mp/meson.build
index a1ad98ca2e..7a0e9ca47a 100644
--- a/examples/multi_process/hotplug_mp/meson.build
+++ b/examples/multi_process/hotplug_mp/meson.build
@@ -7,7 +7,16 @@
 # DPDK instance, use 'make'
 
 allow_experimental_apis = true
+
+cmd_h = custom_target('commands_hdr',
+	output: 'commands.h',
+	input: files('commands.list'),
+	capture: true,
+	command: [cmdline_gen_cmd, '--context-name=main_ctx', '@INPUT@']
+)
+
 sources = files(
         'commands.c',
         'main.c',
 )
+sources += cmd_h
-- 
2.39.2

[PATCH v6 6/9] examples/bond: auto-generate cmdline boilerplate

[Bruce Richardson]

Use the dpdk-cmdline-gen script to autogenerate all the boilerplate
structs and defines for the commandline part of the app.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>

---
Note: the original help text on some of the commands in this example
  were not useful "this command do not handle any arguments". Therefore,
  when converting over to the command script, the per-command help
  info has been updated with reference to the code rather than a literal
  transfer of the existing help text, as was done with the previous 2
  example apps.
---
 examples/bond/Makefile                        |  12 +-
 examples/bond/commands.list                   |   6 +
 examples/bond/main.c                          | 161 ++----------------
 examples/bond/main.h                          |  10 --
 examples/bond/meson.build                     |   8 +
 .../multi_process/simple_mp/mp_commands.list  |   3 +
 6 files changed, 40 insertions(+), 160 deletions(-)
 create mode 100644 examples/bond/commands.list
 delete mode 100644 examples/bond/main.h

diff --git a/examples/bond/Makefile b/examples/bond/Makefile
index ad711a5bee..d87c7a32ba 100644
--- a/examples/bond/Makefile
+++ b/examples/bond/Makefile
@@ -6,6 +6,7 @@ APP = bond_app
 
 # all source are stored in SRCS-y
 SRCS-y := main.c
+SRC-DEPS := build/commands.h
 
 PKGCONF ?= pkg-config
 
@@ -24,10 +25,13 @@ static: build/$(APP)-static
 LDFLAGS += -lrte_net_bond
 
 PC_FILE := $(shell $(PKGCONF) --path libdpdk 2>/dev/null)
-CFLAGS += -O3 $(shell $(PKGCONF) --cflags libdpdk)
+CFLAGS += -O3 $(shell $(PKGCONF) --cflags libdpdk) -I build/
 LDFLAGS_SHARED = $(shell $(PKGCONF) --libs libdpdk)
 LDFLAGS_STATIC = $(shell $(PKGCONF) --static --libs libdpdk)
 
+build/commands.h: commands.list Makefile
+	dpdk-cmdline-gen.py -o $@ --context-name=main_ctx $<
+
 ifeq ($(MAKECMDGOALS),static)
 # check for broken pkg-config
 ifeq ($(shell echo $(LDFLAGS_STATIC) | grep 'whole-archive.*l:lib.*no-whole-archive'),)
@@ -38,10 +42,10 @@ endif
 
 CFLAGS += -DALLOW_EXPERIMENTAL_API
 
-build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
+build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build $(SRC-DEPS)
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
 
-build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build
+build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build  $(SRC-DEPS)
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_STATIC)
 
 build:
@@ -49,5 +53,5 @@ build:
 
 .PHONY: clean
 clean:
-	rm -f build/$(APP) build/$(APP)-static build/$(APP)-shared
+	rm -f build/$(APP) build/$(APP)-static build/$(APP)-shared build/*.h
 	test -d build && rmdir -p build || true
diff --git a/examples/bond/commands.list b/examples/bond/commands.list
new file mode 100644
index 0000000000..a10bf75708
--- /dev/null
+++ b/examples/bond/commands.list
@@ -0,0 +1,6 @@
+send <IP>ip   # sends one ARPrequest through bonding for IP
+start         # starts listening if not started at startup
+stop          # stops listening
+show          # shows some bond info, e.g. active members
+help          # show help
+quit          # close application
diff --git a/examples/bond/main.c b/examples/bond/main.c
index 90f422ec11..8528abf675 100644
--- a/examples/bond/main.c
+++ b/examples/bond/main.c
@@ -45,16 +45,8 @@
 #include <rte_cpuflags.h>
 #include <rte_eth_bond.h>
 
-#include <cmdline_rdline.h>
-#include <cmdline_parse.h>
-#include <cmdline_parse_num.h>
-#include <cmdline_parse_string.h>
-#include <cmdline_parse_ipaddr.h>
-#include <cmdline_parse_etheraddr.h>
 #include <cmdline_socket.h>
-#include <cmdline.h>
-
-#include "main.h"
+#include "commands.h"
 
 #define RTE_LOGTYPE_DCB RTE_LOGTYPE_USER1
 
@@ -462,11 +454,7 @@ static int lcore_main(__rte_unused void *arg1)
 	return 0;
 }
 
-struct cmd_obj_send_result {
-	cmdline_fixed_string_t action;
-	cmdline_ipaddr_t ip;
-};
-static inline void get_string(struct cmd_obj_send_result *res, char *buf, uint8_t size)
+static inline void get_string(struct cmd_send_result *res, char *buf, uint8_t size)
 {
 	snprintf(buf, size, NIPQUAD_FMT,
 		((unsigned)((unsigned char *)&(res->ip.addr.ipv4))[0]),
@@ -475,12 +463,11 @@ static inline void get_string(struct cmd_obj_send_result *res, char *buf, uint8_
 		((unsigned)((unsigned char *)&(res->ip.addr.ipv4))[3])
 		);
 }
-static void cmd_obj_send_parsed(void *parsed_result,
-		__rte_unused struct cmdline *cl,
-			       __rte_unused void *data)
+void
+cmd_send_parsed(void *parsed_result, __rte_unused struct cmdline *cl, __rte_unused void *data)
 {
 
-	struct cmd_obj_send_result *res = parsed_result;
+	struct cmd_send_result *res = parsed_result;
 	char ip_str[INET6_ADDRSTRLEN];
 
 	struct rte_ether_addr bond_mac_addr;
@@ -544,29 +531,8 @@ static void cmd_obj_send_parsed(void *parsed_result,
 	cmdline_printf(cl, "\n");
 }
 
-cmdline_parse_token_string_t cmd_obj_action_send =
-	TOKEN_STRING_INITIALIZER(struct cmd_obj_send_result, action, "send");
-cmdline_parse_token_ipaddr_t cmd_obj_ip =
-	TOKEN_IPV4_INITIALIZER(struct cmd_obj_send_result, ip);
-
-cmdline_parse_inst_t cmd_obj_send = {
-	.f = cmd_obj_send_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "send client_ip",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_obj_action_send,
-		(void *)&cmd_obj_ip,
-		NULL,
-	},
-};
-
-struct cmd_start_result {
-	cmdline_fixed_string_t start;
-};
-
-static void cmd_start_parsed(__rte_unused void *parsed_result,
-			       struct cmdline *cl,
-			       __rte_unused void *data)
+void
+cmd_start_parsed(__rte_unused void *parsed_result, struct cmdline *cl, __rte_unused void *data)
 {
 	int worker_core_id = rte_lcore_id();
 
@@ -605,26 +571,8 @@ static void cmd_start_parsed(__rte_unused void *parsed_result,
 		);
 }
 
-cmdline_parse_token_string_t cmd_start_start =
-	TOKEN_STRING_INITIALIZER(struct cmd_start_result, start, "start");
-
-cmdline_parse_inst_t cmd_start = {
-	.f = cmd_start_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "starts listening if not started at startup",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_start_start,
-		NULL,
-	},
-};
-
-struct cmd_help_result {
-	cmdline_fixed_string_t help;
-};
-
-static void cmd_help_parsed(__rte_unused void *parsed_result,
-			    struct cmdline *cl,
-			    __rte_unused void *data)
+void
+cmd_help_parsed(__rte_unused void *parsed_result, struct cmdline *cl, __rte_unused void *data)
 {
 	cmdline_printf(cl,
 			"ALB - link bonding mode 6 example\n"
@@ -637,26 +585,8 @@ static void cmd_help_parsed(__rte_unused void *parsed_result,
 		       );
 }
 
-cmdline_parse_token_string_t cmd_help_help =
-	TOKEN_STRING_INITIALIZER(struct cmd_help_result, help, "help");
-
-cmdline_parse_inst_t cmd_help = {
-	.f = cmd_help_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "show help",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_help_help,
-		NULL,
-	},
-};
-
-struct cmd_stop_result {
-	cmdline_fixed_string_t stop;
-};
-
-static void cmd_stop_parsed(__rte_unused void *parsed_result,
-			    struct cmdline *cl,
-			    __rte_unused void *data)
+void
+cmd_stop_parsed(__rte_unused void *parsed_result, struct cmdline *cl, __rte_unused void *data)
 {
 	rte_spinlock_lock(&global_flag_stru_p->lock);
 	if (global_flag_stru_p->LcoreMainIsRunning == 0)	{
@@ -678,26 +608,8 @@ static void cmd_stop_parsed(__rte_unused void *parsed_result,
 	rte_spinlock_unlock(&global_flag_stru_p->lock);
 }
 
-cmdline_parse_token_string_t cmd_stop_stop =
-	TOKEN_STRING_INITIALIZER(struct cmd_stop_result, stop, "stop");
-
-cmdline_parse_inst_t cmd_stop = {
-	.f = cmd_stop_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "this command do not handle any arguments",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_stop_stop,
-		NULL,
-	},
-};
-
-struct cmd_quit_result {
-	cmdline_fixed_string_t quit;
-};
-
-static void cmd_quit_parsed(__rte_unused void *parsed_result,
-			    struct cmdline *cl,
-			    __rte_unused void *data)
+void
+cmd_quit_parsed(__rte_unused void *parsed_result, struct cmdline *cl, __rte_unused void *data)
 {
 	rte_spinlock_lock(&global_flag_stru_p->lock);
 	if (global_flag_stru_p->LcoreMainIsRunning == 0)	{
@@ -721,26 +633,8 @@ static void cmd_quit_parsed(__rte_unused void *parsed_result,
 	cmdline_quit(cl);
 }
 
-cmdline_parse_token_string_t cmd_quit_quit =
-	TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit");
-
-cmdline_parse_inst_t cmd_quit = {
-	.f = cmd_quit_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "this command do not handle any arguments",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_quit_quit,
-		NULL,
-	},
-};
-
-struct cmd_show_result {
-	cmdline_fixed_string_t show;
-};
-
-static void cmd_show_parsed(__rte_unused void *parsed_result,
-			    struct cmdline *cl,
-			    __rte_unused void *data)
+void
+cmd_show_parsed(__rte_unused void *parsed_result, struct cmdline *cl, __rte_unused void *data)
 {
 	uint16_t members[16] = {0};
 	uint8_t len = 16;
@@ -772,31 +666,6 @@ static void cmd_show_parsed(__rte_unused void *parsed_result,
 	rte_spinlock_unlock(&global_flag_stru_p->lock);
 }
 
-cmdline_parse_token_string_t cmd_show_show =
-	TOKEN_STRING_INITIALIZER(struct cmd_show_result, show, "show");
-
-cmdline_parse_inst_t cmd_show = {
-	.f = cmd_show_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "this command do not handle any arguments",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_show_show,
-		NULL,
-	},
-};
-
-/****** CONTEXT (list of instruction) */
-
-cmdline_parse_ctx_t main_ctx[] = {
-	(cmdline_parse_inst_t *)&cmd_start,
-	(cmdline_parse_inst_t *)&cmd_obj_send,
-	(cmdline_parse_inst_t *)&cmd_stop,
-	(cmdline_parse_inst_t *)&cmd_show,
-	(cmdline_parse_inst_t *)&cmd_quit,
-	(cmdline_parse_inst_t *)&cmd_help,
-	NULL,
-};
-
 /* prompt function, called from main on MAIN lcore */
 static void prompt(__rte_unused void *arg1)
 {
diff --git a/examples/bond/main.h b/examples/bond/main.h
deleted file mode 100644
index f91ed9c885..0000000000
--- a/examples/bond/main.h
+++ /dev/null
@@ -1,10 +0,0 @@
-/* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2010-2015 Intel Corporation
- */
-
-#ifndef _MAIN_H_
-#define _MAIN_H_
-
-int main(int argc, char *argv[]);
-
-#endif /* ifndef _MAIN_H_ */
diff --git a/examples/bond/meson.build b/examples/bond/meson.build
index ed22b7887d..bfbec04ecd 100644
--- a/examples/bond/meson.build
+++ b/examples/bond/meson.build
@@ -11,3 +11,11 @@ allow_experimental_apis = true
 sources = files(
         'main.c',
 )
+
+cmd_h = custom_target('commands_hdr',
+	output: 'commands.h',
+	input: files('commands.list'),
+	capture: true,
+	command: [cmdline_gen_cmd, '--context-name=main_ctx', '@INPUT@']
+)
+sources += cmd_h
diff --git a/examples/multi_process/simple_mp/mp_commands.list b/examples/multi_process/simple_mp/mp_commands.list
index c8890cb071..afce5eb1c1 100644
--- a/examples/multi_process/simple_mp/mp_commands.list
+++ b/examples/multi_process/simple_mp/mp_commands.list
@@ -1,3 +1,6 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2023 Intel Corporation
+#
 send <STRING>message  # send a string to another process
 help                  # show help
 quit                  # close the application
-- 
2.39.2

[PATCH v6 7/9] examples/vdpa: auto-generate cmdline boilerplate

[Bruce Richardson]

Use the dpdk-cmdline-gen script to autogenerate all the boilerplate
structs and defines for the commandline part of the app.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 examples/vdpa/Makefile      |  12 ++--
 examples/vdpa/commands.list |   8 +++
 examples/vdpa/main.c        | 131 ++----------------------------------
 examples/vdpa/meson.build   |   7 ++
 4 files changed, 30 insertions(+), 128 deletions(-)
 create mode 100644 examples/vdpa/commands.list

diff --git a/examples/vdpa/Makefile b/examples/vdpa/Makefile
index d974db4f40..aa60a000cf 100644
--- a/examples/vdpa/Makefile
+++ b/examples/vdpa/Makefile
@@ -6,6 +6,7 @@ APP = vdpa
 
 # all source are stored in SRCS-y
 SRCS-y := main.c
+SRC-DEPS := build/commands.h
 CFLAGS += -DALLOW_EXPERIMENTAL_API
 
 PKGCONF ?= pkg-config
@@ -23,10 +24,13 @@ static: build/$(APP)-static
 	ln -sf $(APP)-static build/$(APP)
 
 PC_FILE := $(shell $(PKGCONF) --path libdpdk 2>/dev/null)
-CFLAGS += -O3 $(shell $(PKGCONF) --cflags libdpdk)
+CFLAGS += -O3 $(shell $(PKGCONF) --cflags libdpdk) -I build/
 LDFLAGS_SHARED = $(shell $(PKGCONF) --libs libdpdk)
 LDFLAGS_STATIC = $(shell $(PKGCONF) --static --libs libdpdk)
 
+build/commands.h: commands.list Makefile
+	dpdk-cmdline-gen.py -o $@ --context-name=main_ctx $<
+
 ifeq ($(MAKECMDGOALS),static)
 # check for broken pkg-config
 ifeq ($(shell echo $(LDFLAGS_STATIC) | grep 'whole-archive.*l:lib.*no-whole-archive'),)
@@ -35,10 +39,10 @@ $(error "Cannot generate statically-linked binaries with this version of pkg-con
 endif
 endif
 
-build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
+build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build $(SRC-DEPS)
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
 
-build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build
+build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build $(SRC-DEPS)
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_STATIC)
 
 build:
@@ -46,5 +50,5 @@ build:
 
 .PHONY: clean
 clean:
-	rm -f build/$(APP) build/$(APP)-static build/$(APP)-shared
+	rm -f build/$(APP) build/$(APP)-static build/$(APP)-shared build/*.h
 	test -d build && rmdir -p build || true
diff --git a/examples/vdpa/commands.list b/examples/vdpa/commands.list
new file mode 100644
index 0000000000..1eb8486c45
--- /dev/null
+++ b/examples/vdpa/commands.list
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2023 Intel Corporation
+#
+help                                    # show help
+list                                    # list all available vdpa devices
+create <STRING>socket_path <STRING>bdf  # create a new vdpa port
+stats <STRING>bdf <UINT32>qid           # show device statistics
+quit                                    # exit application
diff --git a/examples/vdpa/main.c b/examples/vdpa/main.c
index 4d3203f3a7..289db26498 100644
--- a/examples/vdpa/main.c
+++ b/examples/vdpa/main.c
@@ -16,11 +16,8 @@
 #include <rte_pci.h>
 #include <rte_string_fns.h>
 
-#include <cmdline_parse.h>
 #include <cmdline_socket.h>
-#include <cmdline_parse_string.h>
-#include <cmdline_parse_num.h>
-#include <cmdline.h>
+#include "commands.h"  /* auto-generated file from commands.list */
 #include "vdpa_blk_compact.h"
 
 #define MAX_PATH_LEN 128
@@ -301,14 +298,9 @@ signal_handler(int signum)
 	}
 }
 
-/* interactive cmds */
+/* interactive cmd functions */
 
-/* *** Help command with introduction. *** */
-struct cmd_help_result {
-	cmdline_fixed_string_t help;
-};
-
-static void cmd_help_parsed(__rte_unused void *parsed_result,
+void cmd_help_parsed(__rte_unused void *parsed_result,
 		struct cmdline *cl,
 		__rte_unused void *data)
 {
@@ -325,25 +317,7 @@ static void cmd_help_parsed(__rte_unused void *parsed_result,
 	);
 }
 
-cmdline_parse_token_string_t cmd_help_help =
-	TOKEN_STRING_INITIALIZER(struct cmd_help_result, help, "help");
-
-cmdline_parse_inst_t cmd_help = {
-	.f = cmd_help_parsed,
-	.data = NULL,
-	.help_str = "show help",
-	.tokens = {
-		(void *)&cmd_help_help,
-		NULL,
-	},
-};
-
-/* *** List all available vdpa devices *** */
-struct cmd_list_result {
-	cmdline_fixed_string_t action;
-};
-
-static void cmd_list_vdpa_devices_parsed(
+void cmd_list_parsed(
 		__rte_unused void *parsed_result,
 		struct cmdline *cl,
 		__rte_unused void *data)
@@ -376,27 +350,7 @@ static void cmd_list_vdpa_devices_parsed(
 	}
 }
 
-cmdline_parse_token_string_t cmd_action_list =
-	TOKEN_STRING_INITIALIZER(struct cmd_list_result, action, "list");
-
-cmdline_parse_inst_t cmd_list_vdpa_devices = {
-	.f = cmd_list_vdpa_devices_parsed,
-	.data = NULL,
-	.help_str = "list all available vdpa devices",
-	.tokens = {
-		(void *)&cmd_action_list,
-		NULL,
-	},
-};
-
-/* *** Create new vdpa port *** */
-struct cmd_create_result {
-	cmdline_fixed_string_t action;
-	cmdline_fixed_string_t socket_path;
-	cmdline_fixed_string_t bdf;
-};
-
-static void cmd_create_vdpa_port_parsed(void *parsed_result,
+void cmd_create_parsed(void *parsed_result,
 		struct cmdline *cl,
 		__rte_unused void *data)
 {
@@ -417,33 +371,7 @@ static void cmd_create_vdpa_port_parsed(void *parsed_result,
 		devcnt++;
 }
 
-cmdline_parse_token_string_t cmd_action_create =
-	TOKEN_STRING_INITIALIZER(struct cmd_create_result, action, "create");
-cmdline_parse_token_string_t cmd_socket_path =
-	TOKEN_STRING_INITIALIZER(struct cmd_create_result, socket_path, NULL);
-cmdline_parse_token_string_t cmd_bdf =
-	TOKEN_STRING_INITIALIZER(struct cmd_create_result, bdf, NULL);
-
-cmdline_parse_inst_t cmd_create_vdpa_port = {
-	.f = cmd_create_vdpa_port_parsed,
-	.data = NULL,
-	.help_str = "create a new vdpa port",
-	.tokens = {
-		(void *)&cmd_action_create,
-		(void *)&cmd_socket_path,
-		(void *)&cmd_bdf,
-		NULL,
-	},
-};
-
-/* *** STATS *** */
-struct cmd_stats_result {
-	cmdline_fixed_string_t stats;
-	cmdline_fixed_string_t bdf;
-	uint16_t qid;
-};
-
-static void cmd_device_stats_parsed(void *parsed_result, struct cmdline *cl,
+void cmd_stats_parsed(void *parsed_result, struct cmdline *cl,
 				    __rte_unused void *data)
 {
 	struct cmd_stats_result *res = parsed_result;
@@ -525,31 +453,7 @@ static void cmd_device_stats_parsed(void *parsed_result, struct cmdline *cl,
 	}
 }
 
-cmdline_parse_token_string_t cmd_device_stats_ =
-	TOKEN_STRING_INITIALIZER(struct cmd_stats_result, stats, "stats");
-cmdline_parse_token_string_t cmd_device_bdf =
-	TOKEN_STRING_INITIALIZER(struct cmd_stats_result, bdf, NULL);
-cmdline_parse_token_num_t cmd_queue_id =
-	TOKEN_NUM_INITIALIZER(struct cmd_stats_result, qid, RTE_UINT32);
-
-cmdline_parse_inst_t cmd_device_stats = {
-	.f = cmd_device_stats_parsed,
-	.data = NULL,
-	.help_str = "stats: show device statistics",
-	.tokens = {
-		(void *)&cmd_device_stats_,
-		(void *)&cmd_device_bdf,
-		(void *)&cmd_queue_id,
-		NULL,
-	},
-};
-
-/* *** QUIT *** */
-struct cmd_quit_result {
-	cmdline_fixed_string_t quit;
-};
-
-static void cmd_quit_parsed(__rte_unused void *parsed_result,
+void cmd_quit_parsed(__rte_unused void *parsed_result,
 		struct cmdline *cl,
 		__rte_unused void *data)
 {
@@ -557,27 +461,6 @@ static void cmd_quit_parsed(__rte_unused void *parsed_result,
 	cmdline_quit(cl);
 }
 
-cmdline_parse_token_string_t cmd_quit_quit =
-	TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit");
-
-cmdline_parse_inst_t cmd_quit = {
-	.f = cmd_quit_parsed,
-	.data = NULL,
-	.help_str = "quit: exit application",
-	.tokens = {
-		(void *)&cmd_quit_quit,
-		NULL,
-	},
-};
-cmdline_parse_ctx_t main_ctx[] = {
-	(cmdline_parse_inst_t *)&cmd_help,
-	(cmdline_parse_inst_t *)&cmd_list_vdpa_devices,
-	(cmdline_parse_inst_t *)&cmd_create_vdpa_port,
-	(cmdline_parse_inst_t *)&cmd_device_stats,
-	(cmdline_parse_inst_t *)&cmd_quit,
-	NULL,
-};
-
 int
 main(int argc, char *argv[])
 {
diff --git a/examples/vdpa/meson.build b/examples/vdpa/meson.build
index bd086050dc..a48028da4d 100644
--- a/examples/vdpa/meson.build
+++ b/examples/vdpa/meson.build
@@ -16,3 +16,10 @@ allow_experimental_apis = true
 sources = files(
         'main.c',
 )
+cmd_h = custom_target('commands_hdr',
+	output: 'commands.h',
+	input: files('commands.list'),
+	capture: true,
+	command: [cmdline_gen_cmd, '--context-name=main_ctx', '@INPUT@']
+)
+sources += cmd_h
-- 
2.39.2

[PATCH v6 8/9] buildtools/dpdk-cmdline-gen: support option strings

[Bruce Richardson]

Add support to the commandline generator for option strings, where there
are only a limited number of acceptable values to be passed as a
parameter.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 buildtools/dpdk-cmdline-gen.py    | 7 +++++++
 doc/guides/prog_guide/cmdline.rst | 7 +++++++
 2 files changed, 14 insertions(+)

diff --git a/buildtools/dpdk-cmdline-gen.py b/buildtools/dpdk-cmdline-gen.py
index 6cb7610de4..c4c0411c27 100755
--- a/buildtools/dpdk-cmdline-gen.py
+++ b/buildtools/dpdk-cmdline-gen.py
@@ -73,6 +73,13 @@ def process_command(lineno, tokens, comment):
                 f"cmdline_parse_token_ipaddr_t cmd_{name}_{t_name}_tok =\n"
                 + f"\tTOKEN_IPV4_INITIALIZER(struct cmd_{name}_result, {t_name});"
             )
+        elif t_type.startswith("(") and t_type.endswith(")"):
+            result_struct.append(f"\tcmdline_fixed_string_t {t_name};")
+            t_val = f'"{t_type[1:-1].replace(",","#")}"'
+            initializers.append(
+                f"static cmdline_parse_token_string_t cmd_{name}_{t_name}_tok =\n"
+                + f"\tTOKEN_STRING_INITIALIZER(struct cmd_{name}_result, {t_name}, {t_val});"
+            )
         else:
             raise TypeError(f"Error line {lineno + 1}: unknown token type '{t_type}'")
         token_list.append(f"cmd_{name}_{t_name}_tok")
diff --git a/doc/guides/prog_guide/cmdline.rst b/doc/guides/prog_guide/cmdline.rst
index 0b96b770e2..42e6a54bf4 100644
--- a/doc/guides/prog_guide/cmdline.rst
+++ b/doc/guides/prog_guide/cmdline.rst
@@ -70,6 +70,12 @@ The format of the list file must be:
 
   * ``<IP>src_ip``
 
+* Variable fields, which take their values from a list of options,
+  have the comma-separated option list placed in braces, rather than a the type name.
+  For example,
+
+  * ``<(rx,tx,rxtx)>mode``
+
 * The help text for a command is given in the form of a comment on the same line as the command
 
 An example list file, with a variety of (unrelated) commands, is shown below::
@@ -79,6 +85,7 @@ An example list file, with a variety of (unrelated) commands, is shown below::
    add <UINT16>x <UINT16>y  # add x and y
    echo <STRING>message     # print message to screen
    add socket <STRING>path  # add unix socket with the given path
+   set mode <(rx,tx)>rxtx   # set Rx-only or Tx-only mode
    quit                     # close the application
 
 Running the Generator Script
-- 
2.39.2

[PATCH v6 9/9] examples/ntb: auto-generate cmdline boilerplate

[Bruce Richardson]

Use the dpdk-cmdline-gen script to autogenerate all the boilerplate
structs and defines for the commandline part of the app.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 examples/ntb/Makefile      |  12 ++-
 examples/ntb/commands.list |  11 ++
 examples/ntb/meson.build   |   7 ++
 examples/ntb/ntb_fwd.c     | 200 ++++---------------------------------
 4 files changed, 47 insertions(+), 183 deletions(-)
 create mode 100644 examples/ntb/commands.list

diff --git a/examples/ntb/Makefile b/examples/ntb/Makefile
index d9b6e53090..f5b3405c47 100644
--- a/examples/ntb/Makefile
+++ b/examples/ntb/Makefile
@@ -6,6 +6,7 @@ APP = ntb_fwd
 
 # all source are stored in SRCS-y
 SRCS-y := ntb_fwd.c
+SRC-DEPS := build/commands.h
 
 PKGCONF ?= pkg-config
 
@@ -25,10 +26,13 @@ CFLAGS += -D_FILE_OFFSET_BITS=64
 LDFLAGS += -pthread
 
 PC_FILE := $(shell $(PKGCONF) --path libdpdk 2>/dev/null)
-CFLAGS += -O3 $(shell $(PKGCONF) --cflags libdpdk)
+CFLAGS += -O3 $(shell $(PKGCONF) --cflags libdpdk) -I build/
 LDFLAGS_SHARED = $(shell $(PKGCONF) --libs libdpdk)
 LDFLAGS_STATIC = $(shell $(PKGCONF) --static --libs libdpdk)
 
+build/commands.h: commands.list Makefile
+	dpdk-cmdline-gen.py -o $@ --context-name=main_ctx $<
+
 ifeq ($(MAKECMDGOALS),static)
 # check for broken pkg-config
 ifeq ($(shell echo $(LDFLAGS_STATIC) | grep 'whole-archive.*l:lib.*no-whole-archive'),)
@@ -39,10 +43,10 @@ endif
 
 CFLAGS += -DALLOW_EXPERIMENTAL_API
 
-build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
+build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build $(SRC-DEPS)
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
 
-build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build
+build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build $(SRC-DEPS)
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_STATIC)
 
 build:
@@ -50,5 +54,5 @@ build:
 
 .PHONY: clean
 clean:
-	rm -f build/$(APP) build/$(APP)-static build/$(APP)-shared
+	rm -f build/$(APP) build/$(APP)-static build/$(APP)-shared build/*.h
 	test -d build && rmdir -p build || true
diff --git a/examples/ntb/commands.list b/examples/ntb/commands.list
new file mode 100644
index 0000000000..a26b8acfa3
--- /dev/null
+++ b/examples/ntb/commands.list
@@ -0,0 +1,11 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2023 Intel Corporation
+#
+help                   # show help
+quit                   # exit application
+send <STRING>filepath  # send <file_path>
+start                  # start pkt fwd between ntb and ethdev
+stop                   # stop packet forwarding
+show port stats        # show statistics for all ports
+clear port stats       # clear all port statistics
+set fwd <(file-trans,iofwd,txonly,rxonly)>mode  # set forwarding mode as file-trans|rxonly|txonly|iofwd
diff --git a/examples/ntb/meson.build b/examples/ntb/meson.build
index 18eaffdf21..6c23081c41 100644
--- a/examples/ntb/meson.build
+++ b/examples/ntb/meson.build
@@ -17,3 +17,10 @@ cflags += ['-D_FILE_OFFSET_BITS=64']
 sources = files(
         'ntb_fwd.c',
 )
+cmd_h = custom_target('commands_hdr',
+	output: 'commands.h',
+	input: files('commands.list'),
+	capture: true,
+	command: [cmdline_gen_cmd, '--context-name=main_ctx', '@INPUT@']
+)
+sources += cmd_h
diff --git a/examples/ntb/ntb_fwd.c b/examples/ntb/ntb_fwd.c
index 585aad9d70..95a6148c82 100644
--- a/examples/ntb/ntb_fwd.c
+++ b/examples/ntb/ntb_fwd.c
@@ -21,6 +21,7 @@
 #include <rte_cycles.h>
 #include <rte_pmd_ntb.h>
 #include <rte_mbuf_pool_ops.h>
+#include "commands.h"
 
 /* Per-port statistics struct */
 struct ntb_port_statistics {
@@ -103,12 +104,7 @@ static struct rte_eth_conf eth_port_conf = {
 	},
 };
 
-/* *** Help command with introduction. *** */
-struct cmd_help_result {
-	cmdline_fixed_string_t help;
-};
-
-static void
+void
 cmd_help_parsed(__rte_unused void *parsed_result,
 		struct cmdline *cl,
 		__rte_unused void *data)
@@ -134,26 +130,7 @@ cmd_help_parsed(__rte_unused void *parsed_result,
 	);
 
 }
-
-cmdline_parse_token_string_t cmd_help_help =
-	TOKEN_STRING_INITIALIZER(struct cmd_help_result, help, "help");
-
-cmdline_parse_inst_t cmd_help = {
-	.f = cmd_help_parsed,
-	.data = NULL,
-	.help_str = "show help",
-	.tokens = {
-		(void *)&cmd_help_help,
-		NULL,
-	},
-};
-
-/* *** QUIT *** */
-struct cmd_quit_result {
-	cmdline_fixed_string_t quit;
-};
-
-static void
+void
 cmd_quit_parsed(__rte_unused void *parsed_result,
 		struct cmdline *cl,
 		__rte_unused void *data)
@@ -188,31 +165,12 @@ cmd_quit_parsed(__rte_unused void *parsed_result,
 	cmdline_quit(cl);
 }
 
-cmdline_parse_token_string_t cmd_quit_quit =
-		TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit");
-
-cmdline_parse_inst_t cmd_quit = {
-	.f = cmd_quit_parsed,
-	.data = NULL,
-	.help_str = "exit application",
-	.tokens = {
-		(void *)&cmd_quit_quit,
-		NULL,
-	},
-};
-
-/* *** SEND FILE PARAMETERS *** */
-struct cmd_sendfile_result {
-	cmdline_fixed_string_t send_string;
-	char filepath[];
-};
-
-static void
-cmd_sendfile_parsed(void *parsed_result,
+void
+cmd_send_parsed(void *parsed_result,
 		    __rte_unused struct cmdline *cl,
 		    __rte_unused void *data)
 {
-	struct cmd_sendfile_result *res = parsed_result;
+	struct cmd_send_result *res = parsed_result;
 	struct rte_rawdev_buf *pkts_send[NTB_MAX_PKT_BURST];
 	struct rte_mbuf *mbuf_send[NTB_MAX_PKT_BURST];
 	uint64_t size, count, i, j, nb_burst;
@@ -327,24 +285,6 @@ cmd_sendfile_parsed(void *parsed_result,
 	fclose(file);
 }
 
-cmdline_parse_token_string_t cmd_send_file_send =
-	TOKEN_STRING_INITIALIZER(struct cmd_sendfile_result, send_string,
-				 "send");
-cmdline_parse_token_string_t cmd_send_file_filepath =
-	TOKEN_STRING_INITIALIZER(struct cmd_sendfile_result, filepath, NULL);
-
-
-cmdline_parse_inst_t cmd_send_file = {
-	.f = cmd_sendfile_parsed,
-	.data = NULL,
-	.help_str = "send <file_path>",
-	.tokens = {
-		(void *)&cmd_send_file_send,
-		(void *)&cmd_send_file_filepath,
-		NULL,
-	},
-};
-
 #define RECV_FILE_LEN 30
 static int
 start_polling_recv_file(void *param)
@@ -788,12 +728,7 @@ start_pkt_fwd(void)
 	}
 }
 
-/* *** START FWD PARAMETERS *** */
-struct cmd_start_result {
-	cmdline_fixed_string_t start;
-};
-
-static void
+void
 cmd_start_parsed(__rte_unused void *parsed_result,
 			    __rte_unused struct cmdline *cl,
 			    __rte_unused void *data)
@@ -801,25 +736,7 @@ cmd_start_parsed(__rte_unused void *parsed_result,
 	start_pkt_fwd();
 }
 
-cmdline_parse_token_string_t cmd_start_start =
-		TOKEN_STRING_INITIALIZER(struct cmd_start_result, start, "start");
-
-cmdline_parse_inst_t cmd_start = {
-	.f = cmd_start_parsed,
-	.data = NULL,
-	.help_str = "start pkt fwd between ntb and ethdev",
-	.tokens = {
-		(void *)&cmd_start_start,
-		NULL,
-	},
-};
-
-/* *** STOP *** */
-struct cmd_stop_result {
-	cmdline_fixed_string_t stop;
-};
-
-static void
+void
 cmd_stop_parsed(__rte_unused void *parsed_result,
 		__rte_unused struct cmdline *cl,
 		__rte_unused void *data)
@@ -844,19 +761,6 @@ cmd_stop_parsed(__rte_unused void *parsed_result,
 	printf("\nDone.\n");
 }
 
-cmdline_parse_token_string_t cmd_stop_stop =
-		TOKEN_STRING_INITIALIZER(struct cmd_stop_result, stop, "stop");
-
-cmdline_parse_inst_t cmd_stop = {
-	.f = cmd_stop_parsed,
-	.data = NULL,
-	.help_str = "stop: Stop packet forwarding",
-	.tokens = {
-		(void *)&cmd_stop_stop,
-		NULL,
-	},
-};
-
 static void
 ntb_stats_clear(void)
 {
@@ -975,58 +879,28 @@ ntb_stats_display(void)
 	free(ids);
 }
 
-/* *** SHOW/CLEAR PORT STATS *** */
-struct cmd_stats_result {
-	cmdline_fixed_string_t show;
-	cmdline_fixed_string_t port;
-	cmdline_fixed_string_t stats;
-};
-
-static void
-cmd_stats_parsed(void *parsed_result,
+void
+cmd_show_port_stats_parsed(__rte_unused void *parsed_result,
 		 __rte_unused struct cmdline *cl,
 		 __rte_unused void *data)
 {
-	struct cmd_stats_result *res = parsed_result;
-	if (!strcmp(res->show, "clear"))
-		ntb_stats_clear();
-	else
-		ntb_stats_display();
+	ntb_stats_display();
 }
 
-cmdline_parse_token_string_t cmd_stats_show =
-	TOKEN_STRING_INITIALIZER(struct cmd_stats_result, show, "show#clear");
-cmdline_parse_token_string_t cmd_stats_port =
-	TOKEN_STRING_INITIALIZER(struct cmd_stats_result, port, "port");
-cmdline_parse_token_string_t cmd_stats_stats =
-	TOKEN_STRING_INITIALIZER(struct cmd_stats_result, stats, "stats");
-
-
-cmdline_parse_inst_t cmd_stats = {
-	.f = cmd_stats_parsed,
-	.data = NULL,
-	.help_str = "show|clear port stats",
-	.tokens = {
-		(void *)&cmd_stats_show,
-		(void *)&cmd_stats_port,
-		(void *)&cmd_stats_stats,
-		NULL,
-	},
-};
-
-/* *** SET FORWARDING MODE *** */
-struct cmd_set_fwd_mode_result {
-	cmdline_fixed_string_t set;
-	cmdline_fixed_string_t fwd;
-	cmdline_fixed_string_t mode;
-};
+void
+cmd_clear_port_stats_parsed(__rte_unused void *parsed_result,
+		 __rte_unused struct cmdline *cl,
+		 __rte_unused void *data)
+{
+	ntb_stats_clear();
+}
 
-static void
-cmd_set_fwd_mode_parsed(__rte_unused void *parsed_result,
+void
+cmd_set_fwd_parsed(void *parsed_result,
 			__rte_unused struct cmdline *cl,
 			__rte_unused void *data)
 {
-	struct cmd_set_fwd_mode_result *res = parsed_result;
+	struct cmd_set_fwd_result *res = parsed_result;
 	int i;
 
 	if (in_test) {
@@ -1043,38 +917,6 @@ cmd_set_fwd_mode_parsed(__rte_unused void *parsed_result,
 	printf("Invalid %s packet forwarding mode.\n", res->mode);
 }
 
-cmdline_parse_token_string_t cmd_setfwd_set =
-	TOKEN_STRING_INITIALIZER(struct cmd_set_fwd_mode_result, set, "set");
-cmdline_parse_token_string_t cmd_setfwd_fwd =
-	TOKEN_STRING_INITIALIZER(struct cmd_set_fwd_mode_result, fwd, "fwd");
-cmdline_parse_token_string_t cmd_setfwd_mode =
-	TOKEN_STRING_INITIALIZER(struct cmd_set_fwd_mode_result, mode,
-				"file-trans#iofwd#txonly#rxonly");
-
-cmdline_parse_inst_t cmd_set_fwd_mode = {
-	.f = cmd_set_fwd_mode_parsed,
-	.data = NULL,
-	.help_str = "set forwarding mode as file-trans|rxonly|txonly|iofwd",
-	.tokens = {
-		(void *)&cmd_setfwd_set,
-		(void *)&cmd_setfwd_fwd,
-		(void *)&cmd_setfwd_mode,
-		NULL,
-	},
-};
-
-/* list of instructions */
-cmdline_parse_ctx_t main_ctx[] = {
-	(cmdline_parse_inst_t *)&cmd_help,
-	(cmdline_parse_inst_t *)&cmd_send_file,
-	(cmdline_parse_inst_t *)&cmd_start,
-	(cmdline_parse_inst_t *)&cmd_stop,
-	(cmdline_parse_inst_t *)&cmd_stats,
-	(cmdline_parse_inst_t *)&cmd_set_fwd_mode,
-	(cmdline_parse_inst_t *)&cmd_quit,
-	NULL,
-};
-
 /* prompt function, called from main on MAIN lcore */
 static void
 prompt(void)
-- 
2.39.2

[PATCH v7 0/9] document and simplify use of cmdline

[Bruce Richardson]

The DPDK commandline library is widely used by apps and examples within
DPDK, but it is not documented in our programmers guide and it requires
a lot of boilerplate code definitions in order to used. We can improve
this situation by creating a simple python script to automatically
generate the boilerplate from a list of commands.

This patchset contains a new documentation chapter on cmdline library,
going through step-by-step how to add commands and create the necessary
token lists and parse contexts.

Following that initial doc patch, the set then contains a
boilerplate-generating script, as well as a set of four patches showing
its use, by converting four examples to use the script instead of
having the hard-coded boilerplate. Once the script is used, adding a new
command becomes as simple as adding the desired command to the .list
file, and then writing the required function which will be called for
that command. No other boilerplate coding is necessary.

Final two patches, from V5 onwards, add support for option-lists in values,
and then use that support to convert over a fifth sample app - ntb.

Cmdline script obviously does not cover the full range of capabilities
of the commandline lib, but does cover the most used parts. The
code-saving to each of the examples by auto-generating the boilerplate
is significant, and probably more examples with commandlines can be
converted over in future.

The "cmdline" example itself, is not converted over, as it should
probably remain as a simple example of direct library use without the
script.

v7:
* Increased use of multi-line strings in python script, following feedback.

v6:
* rework syntax of the option lists following feedback from David.
  Now use "(x,y,z)" instead of "|x|y|z|", to avoid giving impression
  of regex support

V5:
* Added copyright headers to command list files
* Add support for option lists
* Convert examples/ntb to use script

V4:
* Reworked script following feedback from Robin J.
   - formatted code with black
   - used multi-line strings
   - replaced sys.exit(1) with proper error handling
   - per-command function returns lists rather than printing directly
   - other small cleanups
* Added change to CI script so the cmdline script is in PATH
* Converted VDPA example to script, saving another 100 LOC

V3:
* Added lots of documentation
* Added support for help text for each command
* Cleaned up script a little so it passes pycodestyle and most flake8
  checks, when line-length is set to max 100.
* Removed RFC tag, as I consider this patchset stable enough for
  consideration in a release.

V2-RFC:
* Add support for IP addresses in commands
* Move to buildtools directory and make installable
* Convert 3 examples to use script, and eliminate their boilerplate


Bruce Richardson (9):
  doc/prog_guide: new chapter on cmdline library
  buildtools: script to generate cmdline boilerplate
  ci: allow use of DPDK tools when building examples
  examples/simple_mp: auto-generate cmdline boilerplate
  examples/hotplug_mp: auto-generate cmdline boilerplate
  examples/bond: auto-generate cmdline boilerplate
  examples/vdpa: auto-generate cmdline boilerplate
  buildtools/dpdk-cmdline-gen: support option strings
  examples/ntb: auto-generate cmdline boilerplate

 .ci/linux-build.sh                            |   1 +
 app/test/commands.c                           |   2 +
 buildtools/dpdk-cmdline-gen.py                | 202 ++++++++
 buildtools/meson.build                        |   7 +
 doc/guides/prog_guide/cmdline.rst             | 473 ++++++++++++++++++
 doc/guides/prog_guide/index.rst               |   1 +
 examples/bond/Makefile                        |  12 +-
 examples/bond/commands.list                   |   6 +
 examples/bond/main.c                          | 161 +-----
 examples/bond/main.h                          |  10 -
 examples/bond/meson.build                     |   8 +
 examples/multi_process/hotplug_mp/Makefile    |  12 +-
 examples/multi_process/hotplug_mp/commands.c  | 147 +-----
 examples/multi_process/hotplug_mp/commands.h  |  10 -
 .../multi_process/hotplug_mp/commands.list    |   8 +
 examples/multi_process/hotplug_mp/meson.build |   9 +
 examples/multi_process/simple_mp/Makefile     |  12 +-
 examples/multi_process/simple_mp/meson.build  |   9 +
 .../multi_process/simple_mp/mp_commands.c     | 106 +---
 .../multi_process/simple_mp/mp_commands.h     |  14 -
 .../multi_process/simple_mp/mp_commands.list  |   6 +
 examples/ntb/Makefile                         |  12 +-
 examples/ntb/commands.list                    |  11 +
 examples/ntb/meson.build                      |   7 +
 examples/ntb/ntb_fwd.c                        | 200 +-------
 examples/vdpa/Makefile                        |  12 +-
 examples/vdpa/commands.list                   |   8 +
 examples/vdpa/main.c                          | 131 +----
 examples/vdpa/meson.build                     |   7 +
 29 files changed, 871 insertions(+), 733 deletions(-)
 create mode 100755 buildtools/dpdk-cmdline-gen.py
 create mode 100644 doc/guides/prog_guide/cmdline.rst
 create mode 100644 examples/bond/commands.list
 delete mode 100644 examples/bond/main.h
 delete mode 100644 examples/multi_process/hotplug_mp/commands.h
 create mode 100644 examples/multi_process/hotplug_mp/commands.list
 delete mode 100644 examples/multi_process/simple_mp/mp_commands.h
 create mode 100644 examples/multi_process/simple_mp/mp_commands.list
 create mode 100644 examples/ntb/commands.list
 create mode 100644 examples/vdpa/commands.list

--
2.39.2

[PATCH v7 1/9] doc/prog_guide: new chapter on cmdline library

[Bruce Richardson]

The cmdline library was not documented in our programmers guide, so add
a new chapter on it. This chapter covers step-by-step how to use the
library, rather than focusing on the library internals. This complements
the existing cmdline example app document, providing more details on the
process of using the library.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 app/test/commands.c               |   2 +
 doc/guides/prog_guide/cmdline.rst | 337 ++++++++++++++++++++++++++++++
 doc/guides/prog_guide/index.rst   |   1 +
 3 files changed, 340 insertions(+)
 create mode 100644 doc/guides/prog_guide/cmdline.rst

diff --git a/app/test/commands.c b/app/test/commands.c
index 31259e5c21..497d8e9952 100644
--- a/app/test/commands.c
+++ b/app/test/commands.c
@@ -108,6 +108,7 @@ dump_struct_sizes(void)
 #undef DUMP_SIZE
 }
 
+/* Add the dump_* tests cases 8< */
 static void cmd_dump_parsed(void *parsed_result,
 			    __rte_unused struct cmdline *cl,
 			    __rte_unused void *data)
@@ -155,6 +156,7 @@ cmdline_parse_inst_t cmd_dump = {
 		NULL,
 	},
 };
+/* >8 End of add the dump_* tests cases */
 
 /****************/
 
diff --git a/doc/guides/prog_guide/cmdline.rst b/doc/guides/prog_guide/cmdline.rst
new file mode 100644
index 0000000000..40f49a30cc
--- /dev/null
+++ b/doc/guides/prog_guide/cmdline.rst
@@ -0,0 +1,337 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2023 Intel Corporation.
+
+Command-line Library
+====================
+
+Since its earliest versions, DPDK has included a command-line library -
+primarily for internal use by, for example, ``dpdk-testpmd`` and the ``dpdk-test`` binaries,
+but the library is also exported on install and can be used by any end application.
+This chapter covers the basics of the command-line library and how to use it in an application.
+
+Library Features
+----------------
+
+The DPDK command-line library supports the following features:
+
+* Tab-completion available for interactive terminal sessions
+
+* Ability to read and process commands taken from an input file, e.g. startup script
+
+* Parameterized commands able to take multiple parameters with different datatypes:
+
+   * Strings
+   * Signed/unsigned 16/32/64-bit integers
+   * IP Addresses
+   * Ethernet Addresses
+
+* Ability to multiplex multiple commands to a single callback function
+
+Adding Command-line to an Application
+-------------------------------------
+
+Adding a command-line instance to an application involves a number of coding steps.
+
+1. Define the result structure for the command, specifying the command parameters
+
+2. Provide an initializer for each field in the result
+
+3. Define the callback function for the command
+
+4. Provide a parse result structure instance for the command, linking the callback to the command
+
+5. Add the parse result structure to a command-line context
+
+6. Within your main application code, create a new command-line instance passing in the context.
+
+The next few subsections will cover each of these steps in more detail,
+working through an example to add two commands to a command-line instance.
+Those two commands will be:
+
+1. ``quit`` - as the name suggests, to close the application
+
+2. ``show port stats <n>`` - to display on-screen the statistics for a given ethernet port
+
+.. note::
+
+   For further examples of use of the command-line, see
+   :doc:`cmdline example application <../sample_app_ug/cmd_line>`
+
+Defining Command Result Structure
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The first structure to be defined is the structure which will be created on successful parse of a command.
+This structure contains one member field for each token, or word, in the command.
+The simplest case is for a one-word command, like ``quit``.
+For this, we only need to define a structure with a single string parameter to contain that word.
+
+.. code-block:: c
+
+   struct cmd_quit_result {
+              cmdline_fixed_string_t quit;
+   };
+
+For readability, the name of the struct member should match that of the token in the command.
+
+For our second command, we need a structure with four member fields in it,
+as there are four words/tokens in our command.
+The first three are strings, and the final one is a 16-bit numeric value.
+The resulting struct looks like:
+
+.. code-block:: c
+
+   struct cmd_show_port_stats_result {
+      cmdline_fixed_string_t show;
+      cmdline_fixed_string_t port;
+      cmdline_fixed_string_t stats;
+      uint16_t n;
+   };
+
+As before, we choose names to match the tokens in the command.
+Since our numeric parameter is a 16-bit value, we use ``uint16_t`` type for it.
+Any of the standard sized integer types can be used as parameters, depending on the desired result.
+
+Beyond the standard integer types,
+the library also allows variable parameters to be of a number of other types,
+as called out in the feature list above.
+
+* For variable string parameters,
+  the type should be ``cmdline_fixed_string_t`` - the same as for fixed tokens,
+  but these will be initialized differently (as described below).
+
+* For ethernet addresses use type ``struct rte_ether_addr``
+
+* For IP addresses use type ``cmdline_ipaddr_t``
+
+Providing Field Initializers
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Each field of our result structure needs an initializer.
+For fixed string tokens, like "quit", "show" and "port", the initializer will be the string itself.
+
+.. code-block:: c
+
+   static cmdline_parse_token_string_t cmd_quit_quit_tok =
+      TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit");
+
+The convention for naming used here is to include the base name of the overall result structure -
+``cmd_quit`` in this case,
+as well as the name of the field within that structure - ``quit`` in this case, followed by ``_tok``.
+(This is why there is a double ``quit`` in the name above).
+
+This naming convention is seen in our second example,
+which also demonstrates how to define a numeric initializer.
+
+
+.. code-block:: c
+
+   static cmdline_parse_token_string_t cmd_show_port_stats_show_tok =
+      TOKEN_STRING_INITIALIZER(struct cmd_show_port_stats_result, show, "show");
+   static cmdline_parse_token_string_t cmd_show_port_stats_port_tok =
+      TOKEN_STRING_INITIALIZER(struct cmd_show_port_stats_result, port, "port");
+   static cmdline_parse_token_string_t cmd_show_port_stats_stats_tok =
+      TOKEN_STRING_INITIALIZER(struct cmd_show_port_stats_result, stats, "stats");
+   static cmdline_parse_token_num_t cmd_show_port_stats_n_tok =
+      TOKEN_NUM_INITIALIZER(struct cmd_show_port_stats_result, n, RTE_UINT16);
+
+For variable string tokens, the same ``TOKEN_STRING_INITIALIZER`` macro should be used.
+However, the final parameter should be ``NULL`` rather than a hard-coded token string.
+
+For numeric parameters, the final parameter to the ``TOKEN_NUM_INITIALIZER`` macro should be the
+cmdline type matching the variable type defined in the result structure,
+e.g. RTE_UINT8, RTE_UINT32, etc.
+
+For IP addresses, the macro ``TOKEN_IPADDR_INITIALIZER`` should be used.
+
+For ethernet addresses, the macro ``TOKEN_ETHERADDR_INITIALIZER`` should be used.
+
+Defining Callback Function
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+For each command, we need to define a function to be called once the command has been recognised.
+The callback function should have type:
+
+.. code:: c
+
+   void (*f)(void *, struct cmdline *, void *)
+
+where the first parameter is a pointer to the result structure defined above,
+the second parameter is the command-line instance,
+and the final parameter is a user-defined pointer provided when we associate the callback with the command.
+Most callback functions only use the first parameter, or none at all,
+but the additional two parameters provide some extra flexibility,
+to allow the callback to work with non-global state in your application.
+
+For our two example commands, the relevant callback functions would look very similar in definition.
+However, within the function body,
+we assume that the user would need to reference the result structure to extract the port number in
+the second case.
+
+.. code:: c
+
+   void
+   cmd_quit_parsed(void *parsed_result, struct cmdline *cl, void *data)
+   {
+      quit = 1;
+   }
+   void
+   cmd_show_port_stats_parsed(void *parsed_result, struct cmdline *cl, void *data)
+   {
+      struct cmd_show_port_stats_result *res = parsed_result;
+      uint16_t port_id = res->n;
+      ...
+   }
+
+
+Associating Callback and Command
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The ``cmdline_parse_inst_t`` type defines a "parse instance",
+i.e. a sequence of tokens to be matched and then an associated function to be called.
+Also included in the instance type are a field for help text for the command,
+and any additional user-defined parameter to be passed to the callback functions referenced above.
+For example, for our simple "quit" command:
+
+.. code-block:: c
+
+   static cmdline_parse_inst_t cmd_quit = {
+       .f = cmd_quit_parsed,
+       .data = NULL,
+       .help_str = "Close the application",
+       .tokens = {
+           (void *)&cmd_quit_quit_tok,
+           NULL
+       }
+   };
+
+In this case, we firstly identify the callback function to be called,
+then set the user-defined parameter to NULL,
+provide a help message to be given, on request, to the user explaining the command,
+before finally listing out the single token to be matched for this command instance.
+
+For our second, port stats, example,
+as well as making things a little more complicated by having multiple tokens to be matched,
+we can also demonstrate passing in a parameter to the function.
+Let us suppose that our application does not always use all the ports available to it,
+but instead only uses a subset of the ports, stored in an array called ``active_ports``.
+Our stats command, therefore, should only display stats for the currently in-use ports,
+so we pass this ``active_ports`` array.
+(For simplicity of illustration, we shall assume that the array uses a terminating marker,
+e.g. -1 for the end of the port list, so we don't need to pass in a length parameter too.)
+
+.. code-block:: c
+
+   extern int16_t active_ports[];
+   ...
+   static cmdline_parse_inst_t cmd_show_port_stats = {
+       .f = cmd_show_port_stats_parsed,
+       .data = active_ports,
+       .help_str = "Show statistics for active network ports",
+       .tokens = {
+           (void *)&cmd_show_port_stats_show_tok,
+           (void *)&cmd_show_port_stats_port_tok,
+           (void *)&cmd_show_port_stats_stats_tok,
+           (void *)&cmd_show_port_stats_n_tok,
+           NULL
+       }
+   };
+
+
+Adding Command to Command-line Context
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Now that we have configured each individual command and callback,
+we need to merge these into a single array of command-line "contexts".
+This context array will be used to create the actual command-line instance in the application.
+Thankfully, each context entry is the same as each parse instance,
+so our array is defined by simply listing out the previously defined command parse instances.
+
+.. code-block:: c
+
+   static cmdline_parse_ctx_t ctx[] = {
+       &cmd_quit,
+       &cmd_show_port_stats,
+       NULL
+   };
+
+The context list must be terminated by a NULL entry.
+
+Creating a Command-line Instance
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Once we have our ``ctx`` variable defined,
+we now just need to call the API to create the new command-line instance in our application.
+The basic API is ``cmdline_new`` which will create an interactive command-line with all commands available.
+However, if additional features for interactive use - such as tab-completion -
+are desired, it is recommended that ``cmdline_new_stdin`` be used instead.
+
+A pattern that can be used in applications is to use ``cmdline_new`` for processing any startup commands,
+either from file or from the environment (as is done in the "dpdk-test" application),
+and then using ``cmdline_stdin_new`` thereafter to handle the interactive part.
+For example, to handle a startup file and then provide an interactive prompt:
+
+.. code-block:: c
+
+   struct cmdline *cl;
+   int fd = open(startup_file, O_RDONLY);
+
+   if (fd >= 0) {
+       cl = cmdline_new(ctx, "", fd, STDOUT_FILENO);
+       if (cl == NULL) {
+           /* error handling */
+       }
+       cmdline_interact(cl);
+       cmdline_quit(cl);
+       close(fd);
+   }
+
+   cl = cmdline_stdin_new(ctx, "Proxy>> ");
+   if (cl == NULL) {
+       /* error handling */
+   }
+   cmdline_interact(cl);
+   cmdline_stdin_exit(cl);
+
+
+Multiplexing Multiple Commands to a Single Function
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To reduce the amount of boiler-plate code needed when creating a command-line for an application,
+it is possible to merge a number of commands together to have them call a separate function.
+This can be done in a number of different ways:
+
+* A callback function can be used as the target for a number of different commands.
+  Which command was used for entry to the function can be determined by examining the first parameter,
+  ``parsed_result`` in our examples above.
+
+* For simple string commands, multiple options can be concatenated using the "#" character.
+  For example: ``exit#quit``, specified as a token initializer,
+  will match either on the string "exit" or the string "quit".
+
+As a concrete example,
+these two techniques are used in the DPDK unit test application ``dpdk-test``,
+where a single command ``cmdline_parse_t`` instance is used for all the "dump_<item>" test cases.
+
+.. literalinclude:: ../../../app/test/commands.c
+    :language: c
+    :start-after: Add the dump_* tests cases 8<
+    :end-before: >8 End of add the dump_* tests cases
+
+
+Examples of Command-line Use in DPDK
+------------------------------------
+
+To help the user follow the steps provided above,
+the following DPDK files can be consulted for examples of command-line use.
+
+.. note::
+
+   This is not an exhaustive list of examples of command-line use in DPDK.
+   It is simply a list of a few files that may be of use to the application developer.
+   Some of these referenced files contain more complex examples of use that others.
+
+* ``commands.c/.h`` in ``examples/cmdline``
+
+* ``mp_commands.c/.h`` in ``examples/multi_process/simple_mp``
+
+* ``commands.c`` in ``app/test``
diff --git a/doc/guides/prog_guide/index.rst b/doc/guides/prog_guide/index.rst
index e517f0e259..94964357ff 100644
--- a/doc/guides/prog_guide/index.rst
+++ b/doc/guides/prog_guide/index.rst
@@ -13,6 +13,7 @@ Programmer's Guide
     source_org
     env_abstraction_layer
     log_lib
+    cmdline
     service_cores
     trace_lib
     rcu_lib
-- 
2.39.2

[PATCH v7 2/9] buildtools: script to generate cmdline boilerplate

[Bruce Richardson]

Provide a "dpdk-cmdline-gen.py" script for application developers to
quickly generate the boilerplate code necessary for using the cmdline
library.

Example of use:
The script takes an input file with a list of commands the user wants in
the app, where the parameter variables are tagged with the type.
For example:

	$ cat commands.list
	list
	add <UINT16>x <UINT16>y
	echo <STRING>message
	add socket <STRING>path
	quit

When run through the script as "./dpdk-cmdline-gen.py commands.list",
the output will be the contents of a header file with all the
boilerplate necessary for a commandline instance with those commands.

If the flag --stubs is passed, an output header filename must also be
passed, in which case both a header file with the definitions and a C
file with function stubs in it is written to disk. The separation is so
that the header file can be rewritten at any future point to add more
commands, while the C file can be kept as-is and extended by the user
with any additional functions needed.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 buildtools/dpdk-cmdline-gen.py    | 195 ++++++++++++++++++++++++++++++
 buildtools/meson.build            |   7 ++
 doc/guides/prog_guide/cmdline.rst | 131 +++++++++++++++++++-
 3 files changed, 332 insertions(+), 1 deletion(-)
 create mode 100755 buildtools/dpdk-cmdline-gen.py

diff --git a/buildtools/dpdk-cmdline-gen.py b/buildtools/dpdk-cmdline-gen.py
new file mode 100755
index 0000000000..c208121363
--- /dev/null
+++ b/buildtools/dpdk-cmdline-gen.py
@@ -0,0 +1,195 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2023 Intel Corporation
+#
+"""
+Script to automatically generate boilerplate for using DPDK cmdline library.
+"""
+
+import argparse
+import sys
+
+PARSE_FN_PARAMS = "void *parsed_result, struct cmdline *cl, void *data"
+PARSE_FN_BODY = """
+    /* TODO: command action */
+    RTE_SET_USED(parsed_result);
+    RTE_SET_USED(cl);
+    RTE_SET_USED(data);
+"""
+NUMERIC_TYPES = [
+    "UINT8",
+    "UINT16",
+    "UINT32",
+    "UINT64",
+    "INT8",
+    "INT16",
+    "INT32",
+    "INT64",
+]
+
+
+def process_command(lineno, tokens, comment):
+    """Generate the structures and definitions for a single command."""
+    out = []
+    cfile_out = []
+
+    if tokens[0].startswith("<"):
+        raise ValueError(f"Error line {lineno + 1}: command must start with a literal string")
+
+    name_tokens = []
+    for t in tokens:
+        if t.startswith("<"):
+            break
+        name_tokens.append(t)
+    name = "_".join(name_tokens)
+
+    result_struct = []
+    initializers = []
+    token_list = []
+    for t in tokens:
+        if t.startswith("<"):
+            t_type, t_name = t[1:].split(">")
+            t_val = "NULL"
+        else:
+            t_type = "STRING"
+            t_name = t
+            t_val = f'"{t}"'
+
+        if t_type == "STRING":
+            result_struct.append(f"\tcmdline_fixed_string_t {t_name};")
+            initializers.append(
+                f"static cmdline_parse_token_string_t cmd_{name}_{t_name}_tok =\n"
+                f"\tTOKEN_STRING_INITIALIZER(struct cmd_{name}_result, {t_name}, {t_val});"
+            )
+        elif t_type in NUMERIC_TYPES:
+            result_struct.append(f"\t{t_type.lower()}_t {t_name};")
+            initializers.append(
+                f"static cmdline_parse_token_num_t cmd_{name}_{t_name}_tok =\n"
+                f"\tTOKEN_NUM_INITIALIZER(struct cmd_{name}_result, {t_name}, RTE_{t_type});"
+            )
+        elif t_type in ["IP", "IP_ADDR", "IPADDR"]:
+            result_struct.append(f"\tcmdline_ipaddr_t {t_name};")
+            initializers.append(
+                f"cmdline_parse_token_ipaddr_t cmd_{name}_{t_name}_tok =\n"
+                f"\tTOKEN_IPV4_INITIALIZER(struct cmd_{name}_result, {t_name});"
+            )
+        else:
+            raise TypeError(f"Error line {lineno + 1}: unknown token type '{t_type}'")
+        token_list.append(f"cmd_{name}_{t_name}_tok")
+
+    out.append(f'/* Auto-generated handling for command "{" ".join(tokens)}" */')
+    # output function prototype
+    func_sig = f"void\ncmd_{name}_parsed({PARSE_FN_PARAMS})"
+    out.append(f"extern {func_sig};\n")
+    # output result data structure
+    out.append(f"struct cmd_{name}_result {{\n" + "\n".join(result_struct) + "\n};\n")
+    # output the initializer tokens
+    out.append("\n".join(initializers) + "\n")
+    # output the instance structure
+    inst_elems = "\n".join([f"\t\t(void *)&{t}," for t in token_list])
+    out.append(
+        f"""\
+static cmdline_parse_inst_t cmd_{name} = {{
+\t.f = cmd_{name}_parsed,
+\t.data = NULL,
+\t.help_str = "{comment}",
+\t.tokens = {{
+{inst_elems}
+\t\tNULL,
+\t}}
+}};
+"""
+    )
+    # output function template if C file being written
+    cfile_out.append(f"{func_sig}\n{{{PARSE_FN_BODY}}}\n")
+
+    # return the instance structure name
+    return (f"cmd_{name}", out, cfile_out)
+
+
+def process_commands(infile, hfile, cfile, ctxname):
+    """Generate boilerplate output for a list of commands from infile."""
+    instances = []
+
+    hfile.write(
+        f"""\
+/* File autogenerated by {sys.argv[0]} */
+#ifndef GENERATED_COMMANDS_H
+#define GENERATED_COMMANDS_H
+#include <rte_common.h>
+#include <cmdline.h>
+#include <cmdline_parse_string.h>
+#include <cmdline_parse_num.h>
+#include <cmdline_parse_ipaddr.h>
+
+"""
+    )
+
+    for lineno, line in enumerate(infile.readlines()):
+        if line.lstrip().startswith("#"):
+            continue
+        if "#" not in line:
+            line = line + "#"  # ensure split always works, even if no help text
+        tokens, comment = line.split("#", 1)
+        cmd_inst, h_out, c_out = process_command(lineno, tokens.strip().split(), comment.strip())
+        hfile.write("\n".join(h_out))
+        if cfile:
+            cfile.write("\n".join(c_out))
+        instances.append(cmd_inst)
+
+    inst_join_str = ",\n\t&"
+    hfile.write(
+        f"""
+static __rte_used cmdline_parse_ctx_t {ctxname}[] = {{
+\t&{inst_join_str.join(instances)},
+\tNULL
+}};
+
+#endif /* GENERATED_COMMANDS_H */
+"""
+    )
+
+
+def main():
+    """Application main entry point."""
+    ap = argparse.ArgumentParser(description=__doc__)
+    ap.add_argument(
+        "--stubs",
+        action="store_true",
+        help="Produce C file with empty function stubs for each command",
+    )
+    ap.add_argument(
+        "--output-file",
+        "-o",
+        default="-",
+        help="Output header filename [default to stdout]",
+    )
+    ap.add_argument(
+        "--context-name",
+        default="ctx",
+        help="Name given to the cmdline context variable in the output header [default=ctx]",
+    )
+    ap.add_argument("infile", type=argparse.FileType("r"), help="File with list of commands")
+    args = ap.parse_args()
+
+    if not args.stubs:
+        if args.output_file == "-":
+            process_commands(args.infile, sys.stdout, None, args.context_name)
+        else:
+            with open(args.output_file, "w") as hfile:
+                process_commands(args.infile, hfile, None, args.context_name)
+    else:
+        if not args.output_file.endswith(".h"):
+            ap.error(
+                "-o/--output-file: specify an output filename ending with .h when creating stubs"
+            )
+
+        cfilename = args.output_file[:-2] + ".c"
+        with open(args.output_file, "w") as hfile:
+            with open(cfilename, "w") as cfile:
+                cfile.write(f'#include "{args.output_file}"\n\n')
+                process_commands(args.infile, hfile, cfile, args.context_name)
+
+
+if __name__ == "__main__":
+    main()
diff --git a/buildtools/meson.build b/buildtools/meson.build
index 948ac17dd2..72447b60a0 100644
--- a/buildtools/meson.build
+++ b/buildtools/meson.build
@@ -19,6 +19,13 @@ get_cpu_count_cmd = py3 + files('get-cpu-count.py')
 get_numa_count_cmd = py3 + files('get-numa-count.py')
 get_test_suites_cmd = py3 + files('get-test-suites.py')
 has_hugepages_cmd = py3 + files('has-hugepages.py')
+cmdline_gen_cmd = py3 + files('dpdk-cmdline-gen.py')
+
+# install any build tools that end-users might want also
+install_data([
+            'dpdk-cmdline-gen.py',
+        ],
+        install_dir: 'bin')
 
 # select library and object file format
 pmdinfo = py3 + files('gen-pmdinfo-cfile.py') + [meson.current_build_dir()]
diff --git a/doc/guides/prog_guide/cmdline.rst b/doc/guides/prog_guide/cmdline.rst
index 40f49a30cc..0b96b770e2 100644
--- a/doc/guides/prog_guide/cmdline.rst
+++ b/doc/guides/prog_guide/cmdline.rst
@@ -44,7 +44,136 @@ Adding a command-line instance to an application involves a number of coding ste
 
 6. Within your main application code, create a new command-line instance passing in the context.
 
-The next few subsections will cover each of these steps in more detail,
+Many of these steps can be automated using the script ``dpdk-cmdline-gen.py`` installed by DPDK,
+and found in the ``buildtools`` folder in the source tree.
+This section covers adding a command-line using this script to generate the boiler plate,
+while the following section,
+`Worked Example of Adding Command-line to an Application`_ covers the steps to do so manually.
+
+Creating a Command List File
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The ``dpdk-cmdline-gen.py`` script takes as input a list of commands to be used by the application.
+While these can be piped to it via standard input, using a list file is probably best.
+
+The format of the list file must be:
+
+* Comment lines start with '#' as first non-whitespace character
+
+* One command per line
+
+* Variable fields are prefixed by the type-name in angle-brackets, for example:
+
+  * ``<STRING>message``
+
+  * ``<UINT16>port_id``
+
+  * ``<IP>src_ip``
+
+* The help text for a command is given in the form of a comment on the same line as the command
+
+An example list file, with a variety of (unrelated) commands, is shown below::
+
+   # example list file
+   list                     # show all entries
+   add <UINT16>x <UINT16>y  # add x and y
+   echo <STRING>message     # print message to screen
+   add socket <STRING>path  # add unix socket with the given path
+   quit                     # close the application
+
+Running the Generator Script
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To generate the necessary definitions for a command-line, run ``dpdk-cmdline-gen.py`` passing the list file as parameter.
+The script will output the generated C code to standard output,
+the contents of which are in the form of a C header file.
+Optionally, an output filename may be specified via the ``-o/--output-file`` argument.
+
+The generated content includes:
+
+* The result structure definitions for each command
+
+* The token initializers for each structure field
+
+* An "extern" function prototype for the callback for each command
+
+* A parse context for each command, including the per-command comments as help string
+
+* A command-line context array definition, suitable for passing to ``cmdline_new``
+
+If so desired, the script can also output function stubs for the callback functions for each command.
+This behaviour is triggered by passing the ``--stubs`` flag to the script.
+In this case, an output file must be provided with a filename ending in ".h",
+and the callback stubs will be written to an equivalent ".c" file.
+
+.. note::
+
+   The stubs are written to a separate file,
+   to allow continuous use of the script to regenerate the command-line header,
+   without overwriting any code the user has added to the callback functions.
+   This makes it easy to incrementally add new commands to an existing application.
+
+Providing the Function Callbacks
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+As discussed above, the script output is a header file, containing structure definitions,
+but the callback functions themselves obviously have to be provided by the user.
+These callback functions must be provided as non-static functions in a C file,
+and named ``cmd_<cmdname>_parsed``.
+The function prototypes can be seen in the generated output header.
+
+The "cmdname" part of the function name is built up by combining the non-variable initial tokens in the command.
+So, given the commands in our worked example below: ``quit`` and ``show port stats <n>``,
+the callback functions would be:
+
+.. code:: c
+
+   void
+   cmd_quit_parsed(void *parsed_result, struct cmdline *cl, void *data)
+   {
+        ...
+   }
+
+   void
+   cmd_show_port_stats_parsed(void *parsed_result, struct cmdline *cl, void *data)
+   {
+        ...
+   }
+
+These functions must be provided by the developer, but, as stated above,
+stub functions may be generated by the script automatically using the ``--stubs`` parameter.
+
+The same "cmdname" stem is used in the naming of the generated structures too.
+To get at the results structure for each command above,
+the ``parsed_result`` parameter should be cast to ``struct cmd_quit_result``
+or ``struct cmd_show_port_stats_result`` respectively.
+
+Integrating with the Application
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To integrate the script output with the application,
+we must ``#include`` the generated header into our applications C file,
+and then have the command-line created via either ``cmdline_new`` or ``cmdline_stdin_new``.
+The first parameter to the function call should be the context array in the generated header file,
+``ctx`` by default. (Modifiable via script parameter).
+
+The callback functions may be in this same file, or in a separate one -
+they just need to be available to the linker at build-time.
+
+Limitations of the Script Approach
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The script approach works for most commands that a user may wish to add to an application.
+However, it does not support the full range of functions possible with the DPDK command-line library.
+For example,
+it is not possible using the script to multiplex multiple commands into a single callback function.
+To use this functionality, the user should follow the instructions in the next section
+`Worked Example of Adding Command-line to an Application`_ to manually configure a command-line instance.
+
+Worked Example of Adding Command-line to an Application
+-------------------------------------------------------
+
+The next few subsections will cover each of the steps listed in `Adding Command-line to an Application`_ in more detail,
 working through an example to add two commands to a command-line instance.
 Those two commands will be:
 
-- 
2.39.2

[PATCH v7 3/9] ci: allow use of DPDK tools when building examples

[Bruce Richardson]

To allow use of the DPDK python scripts (installed in $(prefix)/bin)
from within the makefiles of our examples, we need to export the PATH
variable with the location of our installed scripts from within our CI
scripts. This matches what is already done for other paths e.g. the
PKG_CONFIG_PATH variable.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
Reviewed-by: Aaron Conole <aconole@redhat.com>
---
 .ci/linux-build.sh | 1 +
 1 file changed, 1 insertion(+)

diff --git a/.ci/linux-build.sh b/.ci/linux-build.sh
index b09df07b55..4cdbe9b9e9 100755
--- a/.ci/linux-build.sh
+++ b/.ci/linux-build.sh
@@ -178,6 +178,7 @@ fi
 if [ "$BUILD_EXAMPLES" = "true" ]; then
     [ -d install ] || DESTDIR=$(pwd)/install meson install -C build
     export LD_LIBRARY_PATH=$(dirname $(find $(pwd)/install -name librte_eal.so)):$LD_LIBRARY_PATH
+    export PATH=$(dirname $(find $(pwd)/install -name dpdk-devbind.py)):$PATH
     export PKG_CONFIG_PATH=$(dirname $(find $(pwd)/install -name libdpdk.pc)):$PKG_CONFIG_PATH
     export PKGCONF="pkg-config --define-prefix"
     find build/examples -maxdepth 1 -type f -name "dpdk-*" |
-- 
2.39.2

[PATCH v7 4/9] examples/simple_mp: auto-generate cmdline boilerplate

[Bruce Richardson]

Use the dpdk-cmdline-gen script to autogenerate all the boilerplate
structs and defines for the commandline part of the app.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 examples/multi_process/simple_mp/Makefile     |  12 +-
 examples/multi_process/simple_mp/meson.build  |   9 ++
 .../multi_process/simple_mp/mp_commands.c     | 106 ++----------------
 .../multi_process/simple_mp/mp_commands.h     |  14 ---
 .../multi_process/simple_mp/mp_commands.list  |   3 +
 5 files changed, 30 insertions(+), 114 deletions(-)
 delete mode 100644 examples/multi_process/simple_mp/mp_commands.h
 create mode 100644 examples/multi_process/simple_mp/mp_commands.list

diff --git a/examples/multi_process/simple_mp/Makefile b/examples/multi_process/simple_mp/Makefile
index 1d0a260e64..890b6b7e62 100644
--- a/examples/multi_process/simple_mp/Makefile
+++ b/examples/multi_process/simple_mp/Makefile
@@ -6,6 +6,7 @@ APP = simple_mp
 
 # all source are stored in SRCS-y
 SRCS-y := main.c mp_commands.c
+SRC-DEPS := build/mp_commands.h
 
 PKGCONF ?= pkg-config
 
@@ -22,10 +23,13 @@ static: build/$(APP)-static
 	ln -sf $(APP)-static build/$(APP)
 
 PC_FILE := $(shell $(PKGCONF) --path libdpdk 2>/dev/null)
-CFLAGS += -O3 $(shell $(PKGCONF) --cflags libdpdk)
+CFLAGS += -O3 $(shell $(PKGCONF) --cflags libdpdk) -I build/
 LDFLAGS_SHARED = $(shell $(PKGCONF) --libs libdpdk)
 LDFLAGS_STATIC = $(shell $(PKGCONF) --static --libs libdpdk)
 
+build/mp_commands.h: mp_commands.list Makefile
+	dpdk-cmdline-gen.py -o $@ --context-name=simple_mp_ctx $<
+
 ifeq ($(MAKECMDGOALS),static)
 # check for broken pkg-config
 ifeq ($(shell echo $(LDFLAGS_STATIC) | grep 'whole-archive.*l:lib.*no-whole-archive'),)
@@ -36,10 +40,10 @@ endif
 
 CFLAGS += -DALLOW_EXPERIMENTAL_API
 
-build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
+build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build $(SRC-DEPS)
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
 
-build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build
+build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build $(SRC-DEPS)
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_STATIC)
 
 build:
@@ -47,5 +51,5 @@ build:
 
 .PHONY: clean
 clean:
-	rm -f build/$(APP) build/$(APP)-static build/$(APP)-shared
+	rm -f build/$(APP) build/$(APP)-static build/$(APP)-shared build/mp_commands.h
 	test -d build && rmdir -p build || true
diff --git a/examples/multi_process/simple_mp/meson.build b/examples/multi_process/simple_mp/meson.build
index 359af4384d..e99b7a3f6f 100644
--- a/examples/multi_process/simple_mp/meson.build
+++ b/examples/multi_process/simple_mp/meson.build
@@ -7,7 +7,16 @@
 # DPDK instance, use 'make'
 
 allow_experimental_apis = true
+
+cmd_h = custom_target('commands_hdr',
+	output: 'mp_commands.h',
+	input: files('mp_commands.list'),
+	capture: true,
+	command: [cmdline_gen_cmd, '--context-name=simple_mp_ctx', '@INPUT@']
+)
+
 sources = files(
         'mp_commands.c',
         'main.c',
 )
+sources += cmd_h
diff --git a/examples/multi_process/simple_mp/mp_commands.c b/examples/multi_process/simple_mp/mp_commands.c
index a5f91b00be..df9fa94208 100644
--- a/examples/multi_process/simple_mp/mp_commands.c
+++ b/examples/multi_process/simple_mp/mp_commands.c
@@ -1,44 +1,18 @@
 /* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2010-2014 Intel Corporation
+ * Copyright(c) 2010-2023 Intel Corporation
  */
-#include <stdint.h>
-#include <string.h>
-#include <stdlib.h>
-#include <stdarg.h>
-#include <inttypes.h>
-#include <stdio.h>
-#include <termios.h>
-#include <errno.h>
-#include <sys/queue.h>
-
-#include <rte_common.h>
-#include <rte_memory.h>
-#include <rte_eal.h>
-#include <rte_branch_prediction.h>
-#include <rte_launch.h>
-#include <rte_log.h>
-#include <rte_per_lcore.h>
-#include <rte_lcore.h>
 #include <rte_ring.h>
-#include <rte_debug.h>
 #include <rte_mempool.h>
 #include <rte_string_fns.h>
 
-#include <cmdline_rdline.h>
-#include <cmdline_parse.h>
-#include <cmdline_parse_string.h>
-#include <cmdline_socket.h>
-#include <cmdline.h>
 #include "mp_commands.h"
 
-/**********************************************************/
-
-struct cmd_send_result {
-	cmdline_fixed_string_t action;
-	cmdline_fixed_string_t message;
-};
+extern struct rte_ring *send_ring, *recv_ring;
+extern struct rte_mempool *message_pool;
+extern volatile int quit;
 
-static void cmd_send_parsed(void *parsed_result,
+void
+cmd_send_parsed(void *parsed_result,
 		__rte_unused struct cmdline *cl,
 		__rte_unused void *data)
 {
@@ -54,29 +28,8 @@ static void cmd_send_parsed(void *parsed_result,
 	}
 }
 
-cmdline_parse_token_string_t cmd_send_action =
-	TOKEN_STRING_INITIALIZER(struct cmd_send_result, action, "send");
-cmdline_parse_token_string_t cmd_send_message =
-	TOKEN_STRING_INITIALIZER(struct cmd_send_result, message, NULL);
-
-cmdline_parse_inst_t cmd_send = {
-	.f = cmd_send_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "send a string to another process",
-	.tokens = {        /* token list, NULL terminated */
-			(void *)&cmd_send_action,
-			(void *)&cmd_send_message,
-			NULL,
-	},
-};
-
-/**********************************************************/
-
-struct cmd_quit_result {
-	cmdline_fixed_string_t quit;
-};
-
-static void cmd_quit_parsed(__rte_unused void *parsed_result,
+void
+cmd_quit_parsed(__rte_unused void *parsed_result,
 			    struct cmdline *cl,
 			    __rte_unused void *data)
 {
@@ -84,26 +37,8 @@ static void cmd_quit_parsed(__rte_unused void *parsed_result,
 	cmdline_quit(cl);
 }
 
-cmdline_parse_token_string_t cmd_quit_quit =
-	TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit");
-
-cmdline_parse_inst_t cmd_quit = {
-	.f = cmd_quit_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "close the application",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_quit_quit,
-		NULL,
-	},
-};
-
-/**********************************************************/
-
-struct cmd_help_result {
-	cmdline_fixed_string_t help;
-};
-
-static void cmd_help_parsed(__rte_unused void *parsed_result,
+void
+cmd_help_parsed(__rte_unused void *parsed_result,
 			    struct cmdline *cl,
 			    __rte_unused void *data)
 {
@@ -112,24 +47,3 @@ static void cmd_help_parsed(__rte_unused void *parsed_result,
 			"send commands to the simple app. Commands supported are:\n\n"
 			"- send [string]\n" "- help\n" "- quit\n\n");
 }
-
-cmdline_parse_token_string_t cmd_help_help =
-	TOKEN_STRING_INITIALIZER(struct cmd_help_result, help, "help");
-
-cmdline_parse_inst_t cmd_help = {
-	.f = cmd_help_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "show help",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_help_help,
-		NULL,
-	},
-};
-
-/****** CONTEXT (list of instruction) */
-cmdline_parse_ctx_t simple_mp_ctx[] = {
-		(cmdline_parse_inst_t *)&cmd_send,
-		(cmdline_parse_inst_t *)&cmd_quit,
-		(cmdline_parse_inst_t *)&cmd_help,
-	NULL,
-};
diff --git a/examples/multi_process/simple_mp/mp_commands.h b/examples/multi_process/simple_mp/mp_commands.h
deleted file mode 100644
index 5d67413e7c..0000000000
--- a/examples/multi_process/simple_mp/mp_commands.h
+++ /dev/null
@@ -1,14 +0,0 @@
-/* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2010-2014 Intel Corporation
- */
-
-#ifndef _SIMPLE_MP_COMMANDS_H_
-#define _SIMPLE_MP_COMMANDS_H_
-
-extern struct rte_ring *send_ring;
-extern struct rte_mempool *message_pool;
-extern volatile int quit;
-
-extern cmdline_parse_ctx_t simple_mp_ctx[];
-
-#endif /* _SIMPLE_MP_COMMANDS_H_ */
diff --git a/examples/multi_process/simple_mp/mp_commands.list b/examples/multi_process/simple_mp/mp_commands.list
new file mode 100644
index 0000000000..c8890cb071
--- /dev/null
+++ b/examples/multi_process/simple_mp/mp_commands.list
@@ -0,0 +1,3 @@
+send <STRING>message  # send a string to another process
+help                  # show help
+quit                  # close the application
-- 
2.39.2

[PATCH v7 5/9] examples/hotplug_mp: auto-generate cmdline boilerplate

[Bruce Richardson]

Use the dpdk-cmdline-gen script to autogenerate all the boilerplate
structs and defines for the commandline part of the app.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 examples/multi_process/hotplug_mp/Makefile    |  12 +-
 examples/multi_process/hotplug_mp/commands.c  | 147 ++----------------
 examples/multi_process/hotplug_mp/commands.h  |  10 --
 .../multi_process/hotplug_mp/commands.list    |   8 +
 examples/multi_process/hotplug_mp/meson.build |   9 ++
 5 files changed, 38 insertions(+), 148 deletions(-)
 delete mode 100644 examples/multi_process/hotplug_mp/commands.h
 create mode 100644 examples/multi_process/hotplug_mp/commands.list

diff --git a/examples/multi_process/hotplug_mp/Makefile b/examples/multi_process/hotplug_mp/Makefile
index 6b20d6e49a..81ee85cd6b 100644
--- a/examples/multi_process/hotplug_mp/Makefile
+++ b/examples/multi_process/hotplug_mp/Makefile
@@ -6,6 +6,7 @@ APP = hotplug_mp
 
 # all source are stored in SRCS-y
 SRCS-y := main.c commands.c
+SRC-DEPS := build/commands.h
 
 PKGCONF ?= pkg-config
 
@@ -22,10 +23,13 @@ static: build/$(APP)-static
 	ln -sf $(APP)-static build/$(APP)
 
 PC_FILE := $(shell $(PKGCONF) --path libdpdk 2>/dev/null)
-CFLAGS += -O3 $(shell $(PKGCONF) --cflags libdpdk)
+CFLAGS += -O3 $(shell $(PKGCONF) --cflags libdpdk) -I build/
 LDFLAGS_SHARED = $(shell $(PKGCONF) --libs libdpdk)
 LDFLAGS_STATIC = $(shell $(PKGCONF) --static --libs libdpdk)
 
+build/commands.h: commands.list Makefile
+	dpdk-cmdline-gen.py -o $@ --context-name=main_ctx $<
+
 ifeq ($(MAKECMDGOALS),static)
 # check for broken pkg-config
 ifeq ($(shell echo $(LDFLAGS_STATIC) | grep 'whole-archive.*l:lib.*no-whole-archive'),)
@@ -36,10 +40,10 @@ endif
 
 CFLAGS += -DALLOW_EXPERIMENTAL_API
 
-build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
+build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build $(SRC-DEPS)
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
 
-build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build
+build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build  $(SRC-DEPS)
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_STATIC)
 
 build:
@@ -47,5 +51,5 @@ build:
 
 .PHONY: clean
 clean:
-	rm -f build/$(APP) build/$(APP)-static build/$(APP)-shared
+	rm -f build/$(APP) build/$(APP)-static build/$(APP)-shared build/commands.h
 	test -d build && rmdir -p build || true
diff --git a/examples/multi_process/hotplug_mp/commands.c b/examples/multi_process/hotplug_mp/commands.c
index 88f44e00a0..900eb9f774 100644
--- a/examples/multi_process/hotplug_mp/commands.c
+++ b/examples/multi_process/hotplug_mp/commands.c
@@ -1,24 +1,12 @@
 /* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2018 Intel Corporation.
+ * Copyright(c) 2018-2023 Intel Corporation.
  */
 
-#include <cmdline_rdline.h>
-#include <cmdline_parse.h>
-#include <cmdline_parse_ipaddr.h>
-#include <cmdline_parse_num.h>
-#include <cmdline_parse_string.h>
-#include <cmdline.h>
-
 #include <rte_bus.h>
 #include <rte_ethdev.h>
+#include "commands.h"
 
-/**********************************************************/
-
-struct cmd_help_result {
-	cmdline_fixed_string_t help;
-};
-
-static void cmd_help_parsed(__rte_unused void *parsed_result,
+void cmd_help_parsed(__rte_unused void *parsed_result,
 			    struct cmdline *cl,
 			    __rte_unused void *data)
 {
@@ -29,52 +17,16 @@ static void cmd_help_parsed(__rte_unused void *parsed_result,
 		       "- list\n\n");
 }
 
-cmdline_parse_token_string_t cmd_help_help =
-	TOKEN_STRING_INITIALIZER(struct cmd_help_result, help, "help");
-
-cmdline_parse_inst_t cmd_help = {
-	.f = cmd_help_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "show help",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_help_help,
-		NULL,
-	},
-};
-
-/**********************************************************/
-
-struct cmd_quit_result {
-	cmdline_fixed_string_t quit;
-};
-
-static void cmd_quit_parsed(__rte_unused void *parsed_result,
+void
+cmd_quit_parsed(__rte_unused void *parsed_result,
 			    struct cmdline *cl,
 			    __rte_unused void *data)
 {
 	cmdline_quit(cl);
 }
 
-cmdline_parse_token_string_t cmd_quit_quit =
-	TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit");
-
-cmdline_parse_inst_t cmd_quit = {
-	.f = cmd_quit_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "quit",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_quit_quit,
-		NULL,
-	},
-};
-
-/**********************************************************/
-
-struct cmd_list_result {
-	cmdline_fixed_string_t list;
-};
-
-static void cmd_list_parsed(__rte_unused void *parsed_result,
+void
+cmd_list_parsed(__rte_unused void *parsed_result,
 			    struct cmdline *cl,
 			    __rte_unused void *data)
 {
@@ -92,31 +44,12 @@ static void cmd_list_parsed(__rte_unused void *parsed_result,
 	}
 }
 
-cmdline_parse_token_string_t cmd_list_list =
-	TOKEN_STRING_INITIALIZER(struct cmd_list_result, list, "list");
-
-cmdline_parse_inst_t cmd_list = {
-	.f = cmd_list_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "list all devices",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_list_list,
-		NULL,
-	},
-};
-
-/**********************************************************/
-
-struct cmd_dev_attach_result {
-	cmdline_fixed_string_t attach;
-	cmdline_fixed_string_t devargs;
-};
-
-static void cmd_dev_attach_parsed(void *parsed_result,
+void
+cmd_attach_parsed(void *parsed_result,
 				  struct cmdline *cl,
 				  __rte_unused void *data)
 {
-	struct cmd_dev_attach_result *res = parsed_result;
+	struct cmd_attach_result *res = parsed_result;
 	struct rte_devargs da;
 
 	memset(&da, 0, sizeof(da));
@@ -134,35 +67,12 @@ static void cmd_dev_attach_parsed(void *parsed_result,
 	rte_devargs_reset(&da);
 }
 
-cmdline_parse_token_string_t cmd_dev_attach_attach =
-	TOKEN_STRING_INITIALIZER(struct cmd_dev_attach_result, attach,
-				 "attach");
-cmdline_parse_token_string_t cmd_dev_attach_devargs =
-	TOKEN_STRING_INITIALIZER(struct cmd_dev_attach_result, devargs, NULL);
-
-cmdline_parse_inst_t cmd_attach_device = {
-	.f = cmd_dev_attach_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "attach a device",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_dev_attach_attach,
-		(void *)&cmd_dev_attach_devargs,
-		NULL,
-	},
-};
-
-/**********************************************************/
-
-struct cmd_dev_detach_result {
-	cmdline_fixed_string_t detach;
-	cmdline_fixed_string_t devargs;
-};
-
-static void cmd_dev_detach_parsed(void *parsed_result,
+void
+cmd_detach_parsed(void *parsed_result,
 				   struct cmdline *cl,
 				   __rte_unused void *data)
 {
-	struct cmd_dev_detach_result *res = parsed_result;
+	struct cmd_detach_result *res = parsed_result;
 	struct rte_devargs da;
 
 	memset(&da, 0, sizeof(da));
@@ -181,34 +91,3 @@ static void cmd_dev_detach_parsed(void *parsed_result,
 			da.name);
 	rte_devargs_reset(&da);
 }
-
-cmdline_parse_token_string_t cmd_dev_detach_detach =
-	TOKEN_STRING_INITIALIZER(struct cmd_dev_detach_result, detach,
-				 "detach");
-
-cmdline_parse_token_string_t cmd_dev_detach_devargs =
-	TOKEN_STRING_INITIALIZER(struct cmd_dev_detach_result, devargs, NULL);
-
-cmdline_parse_inst_t cmd_detach_device = {
-	.f = cmd_dev_detach_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "detach a device",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_dev_detach_detach,
-		(void *)&cmd_dev_detach_devargs,
-		NULL,
-	},
-};
-
-/**********************************************************/
-/**********************************************************/
-/****** CONTEXT (list of instruction) */
-
-cmdline_parse_ctx_t main_ctx[] = {
-	(cmdline_parse_inst_t *)&cmd_help,
-	(cmdline_parse_inst_t *)&cmd_quit,
-	(cmdline_parse_inst_t *)&cmd_list,
-	(cmdline_parse_inst_t *)&cmd_attach_device,
-	(cmdline_parse_inst_t *)&cmd_detach_device,
-	NULL,
-};
diff --git a/examples/multi_process/hotplug_mp/commands.h b/examples/multi_process/hotplug_mp/commands.h
deleted file mode 100644
index afcf177dba..0000000000
--- a/examples/multi_process/hotplug_mp/commands.h
+++ /dev/null
@@ -1,10 +0,0 @@
-/* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2018 Intel Corporation
- */
-
-#ifndef _COMMANDS_H_
-#define _COMMANDS_H_
-
-extern cmdline_parse_ctx_t main_ctx[];
-
-#endif /* _COMMANDS_H_ */
diff --git a/examples/multi_process/hotplug_mp/commands.list b/examples/multi_process/hotplug_mp/commands.list
new file mode 100644
index 0000000000..8064df77c0
--- /dev/null
+++ b/examples/multi_process/hotplug_mp/commands.list
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2023 Intel Corporation
+#
+attach <STRING>devargs     # attach a device
+detach <STRING>devargs     # detach a device
+list                       # list all devices
+help                       # show help
+quit                       # quit
diff --git a/examples/multi_process/hotplug_mp/meson.build b/examples/multi_process/hotplug_mp/meson.build
index a1ad98ca2e..7a0e9ca47a 100644
--- a/examples/multi_process/hotplug_mp/meson.build
+++ b/examples/multi_process/hotplug_mp/meson.build
@@ -7,7 +7,16 @@
 # DPDK instance, use 'make'
 
 allow_experimental_apis = true
+
+cmd_h = custom_target('commands_hdr',
+	output: 'commands.h',
+	input: files('commands.list'),
+	capture: true,
+	command: [cmdline_gen_cmd, '--context-name=main_ctx', '@INPUT@']
+)
+
 sources = files(
         'commands.c',
         'main.c',
 )
+sources += cmd_h
-- 
2.39.2

[PATCH v7 6/9] examples/bond: auto-generate cmdline boilerplate

[Bruce Richardson]

Use the dpdk-cmdline-gen script to autogenerate all the boilerplate
structs and defines for the commandline part of the app.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>

---
Note: the original help text on some of the commands in this example
  were not useful "this command do not handle any arguments". Therefore,
  when converting over to the command script, the per-command help
  info has been updated with reference to the code rather than a literal
  transfer of the existing help text, as was done with the previous 2
  example apps.
---
 examples/bond/Makefile                        |  12 +-
 examples/bond/commands.list                   |   6 +
 examples/bond/main.c                          | 161 ++----------------
 examples/bond/main.h                          |  10 --
 examples/bond/meson.build                     |   8 +
 .../multi_process/simple_mp/mp_commands.list  |   3 +
 6 files changed, 40 insertions(+), 160 deletions(-)
 create mode 100644 examples/bond/commands.list
 delete mode 100644 examples/bond/main.h

diff --git a/examples/bond/Makefile b/examples/bond/Makefile
index ad711a5bee..d87c7a32ba 100644
--- a/examples/bond/Makefile
+++ b/examples/bond/Makefile
@@ -6,6 +6,7 @@ APP = bond_app
 
 # all source are stored in SRCS-y
 SRCS-y := main.c
+SRC-DEPS := build/commands.h
 
 PKGCONF ?= pkg-config
 
@@ -24,10 +25,13 @@ static: build/$(APP)-static
 LDFLAGS += -lrte_net_bond
 
 PC_FILE := $(shell $(PKGCONF) --path libdpdk 2>/dev/null)
-CFLAGS += -O3 $(shell $(PKGCONF) --cflags libdpdk)
+CFLAGS += -O3 $(shell $(PKGCONF) --cflags libdpdk) -I build/
 LDFLAGS_SHARED = $(shell $(PKGCONF) --libs libdpdk)
 LDFLAGS_STATIC = $(shell $(PKGCONF) --static --libs libdpdk)
 
+build/commands.h: commands.list Makefile
+	dpdk-cmdline-gen.py -o $@ --context-name=main_ctx $<
+
 ifeq ($(MAKECMDGOALS),static)
 # check for broken pkg-config
 ifeq ($(shell echo $(LDFLAGS_STATIC) | grep 'whole-archive.*l:lib.*no-whole-archive'),)
@@ -38,10 +42,10 @@ endif
 
 CFLAGS += -DALLOW_EXPERIMENTAL_API
 
-build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
+build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build $(SRC-DEPS)
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
 
-build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build
+build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build  $(SRC-DEPS)
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_STATIC)
 
 build:
@@ -49,5 +53,5 @@ build:
 
 .PHONY: clean
 clean:
-	rm -f build/$(APP) build/$(APP)-static build/$(APP)-shared
+	rm -f build/$(APP) build/$(APP)-static build/$(APP)-shared build/*.h
 	test -d build && rmdir -p build || true
diff --git a/examples/bond/commands.list b/examples/bond/commands.list
new file mode 100644
index 0000000000..a10bf75708
--- /dev/null
+++ b/examples/bond/commands.list
@@ -0,0 +1,6 @@
+send <IP>ip   # sends one ARPrequest through bonding for IP
+start         # starts listening if not started at startup
+stop          # stops listening
+show          # shows some bond info, e.g. active members
+help          # show help
+quit          # close application
diff --git a/examples/bond/main.c b/examples/bond/main.c
index 90f422ec11..8528abf675 100644
--- a/examples/bond/main.c
+++ b/examples/bond/main.c
@@ -45,16 +45,8 @@
 #include <rte_cpuflags.h>
 #include <rte_eth_bond.h>
 
-#include <cmdline_rdline.h>
-#include <cmdline_parse.h>
-#include <cmdline_parse_num.h>
-#include <cmdline_parse_string.h>
-#include <cmdline_parse_ipaddr.h>
-#include <cmdline_parse_etheraddr.h>
 #include <cmdline_socket.h>
-#include <cmdline.h>
-
-#include "main.h"
+#include "commands.h"
 
 #define RTE_LOGTYPE_DCB RTE_LOGTYPE_USER1
 
@@ -462,11 +454,7 @@ static int lcore_main(__rte_unused void *arg1)
 	return 0;
 }
 
-struct cmd_obj_send_result {
-	cmdline_fixed_string_t action;
-	cmdline_ipaddr_t ip;
-};
-static inline void get_string(struct cmd_obj_send_result *res, char *buf, uint8_t size)
+static inline void get_string(struct cmd_send_result *res, char *buf, uint8_t size)
 {
 	snprintf(buf, size, NIPQUAD_FMT,
 		((unsigned)((unsigned char *)&(res->ip.addr.ipv4))[0]),
@@ -475,12 +463,11 @@ static inline void get_string(struct cmd_obj_send_result *res, char *buf, uint8_
 		((unsigned)((unsigned char *)&(res->ip.addr.ipv4))[3])
 		);
 }
-static void cmd_obj_send_parsed(void *parsed_result,
-		__rte_unused struct cmdline *cl,
-			       __rte_unused void *data)
+void
+cmd_send_parsed(void *parsed_result, __rte_unused struct cmdline *cl, __rte_unused void *data)
 {
 
-	struct cmd_obj_send_result *res = parsed_result;
+	struct cmd_send_result *res = parsed_result;
 	char ip_str[INET6_ADDRSTRLEN];
 
 	struct rte_ether_addr bond_mac_addr;
@@ -544,29 +531,8 @@ static void cmd_obj_send_parsed(void *parsed_result,
 	cmdline_printf(cl, "\n");
 }
 
-cmdline_parse_token_string_t cmd_obj_action_send =
-	TOKEN_STRING_INITIALIZER(struct cmd_obj_send_result, action, "send");
-cmdline_parse_token_ipaddr_t cmd_obj_ip =
-	TOKEN_IPV4_INITIALIZER(struct cmd_obj_send_result, ip);
-
-cmdline_parse_inst_t cmd_obj_send = {
-	.f = cmd_obj_send_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "send client_ip",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_obj_action_send,
-		(void *)&cmd_obj_ip,
-		NULL,
-	},
-};
-
-struct cmd_start_result {
-	cmdline_fixed_string_t start;
-};
-
-static void cmd_start_parsed(__rte_unused void *parsed_result,
-			       struct cmdline *cl,
-			       __rte_unused void *data)
+void
+cmd_start_parsed(__rte_unused void *parsed_result, struct cmdline *cl, __rte_unused void *data)
 {
 	int worker_core_id = rte_lcore_id();
 
@@ -605,26 +571,8 @@ static void cmd_start_parsed(__rte_unused void *parsed_result,
 		);
 }
 
-cmdline_parse_token_string_t cmd_start_start =
-	TOKEN_STRING_INITIALIZER(struct cmd_start_result, start, "start");
-
-cmdline_parse_inst_t cmd_start = {
-	.f = cmd_start_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "starts listening if not started at startup",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_start_start,
-		NULL,
-	},
-};
-
-struct cmd_help_result {
-	cmdline_fixed_string_t help;
-};
-
-static void cmd_help_parsed(__rte_unused void *parsed_result,
-			    struct cmdline *cl,
-			    __rte_unused void *data)
+void
+cmd_help_parsed(__rte_unused void *parsed_result, struct cmdline *cl, __rte_unused void *data)
 {
 	cmdline_printf(cl,
 			"ALB - link bonding mode 6 example\n"
@@ -637,26 +585,8 @@ static void cmd_help_parsed(__rte_unused void *parsed_result,
 		       );
 }
 
-cmdline_parse_token_string_t cmd_help_help =
-	TOKEN_STRING_INITIALIZER(struct cmd_help_result, help, "help");
-
-cmdline_parse_inst_t cmd_help = {
-	.f = cmd_help_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "show help",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_help_help,
-		NULL,
-	},
-};
-
-struct cmd_stop_result {
-	cmdline_fixed_string_t stop;
-};
-
-static void cmd_stop_parsed(__rte_unused void *parsed_result,
-			    struct cmdline *cl,
-			    __rte_unused void *data)
+void
+cmd_stop_parsed(__rte_unused void *parsed_result, struct cmdline *cl, __rte_unused void *data)
 {
 	rte_spinlock_lock(&global_flag_stru_p->lock);
 	if (global_flag_stru_p->LcoreMainIsRunning == 0)	{
@@ -678,26 +608,8 @@ static void cmd_stop_parsed(__rte_unused void *parsed_result,
 	rte_spinlock_unlock(&global_flag_stru_p->lock);
 }
 
-cmdline_parse_token_string_t cmd_stop_stop =
-	TOKEN_STRING_INITIALIZER(struct cmd_stop_result, stop, "stop");
-
-cmdline_parse_inst_t cmd_stop = {
-	.f = cmd_stop_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "this command do not handle any arguments",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_stop_stop,
-		NULL,
-	},
-};
-
-struct cmd_quit_result {
-	cmdline_fixed_string_t quit;
-};
-
-static void cmd_quit_parsed(__rte_unused void *parsed_result,
-			    struct cmdline *cl,
-			    __rte_unused void *data)
+void
+cmd_quit_parsed(__rte_unused void *parsed_result, struct cmdline *cl, __rte_unused void *data)
 {
 	rte_spinlock_lock(&global_flag_stru_p->lock);
 	if (global_flag_stru_p->LcoreMainIsRunning == 0)	{
@@ -721,26 +633,8 @@ static void cmd_quit_parsed(__rte_unused void *parsed_result,
 	cmdline_quit(cl);
 }
 
-cmdline_parse_token_string_t cmd_quit_quit =
-	TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit");
-
-cmdline_parse_inst_t cmd_quit = {
-	.f = cmd_quit_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "this command do not handle any arguments",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_quit_quit,
-		NULL,
-	},
-};
-
-struct cmd_show_result {
-	cmdline_fixed_string_t show;
-};
-
-static void cmd_show_parsed(__rte_unused void *parsed_result,
-			    struct cmdline *cl,
-			    __rte_unused void *data)
+void
+cmd_show_parsed(__rte_unused void *parsed_result, struct cmdline *cl, __rte_unused void *data)
 {
 	uint16_t members[16] = {0};
 	uint8_t len = 16;
@@ -772,31 +666,6 @@ static void cmd_show_parsed(__rte_unused void *parsed_result,
 	rte_spinlock_unlock(&global_flag_stru_p->lock);
 }
 
-cmdline_parse_token_string_t cmd_show_show =
-	TOKEN_STRING_INITIALIZER(struct cmd_show_result, show, "show");
-
-cmdline_parse_inst_t cmd_show = {
-	.f = cmd_show_parsed,  /* function to call */
-	.data = NULL,      /* 2nd arg of func */
-	.help_str = "this command do not handle any arguments",
-	.tokens = {        /* token list, NULL terminated */
-		(void *)&cmd_show_show,
-		NULL,
-	},
-};
-
-/****** CONTEXT (list of instruction) */
-
-cmdline_parse_ctx_t main_ctx[] = {
-	(cmdline_parse_inst_t *)&cmd_start,
-	(cmdline_parse_inst_t *)&cmd_obj_send,
-	(cmdline_parse_inst_t *)&cmd_stop,
-	(cmdline_parse_inst_t *)&cmd_show,
-	(cmdline_parse_inst_t *)&cmd_quit,
-	(cmdline_parse_inst_t *)&cmd_help,
-	NULL,
-};
-
 /* prompt function, called from main on MAIN lcore */
 static void prompt(__rte_unused void *arg1)
 {
diff --git a/examples/bond/main.h b/examples/bond/main.h
deleted file mode 100644
index f91ed9c885..0000000000
--- a/examples/bond/main.h
+++ /dev/null
@@ -1,10 +0,0 @@
-/* SPDX-License-Identifier: BSD-3-Clause
- * Copyright(c) 2010-2015 Intel Corporation
- */
-
-#ifndef _MAIN_H_
-#define _MAIN_H_
-
-int main(int argc, char *argv[]);
-
-#endif /* ifndef _MAIN_H_ */
diff --git a/examples/bond/meson.build b/examples/bond/meson.build
index ed22b7887d..bfbec04ecd 100644
--- a/examples/bond/meson.build
+++ b/examples/bond/meson.build
@@ -11,3 +11,11 @@ allow_experimental_apis = true
 sources = files(
         'main.c',
 )
+
+cmd_h = custom_target('commands_hdr',
+	output: 'commands.h',
+	input: files('commands.list'),
+	capture: true,
+	command: [cmdline_gen_cmd, '--context-name=main_ctx', '@INPUT@']
+)
+sources += cmd_h
diff --git a/examples/multi_process/simple_mp/mp_commands.list b/examples/multi_process/simple_mp/mp_commands.list
index c8890cb071..afce5eb1c1 100644
--- a/examples/multi_process/simple_mp/mp_commands.list
+++ b/examples/multi_process/simple_mp/mp_commands.list
@@ -1,3 +1,6 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2023 Intel Corporation
+#
 send <STRING>message  # send a string to another process
 help                  # show help
 quit                  # close the application
-- 
2.39.2

[PATCH v7 7/9] examples/vdpa: auto-generate cmdline boilerplate

[Bruce Richardson]

Use the dpdk-cmdline-gen script to autogenerate all the boilerplate
structs and defines for the commandline part of the app.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 examples/vdpa/Makefile      |  12 ++--
 examples/vdpa/commands.list |   8 +++
 examples/vdpa/main.c        | 131 ++----------------------------------
 examples/vdpa/meson.build   |   7 ++
 4 files changed, 30 insertions(+), 128 deletions(-)
 create mode 100644 examples/vdpa/commands.list

diff --git a/examples/vdpa/Makefile b/examples/vdpa/Makefile
index d974db4f40..aa60a000cf 100644
--- a/examples/vdpa/Makefile
+++ b/examples/vdpa/Makefile
@@ -6,6 +6,7 @@ APP = vdpa
 
 # all source are stored in SRCS-y
 SRCS-y := main.c
+SRC-DEPS := build/commands.h
 CFLAGS += -DALLOW_EXPERIMENTAL_API
 
 PKGCONF ?= pkg-config
@@ -23,10 +24,13 @@ static: build/$(APP)-static
 	ln -sf $(APP)-static build/$(APP)
 
 PC_FILE := $(shell $(PKGCONF) --path libdpdk 2>/dev/null)
-CFLAGS += -O3 $(shell $(PKGCONF) --cflags libdpdk)
+CFLAGS += -O3 $(shell $(PKGCONF) --cflags libdpdk) -I build/
 LDFLAGS_SHARED = $(shell $(PKGCONF) --libs libdpdk)
 LDFLAGS_STATIC = $(shell $(PKGCONF) --static --libs libdpdk)
 
+build/commands.h: commands.list Makefile
+	dpdk-cmdline-gen.py -o $@ --context-name=main_ctx $<
+
 ifeq ($(MAKECMDGOALS),static)
 # check for broken pkg-config
 ifeq ($(shell echo $(LDFLAGS_STATIC) | grep 'whole-archive.*l:lib.*no-whole-archive'),)
@@ -35,10 +39,10 @@ $(error "Cannot generate statically-linked binaries with this version of pkg-con
 endif
 endif
 
-build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
+build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build $(SRC-DEPS)
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
 
-build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build
+build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build $(SRC-DEPS)
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_STATIC)
 
 build:
@@ -46,5 +50,5 @@ build:
 
 .PHONY: clean
 clean:
-	rm -f build/$(APP) build/$(APP)-static build/$(APP)-shared
+	rm -f build/$(APP) build/$(APP)-static build/$(APP)-shared build/*.h
 	test -d build && rmdir -p build || true
diff --git a/examples/vdpa/commands.list b/examples/vdpa/commands.list
new file mode 100644
index 0000000000..1eb8486c45
--- /dev/null
+++ b/examples/vdpa/commands.list
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2023 Intel Corporation
+#
+help                                    # show help
+list                                    # list all available vdpa devices
+create <STRING>socket_path <STRING>bdf  # create a new vdpa port
+stats <STRING>bdf <UINT32>qid           # show device statistics
+quit                                    # exit application
diff --git a/examples/vdpa/main.c b/examples/vdpa/main.c
index 4d3203f3a7..289db26498 100644
--- a/examples/vdpa/main.c
+++ b/examples/vdpa/main.c
@@ -16,11 +16,8 @@
 #include <rte_pci.h>
 #include <rte_string_fns.h>
 
-#include <cmdline_parse.h>
 #include <cmdline_socket.h>
-#include <cmdline_parse_string.h>
-#include <cmdline_parse_num.h>
-#include <cmdline.h>
+#include "commands.h"  /* auto-generated file from commands.list */
 #include "vdpa_blk_compact.h"
 
 #define MAX_PATH_LEN 128
@@ -301,14 +298,9 @@ signal_handler(int signum)
 	}
 }
 
-/* interactive cmds */
+/* interactive cmd functions */
 
-/* *** Help command with introduction. *** */
-struct cmd_help_result {
-	cmdline_fixed_string_t help;
-};
-
-static void cmd_help_parsed(__rte_unused void *parsed_result,
+void cmd_help_parsed(__rte_unused void *parsed_result,
 		struct cmdline *cl,
 		__rte_unused void *data)
 {
@@ -325,25 +317,7 @@ static void cmd_help_parsed(__rte_unused void *parsed_result,
 	);
 }
 
-cmdline_parse_token_string_t cmd_help_help =
-	TOKEN_STRING_INITIALIZER(struct cmd_help_result, help, "help");
-
-cmdline_parse_inst_t cmd_help = {
-	.f = cmd_help_parsed,
-	.data = NULL,
-	.help_str = "show help",
-	.tokens = {
-		(void *)&cmd_help_help,
-		NULL,
-	},
-};
-
-/* *** List all available vdpa devices *** */
-struct cmd_list_result {
-	cmdline_fixed_string_t action;
-};
-
-static void cmd_list_vdpa_devices_parsed(
+void cmd_list_parsed(
 		__rte_unused void *parsed_result,
 		struct cmdline *cl,
 		__rte_unused void *data)
@@ -376,27 +350,7 @@ static void cmd_list_vdpa_devices_parsed(
 	}
 }
 
-cmdline_parse_token_string_t cmd_action_list =
-	TOKEN_STRING_INITIALIZER(struct cmd_list_result, action, "list");
-
-cmdline_parse_inst_t cmd_list_vdpa_devices = {
-	.f = cmd_list_vdpa_devices_parsed,
-	.data = NULL,
-	.help_str = "list all available vdpa devices",
-	.tokens = {
-		(void *)&cmd_action_list,
-		NULL,
-	},
-};
-
-/* *** Create new vdpa port *** */
-struct cmd_create_result {
-	cmdline_fixed_string_t action;
-	cmdline_fixed_string_t socket_path;
-	cmdline_fixed_string_t bdf;
-};
-
-static void cmd_create_vdpa_port_parsed(void *parsed_result,
+void cmd_create_parsed(void *parsed_result,
 		struct cmdline *cl,
 		__rte_unused void *data)
 {
@@ -417,33 +371,7 @@ static void cmd_create_vdpa_port_parsed(void *parsed_result,
 		devcnt++;
 }
 
-cmdline_parse_token_string_t cmd_action_create =
-	TOKEN_STRING_INITIALIZER(struct cmd_create_result, action, "create");
-cmdline_parse_token_string_t cmd_socket_path =
-	TOKEN_STRING_INITIALIZER(struct cmd_create_result, socket_path, NULL);
-cmdline_parse_token_string_t cmd_bdf =
-	TOKEN_STRING_INITIALIZER(struct cmd_create_result, bdf, NULL);
-
-cmdline_parse_inst_t cmd_create_vdpa_port = {
-	.f = cmd_create_vdpa_port_parsed,
-	.data = NULL,
-	.help_str = "create a new vdpa port",
-	.tokens = {
-		(void *)&cmd_action_create,
-		(void *)&cmd_socket_path,
-		(void *)&cmd_bdf,
-		NULL,
-	},
-};
-
-/* *** STATS *** */
-struct cmd_stats_result {
-	cmdline_fixed_string_t stats;
-	cmdline_fixed_string_t bdf;
-	uint16_t qid;
-};
-
-static void cmd_device_stats_parsed(void *parsed_result, struct cmdline *cl,
+void cmd_stats_parsed(void *parsed_result, struct cmdline *cl,
 				    __rte_unused void *data)
 {
 	struct cmd_stats_result *res = parsed_result;
@@ -525,31 +453,7 @@ static void cmd_device_stats_parsed(void *parsed_result, struct cmdline *cl,
 	}
 }
 
-cmdline_parse_token_string_t cmd_device_stats_ =
-	TOKEN_STRING_INITIALIZER(struct cmd_stats_result, stats, "stats");
-cmdline_parse_token_string_t cmd_device_bdf =
-	TOKEN_STRING_INITIALIZER(struct cmd_stats_result, bdf, NULL);
-cmdline_parse_token_num_t cmd_queue_id =
-	TOKEN_NUM_INITIALIZER(struct cmd_stats_result, qid, RTE_UINT32);
-
-cmdline_parse_inst_t cmd_device_stats = {
-	.f = cmd_device_stats_parsed,
-	.data = NULL,
-	.help_str = "stats: show device statistics",
-	.tokens = {
-		(void *)&cmd_device_stats_,
-		(void *)&cmd_device_bdf,
-		(void *)&cmd_queue_id,
-		NULL,
-	},
-};
-
-/* *** QUIT *** */
-struct cmd_quit_result {
-	cmdline_fixed_string_t quit;
-};
-
-static void cmd_quit_parsed(__rte_unused void *parsed_result,
+void cmd_quit_parsed(__rte_unused void *parsed_result,
 		struct cmdline *cl,
 		__rte_unused void *data)
 {
@@ -557,27 +461,6 @@ static void cmd_quit_parsed(__rte_unused void *parsed_result,
 	cmdline_quit(cl);
 }
 
-cmdline_parse_token_string_t cmd_quit_quit =
-	TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit");
-
-cmdline_parse_inst_t cmd_quit = {
-	.f = cmd_quit_parsed,
-	.data = NULL,
-	.help_str = "quit: exit application",
-	.tokens = {
-		(void *)&cmd_quit_quit,
-		NULL,
-	},
-};
-cmdline_parse_ctx_t main_ctx[] = {
-	(cmdline_parse_inst_t *)&cmd_help,
-	(cmdline_parse_inst_t *)&cmd_list_vdpa_devices,
-	(cmdline_parse_inst_t *)&cmd_create_vdpa_port,
-	(cmdline_parse_inst_t *)&cmd_device_stats,
-	(cmdline_parse_inst_t *)&cmd_quit,
-	NULL,
-};
-
 int
 main(int argc, char *argv[])
 {
diff --git a/examples/vdpa/meson.build b/examples/vdpa/meson.build
index bd086050dc..a48028da4d 100644
--- a/examples/vdpa/meson.build
+++ b/examples/vdpa/meson.build
@@ -16,3 +16,10 @@ allow_experimental_apis = true
 sources = files(
         'main.c',
 )
+cmd_h = custom_target('commands_hdr',
+	output: 'commands.h',
+	input: files('commands.list'),
+	capture: true,
+	command: [cmdline_gen_cmd, '--context-name=main_ctx', '@INPUT@']
+)
+sources += cmd_h
-- 
2.39.2

[PATCH v7 8/9] buildtools/dpdk-cmdline-gen: support option strings

[Bruce Richardson]

Add support to the commandline generator for option strings, where there
are only a limited number of acceptable values to be passed as a
parameter.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 buildtools/dpdk-cmdline-gen.py    | 7 +++++++
 doc/guides/prog_guide/cmdline.rst | 7 +++++++
 2 files changed, 14 insertions(+)

diff --git a/buildtools/dpdk-cmdline-gen.py b/buildtools/dpdk-cmdline-gen.py
index c208121363..8922bb5fc3 100755
--- a/buildtools/dpdk-cmdline-gen.py
+++ b/buildtools/dpdk-cmdline-gen.py
@@ -73,6 +73,13 @@ def process_command(lineno, tokens, comment):
                 f"cmdline_parse_token_ipaddr_t cmd_{name}_{t_name}_tok =\n"
                 f"\tTOKEN_IPV4_INITIALIZER(struct cmd_{name}_result, {t_name});"
             )
+        elif t_type.startswith("(") and t_type.endswith(")"):
+            result_struct.append(f"\tcmdline_fixed_string_t {t_name};")
+            t_val = f'"{t_type[1:-1].replace(",","#")}"'
+            initializers.append(
+                f"static cmdline_parse_token_string_t cmd_{name}_{t_name}_tok =\n"
+                f"\tTOKEN_STRING_INITIALIZER(struct cmd_{name}_result, {t_name}, {t_val});"
+            )
         else:
             raise TypeError(f"Error line {lineno + 1}: unknown token type '{t_type}'")
         token_list.append(f"cmd_{name}_{t_name}_tok")
diff --git a/doc/guides/prog_guide/cmdline.rst b/doc/guides/prog_guide/cmdline.rst
index 0b96b770e2..42e6a54bf4 100644
--- a/doc/guides/prog_guide/cmdline.rst
+++ b/doc/guides/prog_guide/cmdline.rst
@@ -70,6 +70,12 @@ The format of the list file must be:
 
   * ``<IP>src_ip``
 
+* Variable fields, which take their values from a list of options,
+  have the comma-separated option list placed in braces, rather than a the type name.
+  For example,
+
+  * ``<(rx,tx,rxtx)>mode``
+
 * The help text for a command is given in the form of a comment on the same line as the command
 
 An example list file, with a variety of (unrelated) commands, is shown below::
@@ -79,6 +85,7 @@ An example list file, with a variety of (unrelated) commands, is shown below::
    add <UINT16>x <UINT16>y  # add x and y
    echo <STRING>message     # print message to screen
    add socket <STRING>path  # add unix socket with the given path
+   set mode <(rx,tx)>rxtx   # set Rx-only or Tx-only mode
    quit                     # close the application
 
 Running the Generator Script
-- 
2.39.2

[PATCH v7 9/9] examples/ntb: auto-generate cmdline boilerplate

[Bruce Richardson]

Use the dpdk-cmdline-gen script to autogenerate all the boilerplate
structs and defines for the commandline part of the app.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 examples/ntb/Makefile      |  12 ++-
 examples/ntb/commands.list |  11 ++
 examples/ntb/meson.build   |   7 ++
 examples/ntb/ntb_fwd.c     | 200 ++++---------------------------------
 4 files changed, 47 insertions(+), 183 deletions(-)
 create mode 100644 examples/ntb/commands.list

diff --git a/examples/ntb/Makefile b/examples/ntb/Makefile
index d9b6e53090..f5b3405c47 100644
--- a/examples/ntb/Makefile
+++ b/examples/ntb/Makefile
@@ -6,6 +6,7 @@ APP = ntb_fwd
 
 # all source are stored in SRCS-y
 SRCS-y := ntb_fwd.c
+SRC-DEPS := build/commands.h
 
 PKGCONF ?= pkg-config
 
@@ -25,10 +26,13 @@ CFLAGS += -D_FILE_OFFSET_BITS=64
 LDFLAGS += -pthread
 
 PC_FILE := $(shell $(PKGCONF) --path libdpdk 2>/dev/null)
-CFLAGS += -O3 $(shell $(PKGCONF) --cflags libdpdk)
+CFLAGS += -O3 $(shell $(PKGCONF) --cflags libdpdk) -I build/
 LDFLAGS_SHARED = $(shell $(PKGCONF) --libs libdpdk)
 LDFLAGS_STATIC = $(shell $(PKGCONF) --static --libs libdpdk)
 
+build/commands.h: commands.list Makefile
+	dpdk-cmdline-gen.py -o $@ --context-name=main_ctx $<
+
 ifeq ($(MAKECMDGOALS),static)
 # check for broken pkg-config
 ifeq ($(shell echo $(LDFLAGS_STATIC) | grep 'whole-archive.*l:lib.*no-whole-archive'),)
@@ -39,10 +43,10 @@ endif
 
 CFLAGS += -DALLOW_EXPERIMENTAL_API
 
-build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
+build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build $(SRC-DEPS)
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
 
-build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build
+build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build $(SRC-DEPS)
 	$(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_STATIC)
 
 build:
@@ -50,5 +54,5 @@ build:
 
 .PHONY: clean
 clean:
-	rm -f build/$(APP) build/$(APP)-static build/$(APP)-shared
+	rm -f build/$(APP) build/$(APP)-static build/$(APP)-shared build/*.h
 	test -d build && rmdir -p build || true
diff --git a/examples/ntb/commands.list b/examples/ntb/commands.list
new file mode 100644
index 0000000000..a26b8acfa3
--- /dev/null
+++ b/examples/ntb/commands.list
@@ -0,0 +1,11 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2023 Intel Corporation
+#
+help                   # show help
+quit                   # exit application
+send <STRING>filepath  # send <file_path>
+start                  # start pkt fwd between ntb and ethdev
+stop                   # stop packet forwarding
+show port stats        # show statistics for all ports
+clear port stats       # clear all port statistics
+set fwd <(file-trans,iofwd,txonly,rxonly)>mode  # set forwarding mode as file-trans|rxonly|txonly|iofwd
diff --git a/examples/ntb/meson.build b/examples/ntb/meson.build
index 18eaffdf21..6c23081c41 100644
--- a/examples/ntb/meson.build
+++ b/examples/ntb/meson.build
@@ -17,3 +17,10 @@ cflags += ['-D_FILE_OFFSET_BITS=64']
 sources = files(
         'ntb_fwd.c',
 )
+cmd_h = custom_target('commands_hdr',
+	output: 'commands.h',
+	input: files('commands.list'),
+	capture: true,
+	command: [cmdline_gen_cmd, '--context-name=main_ctx', '@INPUT@']
+)
+sources += cmd_h
diff --git a/examples/ntb/ntb_fwd.c b/examples/ntb/ntb_fwd.c
index 585aad9d70..95a6148c82 100644
--- a/examples/ntb/ntb_fwd.c
+++ b/examples/ntb/ntb_fwd.c
@@ -21,6 +21,7 @@
 #include <rte_cycles.h>
 #include <rte_pmd_ntb.h>
 #include <rte_mbuf_pool_ops.h>
+#include "commands.h"
 
 /* Per-port statistics struct */
 struct ntb_port_statistics {
@@ -103,12 +104,7 @@ static struct rte_eth_conf eth_port_conf = {
 	},
 };
 
-/* *** Help command with introduction. *** */
-struct cmd_help_result {
-	cmdline_fixed_string_t help;
-};
-
-static void
+void
 cmd_help_parsed(__rte_unused void *parsed_result,
 		struct cmdline *cl,
 		__rte_unused void *data)
@@ -134,26 +130,7 @@ cmd_help_parsed(__rte_unused void *parsed_result,
 	);
 
 }
-
-cmdline_parse_token_string_t cmd_help_help =
-	TOKEN_STRING_INITIALIZER(struct cmd_help_result, help, "help");
-
-cmdline_parse_inst_t cmd_help = {
-	.f = cmd_help_parsed,
-	.data = NULL,
-	.help_str = "show help",
-	.tokens = {
-		(void *)&cmd_help_help,
-		NULL,
-	},
-};
-
-/* *** QUIT *** */
-struct cmd_quit_result {
-	cmdline_fixed_string_t quit;
-};
-
-static void
+void
 cmd_quit_parsed(__rte_unused void *parsed_result,
 		struct cmdline *cl,
 		__rte_unused void *data)
@@ -188,31 +165,12 @@ cmd_quit_parsed(__rte_unused void *parsed_result,
 	cmdline_quit(cl);
 }
 
-cmdline_parse_token_string_t cmd_quit_quit =
-		TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit");
-
-cmdline_parse_inst_t cmd_quit = {
-	.f = cmd_quit_parsed,
-	.data = NULL,
-	.help_str = "exit application",
-	.tokens = {
-		(void *)&cmd_quit_quit,
-		NULL,
-	},
-};
-
-/* *** SEND FILE PARAMETERS *** */
-struct cmd_sendfile_result {
-	cmdline_fixed_string_t send_string;
-	char filepath[];
-};
-
-static void
-cmd_sendfile_parsed(void *parsed_result,
+void
+cmd_send_parsed(void *parsed_result,
 		    __rte_unused struct cmdline *cl,
 		    __rte_unused void *data)
 {
-	struct cmd_sendfile_result *res = parsed_result;
+	struct cmd_send_result *res = parsed_result;
 	struct rte_rawdev_buf *pkts_send[NTB_MAX_PKT_BURST];
 	struct rte_mbuf *mbuf_send[NTB_MAX_PKT_BURST];
 	uint64_t size, count, i, j, nb_burst;
@@ -327,24 +285,6 @@ cmd_sendfile_parsed(void *parsed_result,
 	fclose(file);
 }
 
-cmdline_parse_token_string_t cmd_send_file_send =
-	TOKEN_STRING_INITIALIZER(struct cmd_sendfile_result, send_string,
-				 "send");
-cmdline_parse_token_string_t cmd_send_file_filepath =
-	TOKEN_STRING_INITIALIZER(struct cmd_sendfile_result, filepath, NULL);
-
-
-cmdline_parse_inst_t cmd_send_file = {
-	.f = cmd_sendfile_parsed,
-	.data = NULL,
-	.help_str = "send <file_path>",
-	.tokens = {
-		(void *)&cmd_send_file_send,
-		(void *)&cmd_send_file_filepath,
-		NULL,
-	},
-};
-
 #define RECV_FILE_LEN 30
 static int
 start_polling_recv_file(void *param)
@@ -788,12 +728,7 @@ start_pkt_fwd(void)
 	}
 }
 
-/* *** START FWD PARAMETERS *** */
-struct cmd_start_result {
-	cmdline_fixed_string_t start;
-};
-
-static void
+void
 cmd_start_parsed(__rte_unused void *parsed_result,
 			    __rte_unused struct cmdline *cl,
 			    __rte_unused void *data)
@@ -801,25 +736,7 @@ cmd_start_parsed(__rte_unused void *parsed_result,
 	start_pkt_fwd();
 }
 
-cmdline_parse_token_string_t cmd_start_start =
-		TOKEN_STRING_INITIALIZER(struct cmd_start_result, start, "start");
-
-cmdline_parse_inst_t cmd_start = {
-	.f = cmd_start_parsed,
-	.data = NULL,
-	.help_str = "start pkt fwd between ntb and ethdev",
-	.tokens = {
-		(void *)&cmd_start_start,
-		NULL,
-	},
-};
-
-/* *** STOP *** */
-struct cmd_stop_result {
-	cmdline_fixed_string_t stop;
-};
-
-static void
+void
 cmd_stop_parsed(__rte_unused void *parsed_result,
 		__rte_unused struct cmdline *cl,
 		__rte_unused void *data)
@@ -844,19 +761,6 @@ cmd_stop_parsed(__rte_unused void *parsed_result,
 	printf("\nDone.\n");
 }
 
-cmdline_parse_token_string_t cmd_stop_stop =
-		TOKEN_STRING_INITIALIZER(struct cmd_stop_result, stop, "stop");
-
-cmdline_parse_inst_t cmd_stop = {
-	.f = cmd_stop_parsed,
-	.data = NULL,
-	.help_str = "stop: Stop packet forwarding",
-	.tokens = {
-		(void *)&cmd_stop_stop,
-		NULL,
-	},
-};
-
 static void
 ntb_stats_clear(void)
 {
@@ -975,58 +879,28 @@ ntb_stats_display(void)
 	free(ids);
 }
 
-/* *** SHOW/CLEAR PORT STATS *** */
-struct cmd_stats_result {
-	cmdline_fixed_string_t show;
-	cmdline_fixed_string_t port;
-	cmdline_fixed_string_t stats;
-};
-
-static void
-cmd_stats_parsed(void *parsed_result,
+void
+cmd_show_port_stats_parsed(__rte_unused void *parsed_result,
 		 __rte_unused struct cmdline *cl,
 		 __rte_unused void *data)
 {
-	struct cmd_stats_result *res = parsed_result;
-	if (!strcmp(res->show, "clear"))
-		ntb_stats_clear();
-	else
-		ntb_stats_display();
+	ntb_stats_display();
 }
 
-cmdline_parse_token_string_t cmd_stats_show =
-	TOKEN_STRING_INITIALIZER(struct cmd_stats_result, show, "show#clear");
-cmdline_parse_token_string_t cmd_stats_port =
-	TOKEN_STRING_INITIALIZER(struct cmd_stats_result, port, "port");
-cmdline_parse_token_string_t cmd_stats_stats =
-	TOKEN_STRING_INITIALIZER(struct cmd_stats_result, stats, "stats");
-
-
-cmdline_parse_inst_t cmd_stats = {
-	.f = cmd_stats_parsed,
-	.data = NULL,
-	.help_str = "show|clear port stats",
-	.tokens = {
-		(void *)&cmd_stats_show,
-		(void *)&cmd_stats_port,
-		(void *)&cmd_stats_stats,
-		NULL,
-	},
-};
-
-/* *** SET FORWARDING MODE *** */
-struct cmd_set_fwd_mode_result {
-	cmdline_fixed_string_t set;
-	cmdline_fixed_string_t fwd;
-	cmdline_fixed_string_t mode;
-};
+void
+cmd_clear_port_stats_parsed(__rte_unused void *parsed_result,
+		 __rte_unused struct cmdline *cl,
+		 __rte_unused void *data)
+{
+	ntb_stats_clear();
+}
 
-static void
-cmd_set_fwd_mode_parsed(__rte_unused void *parsed_result,
+void
+cmd_set_fwd_parsed(void *parsed_result,
 			__rte_unused struct cmdline *cl,
 			__rte_unused void *data)
 {
-	struct cmd_set_fwd_mode_result *res = parsed_result;
+	struct cmd_set_fwd_result *res = parsed_result;
 	int i;
 
 	if (in_test) {
@@ -1043,38 +917,6 @@ cmd_set_fwd_mode_parsed(__rte_unused void *parsed_result,
 	printf("Invalid %s packet forwarding mode.\n", res->mode);
 }
 
-cmdline_parse_token_string_t cmd_setfwd_set =
-	TOKEN_STRING_INITIALIZER(struct cmd_set_fwd_mode_result, set, "set");
-cmdline_parse_token_string_t cmd_setfwd_fwd =
-	TOKEN_STRING_INITIALIZER(struct cmd_set_fwd_mode_result, fwd, "fwd");
-cmdline_parse_token_string_t cmd_setfwd_mode =
-	TOKEN_STRING_INITIALIZER(struct cmd_set_fwd_mode_result, mode,
-				"file-trans#iofwd#txonly#rxonly");
-
-cmdline_parse_inst_t cmd_set_fwd_mode = {
-	.f = cmd_set_fwd_mode_parsed,
-	.data = NULL,
-	.help_str = "set forwarding mode as file-trans|rxonly|txonly|iofwd",
-	.tokens = {
-		(void *)&cmd_setfwd_set,
-		(void *)&cmd_setfwd_fwd,
-		(void *)&cmd_setfwd_mode,
-		NULL,
-	},
-};
-
-/* list of instructions */
-cmdline_parse_ctx_t main_ctx[] = {
-	(cmdline_parse_inst_t *)&cmd_help,
-	(cmdline_parse_inst_t *)&cmd_send_file,
-	(cmdline_parse_inst_t *)&cmd_start,
-	(cmdline_parse_inst_t *)&cmd_stop,
-	(cmdline_parse_inst_t *)&cmd_stats,
-	(cmdline_parse_inst_t *)&cmd_set_fwd_mode,
-	(cmdline_parse_inst_t *)&cmd_quit,
-	NULL,
-};
-
 /* prompt function, called from main on MAIN lcore */
 static void
 prompt(void)
-- 
2.39.2

Re: [PATCH v7 0/9] document and simplify use of cmdline

[David Marchand]

On Fri, Oct 27, 2023 at 1:01 PM Bruce Richardson
<bruce.richardson@intel.com> wrote:
>
> The DPDK commandline library is widely used by apps and examples within
> DPDK, but it is not documented in our programmers guide and it requires
> a lot of boilerplate code definitions in order to used. We can improve
> this situation by creating a simple python script to automatically
> generate the boilerplate from a list of commands.
>
> This patchset contains a new documentation chapter on cmdline library,
> going through step-by-step how to add commands and create the necessary
> token lists and parse contexts.
>
> Following that initial doc patch, the set then contains a
> boilerplate-generating script, as well as a set of four patches showing
> its use, by converting four examples to use the script instead of
> having the hard-coded boilerplate. Once the script is used, adding a new
> command becomes as simple as adding the desired command to the .list
> file, and then writing the required function which will be called for
> that command. No other boilerplate coding is necessary.
>
> Final two patches, from V5 onwards, add support for option-lists in values,
> and then use that support to convert over a fifth sample app - ntb.
>
> Cmdline script obviously does not cover the full range of capabilities
> of the commandline lib, but does cover the most used parts. The
> code-saving to each of the examples by auto-generating the boilerplate
> is significant, and probably more examples with commandlines can be
> converted over in future.
>
> The "cmdline" example itself, is not converted over, as it should
> probably remain as a simple example of direct library use without the
> script.
>
> v7:
> * Increased use of multi-line strings in python script, following feedback.
>
> v6:
> * rework syntax of the option lists following feedback from David.
>   Now use "(x,y,z)" instead of "|x|y|z|", to avoid giving impression
>   of regex support
>
> V5:
> * Added copyright headers to command list files
> * Add support for option lists
> * Convert examples/ntb to use script
>
> V4:
> * Reworked script following feedback from Robin J.
>    - formatted code with black
>    - used multi-line strings
>    - replaced sys.exit(1) with proper error handling
>    - per-command function returns lists rather than printing directly
>    - other small cleanups
> * Added change to CI script so the cmdline script is in PATH
> * Converted VDPA example to script, saving another 100 LOC
>
> V3:
> * Added lots of documentation
> * Added support for help text for each command
> * Cleaned up script a little so it passes pycodestyle and most flake8
>   checks, when line-length is set to max 100.
> * Removed RFC tag, as I consider this patchset stable enough for
>   consideration in a release.
>
> V2-RFC:
> * Add support for IP addresses in commands
> * Move to buildtools directory and make installable
> * Convert 3 examples to use script, and eliminate their boilerplate
>
>
> Bruce Richardson (9):
>   doc/prog_guide: new chapter on cmdline library
>   buildtools: script to generate cmdline boilerplate
>   ci: allow use of DPDK tools when building examples
>   examples/simple_mp: auto-generate cmdline boilerplate
>   examples/hotplug_mp: auto-generate cmdline boilerplate
>   examples/bond: auto-generate cmdline boilerplate
>   examples/vdpa: auto-generate cmdline boilerplate
>   buildtools/dpdk-cmdline-gen: support option strings
>   examples/ntb: auto-generate cmdline boilerplate
>
>  .ci/linux-build.sh                            |   1 +
>  app/test/commands.c                           |   2 +
>  buildtools/dpdk-cmdline-gen.py                | 202 ++++++++
>  buildtools/meson.build                        |   7 +
>  doc/guides/prog_guide/cmdline.rst             | 473 ++++++++++++++++++
>  doc/guides/prog_guide/index.rst               |   1 +
>  examples/bond/Makefile                        |  12 +-
>  examples/bond/commands.list                   |   6 +
>  examples/bond/main.c                          | 161 +-----
>  examples/bond/main.h                          |  10 -
>  examples/bond/meson.build                     |   8 +
>  examples/multi_process/hotplug_mp/Makefile    |  12 +-
>  examples/multi_process/hotplug_mp/commands.c  | 147 +-----
>  examples/multi_process/hotplug_mp/commands.h  |  10 -
>  .../multi_process/hotplug_mp/commands.list    |   8 +
>  examples/multi_process/hotplug_mp/meson.build |   9 +
>  examples/multi_process/simple_mp/Makefile     |  12 +-
>  examples/multi_process/simple_mp/meson.build  |   9 +
>  .../multi_process/simple_mp/mp_commands.c     | 106 +---
>  .../multi_process/simple_mp/mp_commands.h     |  14 -
>  .../multi_process/simple_mp/mp_commands.list  |   6 +
>  examples/ntb/Makefile                         |  12 +-
>  examples/ntb/commands.list                    |  11 +
>  examples/ntb/meson.build                      |   7 +
>  examples/ntb/ntb_fwd.c                        | 200 +-------
>  examples/vdpa/Makefile                        |  12 +-
>  examples/vdpa/commands.list                   |   8 +
>  examples/vdpa/main.c                          | 131 +----
>  examples/vdpa/meson.build                     |   7 +
>  29 files changed, 871 insertions(+), 733 deletions(-)
>  create mode 100755 buildtools/dpdk-cmdline-gen.py
>  create mode 100644 doc/guides/prog_guide/cmdline.rst
>  create mode 100644 examples/bond/commands.list
>  delete mode 100644 examples/bond/main.h
>  delete mode 100644 examples/multi_process/hotplug_mp/commands.h
>  create mode 100644 examples/multi_process/hotplug_mp/commands.list
>  delete mode 100644 examples/multi_process/simple_mp/mp_commands.h
>  create mode 100644 examples/multi_process/simple_mp/mp_commands.list
>  create mode 100644 examples/ntb/commands.list
>  create mode 100644 examples/vdpa/commands.list

I fixed a few issues:
- devtools/test-meson.builds.sh was missing some update so the new
script is used when building examples out of dpdk tree,
- the examples/bond patch contained a change belonging to a previous patch,
- some meson.build updates contained tabs,
- the documentation and the new script were not referenced in the
MAINTAINERS file,
- the cmdline documentation used explicitly numbered items while the
coding style for the doc recommends use of #.,

For the series,
Acked-by: David Marchand <david.marchand@redhat.com>


Series applied, thanks for this nice cleanup/improvement.

-- 
David Marchand