[PATCH nft 0/7] add and check dump files for JSON in tests/shell

[PATCH nft 0/7] add and check dump files for JSON in tests/shell

[Thomas Haller]

Like we have .nft dump files to compare the expected result, add
.json-nft files that compare the JSON output.

Thomas Haller (7):
  json: fix use after free in table_flags_json()
  json: drop messages "warning: stmt ops chain have no json callback"
  tests/shell: check and generate JSON dump files
  tests/shell: add JSON dump files
  tools: simplify error handling in "check-tree.sh" by adding
    msg_err()/msg_warn()
  tools: check more strictly for bash shebang in "check-tree.sh"
  tools: check for consistency of .json-nft dumps in "check-tree.sh"

 src/json.c                                    |   8 +-
 tests/shell/helpers/test-wrapper.sh           | 138 +++++++++++++-----
 tests/shell/run-tests.sh                      |  11 +-
 .../bitwise/dumps/0040mark_binop_0.json-nft   |   1 +
 .../bitwise/dumps/0040mark_binop_1.json-nft   |   1 +
 .../bitwise/dumps/0040mark_binop_2.json-nft   |   1 +
 .../bitwise/dumps/0040mark_binop_3.json-nft   |   1 +
 .../bitwise/dumps/0040mark_binop_4.json-nft   |   1 +
 .../bitwise/dumps/0040mark_binop_5.json-nft   |   1 +
 .../bitwise/dumps/0040mark_binop_6.json-nft   |   1 +
 .../bitwise/dumps/0040mark_binop_7.json-nft   |   1 +
 .../bitwise/dumps/0040mark_binop_8.json-nft   |   1 +
 .../bitwise/dumps/0040mark_binop_9.json-nft   |   1 +
 .../bogons/dumps/assert_failures.json-nft     |   1 +
 .../dumps/0001_cache_handling_0.json-nft      |   1 +
 .../cache/dumps/0002_interval_0.json-nft      |   1 +
 .../cache/dumps/0003_cache_update_0.json-nft  |   1 +
 .../cache/dumps/0004_cache_update_0.json-nft  |   1 +
 .../dumps/0005_cache_chain_flush.json-nft     |   1 +
 .../dumps/0006_cache_table_flush.json-nft     |   1 +
 .../dumps/0007_echo_cache_init_0.json-nft     |   1 +
 .../dumps/0008_delete_by_handle_0.json-nft    |   1 +
 ...0009_delete_by_handle_incorrect_0.json-nft |   1 +
 .../cache/dumps/0011_index_0.json-nft         |   1 +
 .../chains/dumps/0001jumps_0.json-nft         |   1 +
 .../chains/dumps/0002jumps_1.json-nft         |   1 +
 .../chains/dumps/0003jump_loop_1.json-nft     |   1 +
 .../chains/dumps/0004busy_1.json-nft          |   1 +
 .../chains/dumps/0005busy_map_1.json-nft      |   1 +
 .../chains/dumps/0006masquerade_0.json-nft    |   1 +
 .../chains/dumps/0007masquerade_1.json-nft    |   1 +
 .../dumps/0008masquerade_jump_1.json-nft      |   1 +
 .../dumps/0009masquerade_jump_1.json-nft      |   1 +
 .../dumps/0010endless_jump_loop_1.json-nft    |   1 +
 .../chains/dumps/0013rename_0.json-nft        |   1 +
 .../chains/dumps/0014rename_0.json-nft        |   1 +
 .../dumps/0015check_jump_loop_1.json-nft      |   1 +
 .../chains/dumps/0016delete_handle_0.json-nft |   1 +
 .../dumps/0017masquerade_jump_1.json-nft      |   1 +
 .../dumps/0018check_jump_loop_1.json-nft      |   1 +
 .../dumps/0019masquerade_jump_1.json-nft      |   1 +
 .../chains/dumps/0020depth_1.json-nft         |   1 +
 .../chains/dumps/0021prio_0.json-nft          |   1 +
 .../chains/dumps/0022prio_dummy_1.json-nft    |   1 +
 .../dumps/0023prio_inet_srcnat_1.json-nft     |   1 +
 .../dumps/0024prio_inet_dstnat_1.json-nft     |   1 +
 .../chains/dumps/0025prio_arp_1.json-nft      |   1 +
 .../chains/dumps/0026prio_netdev_1.json-nft   |   1 +
 .../dumps/0027prio_bridge_dstnat_1.json-nft   |   1 +
 .../dumps/0028prio_bridge_out_1.json-nft      |   1 +
 .../dumps/0029prio_bridge_srcnat_1.json-nft   |   1 +
 .../chains/dumps/0030create_0.json-nft        |   1 +
 .../dumps/0031priority_variable_0.json-nft    |   1 +
 .../dumps/0032priority_variable_0.json-nft    |   1 +
 .../dumps/0033priority_variable_1.json-nft    |   1 +
 .../dumps/0034priority_variable_1.json-nft    |   1 +
 .../dumps/0035policy_variable_0.json-nft      |   1 +
 .../dumps/0036policy_variable_0.json-nft      |   1 +
 .../dumps/0037policy_variable_1.json-nft      |   1 +
 .../dumps/0038policy_variable_1.json-nft      |   1 +
 .../dumps/0039negative_priority_0.json-nft    |   1 +
 .../dumps/0042chain_variable_0.json-nft       |   1 +
 .../chains/dumps/0043chain_ingress_0.json-nft |   1 +
 .../chains/dumps/0044chain_destroy_0.json-nft |   1 +
 .../chains/dumps/netdev_chain_0.json-nft      |   1 +
 .../dumps/netdev_chain_autoremove.json-nft    |   1 +
 .../comments/dumps/comments_0.json-nft        |   1 +
 .../flowtable/dumps/0001flowtable_0.json-nft  |   1 +
 .../dumps/0002create_flowtable_0.json-nft     |   1 +
 .../dumps/0003add_after_flush_0.json-nft      |   1 +
 .../dumps/0004delete_after_add_0.json-nft     |   1 +
 .../dumps/0005delete_in_use_1.json-nft        |   1 +
 .../flowtable/dumps/0006segfault_0.json-nft   |   1 +
 .../flowtable/dumps/0007prio_0.json-nft       |   1 +
 .../flowtable/dumps/0008prio_1.json-nft       |   1 +
 .../dumps/0009deleteafterflush_0.json-nft     |   1 +
 .../dumps/0010delete_handle_0.json-nft        |   1 +
 .../dumps/0011deleteafterflush_0.json-nft     |   1 +
 .../dumps/0012flowtable_variable_0.json-nft   |   1 +
 .../dumps/0013addafterdelete_0.json-nft       |   1 +
 .../dumps/0014addafterdelete_0.json-nft       |   1 +
 .../flowtable/dumps/0015destroy_0.json-nft    |   1 +
 .../include/dumps/0001absolute_0.json-nft     |   1 +
 .../include/dumps/0002relative_0.json-nft     |   1 +
 .../include/dumps/0003includepath_0.json-nft  |   1 +
 .../include/dumps/0004endlessloop_1.json-nft  |   1 +
 .../include/dumps/0005glob_empty_0.json-nft   |   1 +
 .../include/dumps/0006glob_single_0.json-nft  |   1 +
 .../include/dumps/0007glob_double_0.json-nft  |   1 +
 .../dumps/0008glob_nofile_wildcard_0.json-nft |   1 +
 .../include/dumps/0009glob_nofile_1.json-nft  |   1 +
 .../dumps/0010glob_broken_file_1.json-nft     |   1 +
 .../dumps/0011glob_dependency_0.json-nft      |   1 +
 .../dumps/0012glob_dependency_1.json-nft      |   1 +
 .../include/dumps/0013glob_dotfile_0.json-nft |   1 +
 ...nput_descriptors_included_files_0.json-nft |   1 +
 .../dumps/0014glob_directory_0.json-nft       |   1 +
 .../dumps/0015doubleincludepath_0.json-nft    |   1 +
 .../include/dumps/0016maxdepth_0.json-nft     |   1 +
 .../0017glob_more_than_maxdepth_1.json-nft    |   1 +
 .../dumps/0018include_error_0.json-nft        |   1 +
 .../dumps/0019include_error_0.json-nft        |   1 +
 .../dumps/0020include_chain_0.json-nft        |   1 +
 .../json/dumps/0001set_statements_0.json-nft  |   1 +
 .../json/dumps/0002table_map_0.json-nft       |   1 +
 .../dumps/0003json_schema_version_0.json-nft  |   1 +
 .../dumps/0004json_schema_version_1.json-nft  |   1 +
 .../json/dumps/0005secmark_objref_0.json-nft  |   1 +
 .../json/dumps/0006obj_comment_0.json-nft     |   1 +
 .../testcases/json/dumps/netdev.json-nft      |   1 +
 .../listing/dumps/0001ruleset_0.json-nft      |   1 +
 .../listing/dumps/0002ruleset_0.json-nft      |   1 +
 .../listing/dumps/0003table_0.json-nft        |   1 +
 .../listing/dumps/0004table_0.json-nft        |   1 +
 .../listing/dumps/0005ruleset_ip_0.json-nft   |   1 +
 .../listing/dumps/0006ruleset_ip6_0.json-nft  |   1 +
 .../listing/dumps/0007ruleset_inet_0.json-nft |   1 +
 .../listing/dumps/0008ruleset_arp_0.json-nft  |   1 +
 .../dumps/0009ruleset_bridge_0.json-nft       |   1 +
 .../listing/dumps/0010sets_0.json-nft         |   1 +
 .../listing/dumps/0011sets_0.json-nft         |   1 +
 .../listing/dumps/0012sets_0.json-nft         |   1 +
 .../listing/dumps/0014objects_0.json-nft      |   1 +
 .../listing/dumps/0015dynamic_0.json-nft      |   1 +
 .../listing/dumps/0016anonymous_0.json-nft    |   1 +
 .../listing/dumps/0017objects_0.json-nft      |   1 +
 .../listing/dumps/0018data_0.json-nft         |   1 +
 .../listing/dumps/0019set_0.json-nft          |   1 +
 .../listing/dumps/0020flowtable_0.json-nft    |   1 +
 .../dumps/0021ruleset_json_terse_0.json-nft   |   1 +
 .../listing/dumps/0022terse_0.json-nft        |   1 +
 .../0003map_add_many_elements_0.json-nft      |   1 +
 ...5interval_map_add_many_elements_0.json-nft |   1 +
 .../dumps/0006interval_map_overlap_0.json-nft |   1 +
 .../dumps/0007named_ifname_dtype_0.json-nft   |   1 +
 .../dumps/0008interval_map_delete_0.json-nft  |   1 +
 .../testcases/maps/dumps/0009vmap_0.json-nft  |   1 +
 .../testcases/maps/dumps/0012map_0.json-nft   |   1 +
 .../testcases/maps/dumps/0013map_0.json-nft   |   1 +
 .../maps/dumps/0014destroy_0.json-nft         |   1 +
 .../maps/dumps/0016map_leak_0.json-nft        |   1 +
 .../maps/dumps/0017_map_variable_0.json-nft   |   1 +
 .../dumps/0018map_leak_timeout_0.json-nft     |   1 +
 .../maps/dumps/anon_objmap_concat.json-nft    |   1 +
 .../maps/dumps/anonymous_snat_map_0.json-nft  |   1 +
 .../maps/dumps/different_map_types_1.json-nft |   1 +
 .../map_catchall_double_deactivate.json-nft   |   1 +
 .../maps/dumps/map_with_flags_0.json-nft      |   1 +
 .../maps/dumps/named_snat_map_0.json-nft      |   1 +
 .../dumps/typeof_maps_add_delete.json-nft     |   1 +
 .../maps/dumps/typeof_maps_update_0.json-nft  |   1 +
 .../netns/dumps/0001nft-f_0.json-nft          |   1 +
 .../netns/dumps/0002loosecommands_0.json-nft  |   1 +
 .../testcases/netns/dumps/0003many_0.json-nft |   1 +
 .../nft-f/dumps/0001define_slash_0.json-nft   |   1 +
 .../nft-f/dumps/0002rollback_rule_0.json-nft  |   1 +
 .../nft-f/dumps/0003rollback_jump_0.json-nft  |   1 +
 .../nft-f/dumps/0004rollback_set_0.json-nft   |   1 +
 .../nft-f/dumps/0005rollback_map_0.json-nft   |   1 +
 .../nft-f/dumps/0006action_object_0.json-nft  |   1 +
 .../0007action_object_set_segfault_1.json-nft |   1 +
 .../nft-f/dumps/0008split_tables_0.json-nft   |   1 +
 .../nft-f/dumps/0009variable_0.json-nft       |   1 +
 .../nft-f/dumps/0010variable_0.json-nft       |   1 +
 .../nft-f/dumps/0013defines_1.json-nft        |   1 +
 .../nft-f/dumps/0014defines_1.json-nft        |   1 +
 .../nft-f/dumps/0015defines_1.json-nft        |   1 +
 .../nft-f/dumps/0016redefines_1.json-nft      |   1 +
 .../dumps/0018ct_expectation_obj_0.json-nft   |   1 +
 .../nft-f/dumps/0018jump_variable_0.json-nft  |   1 +
 .../nft-f/dumps/0019jump_variable_1.json-nft  |   1 +
 .../nft-f/dumps/0020jump_variable_1.json-nft  |   1 +
 .../nft-f/dumps/0021list_ruleset_0.json-nft   |   1 +
 .../nft-f/dumps/0022variables_0.json-nft      |   1 +
 .../nft-f/dumps/0023check_1.json-nft          |   1 +
 .../nft-f/dumps/0025empty_dynset_0.json-nft   |   1 +
 .../nft-f/dumps/0026listing_0.json-nft        |   1 +
 .../nft-f/dumps/0027split_chains_0.json-nft   |   1 +
 .../dumps/0028variable_cmdline_0.json-nft     |   1 +
 .../nft-f/dumps/0029split_file_0.json-nft     |   1 +
 .../nft-f/dumps/0030variable_reuse_0.json-nft |   1 +
 .../nft-f/dumps/0031vmap_string_0.json-nft    |   1 +
 .../nft-f/dumps/0032pknock_0.json-nft         |   1 +
 .../nft-i/dumps/0001define_0.json-nft         |   1 +
 .../dumps/dependency_kill.json-nft            |   1 +
 .../optimizations/dumps/merge_nat.json-nft    |   1 +
 .../optimizations/dumps/merge_reject.json-nft |   1 +
 .../optimizations/dumps/merge_stmts.json-nft  |   1 +
 .../dumps/merge_stmts_concat.json-nft         |   1 +
 .../dumps/merge_stmts_concat_vmap.json-nft    |   1 +
 .../dumps/merge_stmts_vmap.json-nft           |   1 +
 .../dumps/merge_vmap_raw.json-nft             |   1 +
 .../optimizations/dumps/merge_vmaps.json-nft  |   1 +
 .../dumps/not_mergeable.json-nft              |   1 +
 .../optimizations/dumps/ruleset.json-nft      |   1 +
 .../dumps/single_anon_set.json-nft            |   1 +
 .../optimizations/dumps/skip_merge.json-nft   |   1 +
 .../optimizations/dumps/skip_non_eq.json-nft  |   1 +
 .../dumps/skip_unsupported.json-nft           |   1 +
 .../optimizations/dumps/variables.json-nft    |   1 +
 .../optionals/dumps/comments_0.json-nft       |   1 +
 .../optionals/dumps/comments_chain_0.json-nft |   1 +
 .../dumps/comments_handles_0.json-nft         |   1 +
 .../dumps/comments_objects_dup_0.json-nft     |   1 +
 .../optionals/dumps/comments_table_0.json-nft |   1 +
 .../dumps/delete_object_handles_0.json-nft    |   1 +
 .../optionals/dumps/handles_0.json-nft        |   1 +
 .../optionals/dumps/handles_1.json-nft        |   1 +
 .../optionals/dumps/log_prefix_0.json-nft     |   1 +
 .../dumps/update_object_handles_0.json-nft    |   1 +
 .../owner/dumps/0001-flowtable-uaf.json-nft   |   1 +
 .../testcases/parsing/dumps/describe.json-nft |   1 +
 .../parsing/dumps/large_rule_pipe.json-nft    |   1 +
 .../testcases/parsing/dumps/log.json-nft      |   1 +
 .../testcases/parsing/dumps/octal.json-nft    |   1 +
 .../dumps/0001addinsertposition_0.json-nft    |   1 +
 .../dumps/0002addinsertlocation_1.json-nft    |   1 +
 .../dumps/0003insert_0.json-nft               |   1 +
 .../dumps/0004replace_0.json-nft              |   1 +
 .../dumps/0005replace_1.json-nft              |   1 +
 .../dumps/0006replace_1.json-nft              |   1 +
 .../dumps/0007delete_0.json-nft               |   1 +
 .../dumps/0008delete_1.json-nft               |   1 +
 .../dumps/0009delete_1.json-nft               |   1 +
 .../dumps/0010replace_0.json-nft              |   1 +
 .../dumps/0011reset_0.json-nft                |   1 +
 .../dumps/0012destroy_0.json-nft              |   1 +
 .../sets/dumps/0001named_interval_0.json-nft  |   1 +
 .../0002named_interval_automerging_0.json-nft |   1 +
 ...0003named_interval_missing_flag_0.json-nft |   1 +
 .../0004named_interval_shadow_0.json-nft      |   1 +
 .../0005named_interval_shadow_0.json-nft      |   1 +
 .../sets/dumps/0006create_set_0.json-nft      |   1 +
 .../sets/dumps/0007create_element_0.json-nft  |   1 +
 .../dumps/0008comments_interval_0.json-nft    |   1 +
 .../dumps/0009comments_timeout_0.json-nft     |   1 +
 .../sets/dumps/0010comments_0.json-nft        |   1 +
 .../0012add_delete_many_elements_0.json-nft   |   1 +
 .../0013add_delete_many_elements_0.json-nft   |   1 +
 ...014malformed_set_is_not_defined_0.json-nft |   1 +
 .../sets/dumps/0015rulesetflush_0.json-nft    |   1 +
 .../sets/dumps/0016element_leak_0.json-nft    |   1 +
 .../sets/dumps/0017add_after_flush_0.json-nft |   1 +
 .../sets/dumps/0018set_check_size_1.json-nft  |   1 +
 .../sets/dumps/0019set_check_size_0.json-nft  |   1 +
 .../sets/dumps/0020comments_0.json-nft        |   1 +
 .../sets/dumps/0021nesting_0.json-nft         |   1 +
 .../dumps/0022type_selective_flush_0.json-nft |   1 +
 .../0023incomplete_add_set_command_0.json-nft |   1 +
 .../sets/dumps/0025anonymous_set_0.json-nft   |   1 +
 .../sets/dumps/0026named_limit_0.json-nft     |   1 +
 .../sets/dumps/0027ipv6_maps_ipv4_0.json-nft  |   1 +
 .../sets/dumps/0028autoselect_0.json-nft      |   1 +
 .../sets/dumps/0028delete_handle_0.json-nft   |   1 +
 .../dumps/0032restore_set_simple_0.json-nft   |   1 +
 .../dumps/0033add_set_simple_flat_0.json-nft  |   1 +
 .../sets/dumps/0034get_element_0.json-nft     |   1 +
 .../0035add_set_elements_flat_0.json-nft      |   1 +
 .../0037_set_with_inet_service_0.json-nft     |   1 +
 .../sets/dumps/0038meter_list_0.json-nft      |   1 +
 .../sets/dumps/0039delete_interval_0.json-nft |   1 +
 .../0040get_host_endian_elements_0.json-nft   |   1 +
 .../sets/dumps/0041interval_0.json-nft        |   1 +
 .../sets/dumps/0042update_set_0.json-nft      |   1 +
 .../dumps/0043concatenated_ranges_0.json-nft  |   1 +
 .../dumps/0043concatenated_ranges_1.json-nft  |   1 +
 .../dumps/0044interval_overlap_1.json-nft     |   1 +
 .../dumps/0045concat_ipv4_service.json-nft    |   1 +
 .../sets/dumps/0046netmap_0.json-nft          |   1 +
 .../sets/dumps/0048set_counters_0.json-nft    |   1 +
 .../sets/dumps/0049set_define_0.json-nft      |   1 +
 .../sets/dumps/0050set_define_1.json-nft      |   1 +
 .../dumps/0051set_interval_counter_0.json-nft |   1 +
 .../sets/dumps/0052overlap_0.json-nft         |   1 +
 .../testcases/sets/dumps/0053echo_0.json-nft  |   1 +
 .../sets/dumps/0054comments_set_0.json-nft    |   1 +
 .../sets/dumps/0055tcpflags_0.json-nft        |   1 +
 .../sets/dumps/0056dynamic_limit_0.json-nft   |   1 +
 .../dumps/0057set_create_fails_0.json-nft     |   1 +
 .../dumps/0058_setupdate_timeout_0.json-nft   |   1 +
 .../dumps/0059set_update_multistmt_0.json-nft |   1 +
 .../sets/dumps/0060set_multistmt_0.json-nft   |   1 +
 .../sets/dumps/0060set_multistmt_1.json-nft   |   1 +
 .../dumps/0061anonymous_automerge_0.json-nft  |   1 +
 .../sets/dumps/0062set_connlimit_0.json-nft   |   1 +
 .../sets/dumps/0063set_catchall_0.json-nft    |   1 +
 .../sets/dumps/0064map_catchall_0.json-nft    |   1 +
 .../dumps/0065_icmp_postprocessing.json-nft   |   1 +
 .../sets/dumps/0069interval_merge_0.json-nft  |   1 +
 .../0071unclosed_prefix_interval_0.json-nft   |   1 +
 .../sets/dumps/0072destroy_0.json-nft         |   1 +
 .../sets/dumps/0073flat_interval_set.json-nft |   1 +
 .../dumps/0074nested_interval_set.json-nft    |   1 +
 .../sets/dumps/collapse_elem_0.json-nft       |   1 +
 .../sets/dumps/concat_interval_0.json-nft     |   1 +
 .../sets/dumps/dynset_missing.json-nft        |   1 +
 .../testcases/sets/dumps/errors_0.json-nft    |   1 +
 .../sets/dumps/exact_overlap_0.json-nft       |   1 +
 .../testcases/sets/dumps/inner_0.json-nft     |   1 +
 .../testcases/sets/dumps/set_eval_0.json-nft  |   1 +
 .../sets/dumps/type_set_symbol.json-nft       |   1 +
 .../transactions/dumps/0001table_0.json-nft   |   1 +
 .../transactions/dumps/0002table_0.json-nft   |   1 +
 .../transactions/dumps/0003table_0.json-nft   |   1 +
 .../transactions/dumps/0010chain_0.json-nft   |   1 +
 .../transactions/dumps/0011chain_0.json-nft   |   1 +
 .../transactions/dumps/0012chain_0.json-nft   |   1 +
 .../transactions/dumps/0013chain_0.json-nft   |   1 +
 .../transactions/dumps/0014chain_1.json-nft   |   1 +
 .../transactions/dumps/0015chain_0.json-nft   |   1 +
 .../transactions/dumps/0020rule_0.json-nft    |   1 +
 .../transactions/dumps/0021rule_0.json-nft    |   1 +
 .../transactions/dumps/0022rule_1.json-nft    |   1 +
 .../transactions/dumps/0023rule_1.json-nft    |   1 +
 .../transactions/dumps/0024rule_0.json-nft    |   1 +
 .../transactions/dumps/0025rule_0.json-nft    |   1 +
 .../transactions/dumps/0030set_0.json-nft     |   1 +
 .../transactions/dumps/0031set_0.json-nft     |   1 +
 .../transactions/dumps/0032set_0.json-nft     |   1 +
 .../transactions/dumps/0033set_0.json-nft     |   1 +
 .../transactions/dumps/0034set_0.json-nft     |   1 +
 .../transactions/dumps/0035set_0.json-nft     |   1 +
 .../transactions/dumps/0036set_1.json-nft     |   1 +
 .../transactions/dumps/0037set_0.json-nft     |   1 +
 .../transactions/dumps/0038set_0.json-nft     |   1 +
 .../transactions/dumps/0039set_0.json-nft     |   1 +
 .../transactions/dumps/0040set_0.json-nft     |   1 +
 .../dumps/0041nat_restore_0.json-nft          |   1 +
 .../dumps/0042_stateful_expr_0.json-nft       |   1 +
 .../transactions/dumps/0043set_1.json-nft     |   1 +
 .../transactions/dumps/0044rule_0.json-nft    |   1 +
 .../dumps/0045anon-unbind_0.json-nft          |   1 +
 .../transactions/dumps/0046set_0.json-nft     |   1 +
 .../transactions/dumps/0047set_0.json-nft     |   1 +
 .../transactions/dumps/0048helpers_0.json-nft |   1 +
 .../transactions/dumps/0049huge_0.json-nft    |   1 +
 .../transactions/dumps/0050rule_1.json-nft    |   1 +
 .../transactions/dumps/30s-stress.json-nft    |   1 +
 .../dumps/anon_chain_loop.json-nft            |   1 +
 .../dumps/bad_expression.json-nft             |   1 +
 .../transactions/dumps/table_onoff.json-nft   |   1 +
 tools/check-tree.sh                           |  63 +++++---
 342 files changed, 494 insertions(+), 64 deletions(-)
 create mode 100644 tests/shell/testcases/bitwise/dumps/0040mark_binop_0.json-nft
 create mode 100644 tests/shell/testcases/bitwise/dumps/0040mark_binop_1.json-nft
 create mode 100644 tests/shell/testcases/bitwise/dumps/0040mark_binop_2.json-nft
 create mode 100644 tests/shell/testcases/bitwise/dumps/0040mark_binop_3.json-nft
 create mode 100644 tests/shell/testcases/bitwise/dumps/0040mark_binop_4.json-nft
 create mode 100644 tests/shell/testcases/bitwise/dumps/0040mark_binop_5.json-nft
 create mode 100644 tests/shell/testcases/bitwise/dumps/0040mark_binop_6.json-nft
 create mode 100644 tests/shell/testcases/bitwise/dumps/0040mark_binop_7.json-nft
 create mode 100644 tests/shell/testcases/bitwise/dumps/0040mark_binop_8.json-nft
 create mode 100644 tests/shell/testcases/bitwise/dumps/0040mark_binop_9.json-nft
 create mode 100644 tests/shell/testcases/bogons/dumps/assert_failures.json-nft
 create mode 100644 tests/shell/testcases/cache/dumps/0001_cache_handling_0.json-nft
 create mode 100644 tests/shell/testcases/cache/dumps/0002_interval_0.json-nft
 create mode 100644 tests/shell/testcases/cache/dumps/0003_cache_update_0.json-nft
 create mode 100644 tests/shell/testcases/cache/dumps/0004_cache_update_0.json-nft
 create mode 100644 tests/shell/testcases/cache/dumps/0005_cache_chain_flush.json-nft
 create mode 100644 tests/shell/testcases/cache/dumps/0006_cache_table_flush.json-nft
 create mode 100644 tests/shell/testcases/cache/dumps/0007_echo_cache_init_0.json-nft
 create mode 100644 tests/shell/testcases/cache/dumps/0008_delete_by_handle_0.json-nft
 create mode 100644 tests/shell/testcases/cache/dumps/0009_delete_by_handle_incorrect_0.json-nft
 create mode 100644 tests/shell/testcases/cache/dumps/0011_index_0.json-nft
 create mode 100644 tests/shell/testcases/chains/dumps/0001jumps_0.json-nft
 create mode 100644 tests/shell/testcases/chains/dumps/0002jumps_1.json-nft
 create mode 100644 tests/shell/testcases/chains/dumps/0003jump_loop_1.json-nft
 create mode 100644 tests/shell/testcases/chains/dumps/0004busy_1.json-nft
 create mode 100644 tests/shell/testcases/chains/dumps/0005busy_map_1.json-nft
 create mode 100644 tests/shell/testcases/chains/dumps/0006masquerade_0.json-nft
 create mode 100644 tests/shell/testcases/chains/dumps/0007masquerade_1.json-nft
 create mode 100644 tests/shell/testcases/chains/dumps/0008masquerade_jump_1.json-nft
 create mode 100644 tests/shell/testcases/chains/dumps/0009masquerade_jump_1.json-nft
 create mode 100644 tests/shell/testcases/chains/dumps/0010endless_jump_loop_1.json-nft
 create mode 100644 tests/shell/testcases/chains/dumps/0013rename_0.json-nft
 create mode 100644 tests/shell/testcases/chains/dumps/0014rename_0.json-nft
 create mode 100644 tests/shell/testcases/chains/dumps/0015check_jump_loop_1.json-nft
 create mode 100644 tests/shell/testcases/chains/dumps/0016delete_handle_0.json-nft
 create mode 100644 tests/shell/testcases/chains/dumps/0017masquerade_jump_1.json-nft
 create mode 100644 tests/shell/testcases/chains/dumps/0018check_jump_loop_1.json-nft
 create mode 100644 tests/shell/testcases/chains/dumps/0019masquerade_jump_1.json-nft
 create mode 100644 tests/shell/testcases/chains/dumps/0020depth_1.json-nft
 create mode 100644 tests/shell/testcases/chains/dumps/0021prio_0.json-nft
 create mode 100644 tests/shell/testcases/chains/dumps/0022prio_dummy_1.json-nft
 create mode 100644 tests/shell/testcases/chains/dumps/0023prio_inet_srcnat_1.json-nft
 create mode 100644 tests/shell/testcases/chains/dumps/0024prio_inet_dstnat_1.json-nft
 create mode 100644 tests/shell/testcases/chains/dumps/0025prio_arp_1.json-nft
 create mode 100644 tests/shell/testcases/chains/dumps/0026prio_netdev_1.json-nft
 create mode 100644 tests/shell/testcases/chains/dumps/0027prio_bridge_dstnat_1.json-nft
 create mode 100644 tests/shell/testcases/chains/dumps/0028prio_bridge_out_1.json-nft
 create mode 100644 tests/shell/testcases/chains/dumps/0029prio_bridge_srcnat_1.json-nft
 create mode 100644 tests/shell/testcases/chains/dumps/0030create_0.json-nft
 create mode 100644 tests/shell/testcases/chains/dumps/0031priority_variable_0.json-nft
 create mode 100644 tests/shell/testcases/chains/dumps/0032priority_variable_0.json-nft
 create mode 100644 tests/shell/testcases/chains/dumps/0033priority_variable_1.json-nft
 create mode 100644 tests/shell/testcases/chains/dumps/0034priority_variable_1.json-nft
 create mode 100644 tests/shell/testcases/chains/dumps/0035policy_variable_0.json-nft
 create mode 100644 tests/shell/testcases/chains/dumps/0036policy_variable_0.json-nft
 create mode 100644 tests/shell/testcases/chains/dumps/0037policy_variable_1.json-nft
 create mode 100644 tests/shell/testcases/chains/dumps/0038policy_variable_1.json-nft
 create mode 100644 tests/shell/testcases/chains/dumps/0039negative_priority_0.json-nft
 create mode 100644 tests/shell/testcases/chains/dumps/0042chain_variable_0.json-nft
 create mode 100644 tests/shell/testcases/chains/dumps/0043chain_ingress_0.json-nft
 create mode 100644 tests/shell/testcases/chains/dumps/0044chain_destroy_0.json-nft
 create mode 100644 tests/shell/testcases/chains/dumps/netdev_chain_0.json-nft
 create mode 100644 tests/shell/testcases/chains/dumps/netdev_chain_autoremove.json-nft
 create mode 100644 tests/shell/testcases/comments/dumps/comments_0.json-nft
 create mode 100644 tests/shell/testcases/flowtable/dumps/0001flowtable_0.json-nft
 create mode 100644 tests/shell/testcases/flowtable/dumps/0002create_flowtable_0.json-nft
 create mode 100644 tests/shell/testcases/flowtable/dumps/0003add_after_flush_0.json-nft
 create mode 100644 tests/shell/testcases/flowtable/dumps/0004delete_after_add_0.json-nft
 create mode 100644 tests/shell/testcases/flowtable/dumps/0005delete_in_use_1.json-nft
 create mode 100644 tests/shell/testcases/flowtable/dumps/0006segfault_0.json-nft
 create mode 100644 tests/shell/testcases/flowtable/dumps/0007prio_0.json-nft
 create mode 100644 tests/shell/testcases/flowtable/dumps/0008prio_1.json-nft
 create mode 100644 tests/shell/testcases/flowtable/dumps/0009deleteafterflush_0.json-nft
 create mode 100644 tests/shell/testcases/flowtable/dumps/0010delete_handle_0.json-nft
 create mode 100644 tests/shell/testcases/flowtable/dumps/0011deleteafterflush_0.json-nft
 create mode 100644 tests/shell/testcases/flowtable/dumps/0012flowtable_variable_0.json-nft
 create mode 100644 tests/shell/testcases/flowtable/dumps/0013addafterdelete_0.json-nft
 create mode 100644 tests/shell/testcases/flowtable/dumps/0014addafterdelete_0.json-nft
 create mode 100644 tests/shell/testcases/flowtable/dumps/0015destroy_0.json-nft
 create mode 100644 tests/shell/testcases/include/dumps/0001absolute_0.json-nft
 create mode 100644 tests/shell/testcases/include/dumps/0002relative_0.json-nft
 create mode 100644 tests/shell/testcases/include/dumps/0003includepath_0.json-nft
 create mode 100644 tests/shell/testcases/include/dumps/0004endlessloop_1.json-nft
 create mode 100644 tests/shell/testcases/include/dumps/0005glob_empty_0.json-nft
 create mode 100644 tests/shell/testcases/include/dumps/0006glob_single_0.json-nft
 create mode 100644 tests/shell/testcases/include/dumps/0007glob_double_0.json-nft
 create mode 100644 tests/shell/testcases/include/dumps/0008glob_nofile_wildcard_0.json-nft
 create mode 100644 tests/shell/testcases/include/dumps/0009glob_nofile_1.json-nft
 create mode 100644 tests/shell/testcases/include/dumps/0010glob_broken_file_1.json-nft
 create mode 100644 tests/shell/testcases/include/dumps/0011glob_dependency_0.json-nft
 create mode 100644 tests/shell/testcases/include/dumps/0012glob_dependency_1.json-nft
 create mode 100644 tests/shell/testcases/include/dumps/0013glob_dotfile_0.json-nft
 create mode 100644 tests/shell/testcases/include/dumps/0013input_descriptors_included_files_0.json-nft
 create mode 100644 tests/shell/testcases/include/dumps/0014glob_directory_0.json-nft
 create mode 100644 tests/shell/testcases/include/dumps/0015doubleincludepath_0.json-nft
 create mode 100644 tests/shell/testcases/include/dumps/0016maxdepth_0.json-nft
 create mode 100644 tests/shell/testcases/include/dumps/0017glob_more_than_maxdepth_1.json-nft
 create mode 100644 tests/shell/testcases/include/dumps/0018include_error_0.json-nft
 create mode 100644 tests/shell/testcases/include/dumps/0019include_error_0.json-nft
 create mode 100644 tests/shell/testcases/include/dumps/0020include_chain_0.json-nft
 create mode 100644 tests/shell/testcases/json/dumps/0001set_statements_0.json-nft
 create mode 100644 tests/shell/testcases/json/dumps/0002table_map_0.json-nft
 create mode 100644 tests/shell/testcases/json/dumps/0003json_schema_version_0.json-nft
 create mode 100644 tests/shell/testcases/json/dumps/0004json_schema_version_1.json-nft
 create mode 100644 tests/shell/testcases/json/dumps/0005secmark_objref_0.json-nft
 create mode 100644 tests/shell/testcases/json/dumps/0006obj_comment_0.json-nft
 create mode 100644 tests/shell/testcases/json/dumps/netdev.json-nft
 create mode 100644 tests/shell/testcases/listing/dumps/0001ruleset_0.json-nft
 create mode 100644 tests/shell/testcases/listing/dumps/0002ruleset_0.json-nft
 create mode 100644 tests/shell/testcases/listing/dumps/0003table_0.json-nft
 create mode 100644 tests/shell/testcases/listing/dumps/0004table_0.json-nft
 create mode 100644 tests/shell/testcases/listing/dumps/0005ruleset_ip_0.json-nft
 create mode 100644 tests/shell/testcases/listing/dumps/0006ruleset_ip6_0.json-nft
 create mode 100644 tests/shell/testcases/listing/dumps/0007ruleset_inet_0.json-nft
 create mode 100644 tests/shell/testcases/listing/dumps/0008ruleset_arp_0.json-nft
 create mode 100644 tests/shell/testcases/listing/dumps/0009ruleset_bridge_0.json-nft
 create mode 100644 tests/shell/testcases/listing/dumps/0010sets_0.json-nft
 create mode 100644 tests/shell/testcases/listing/dumps/0011sets_0.json-nft
 create mode 100644 tests/shell/testcases/listing/dumps/0012sets_0.json-nft
 create mode 100644 tests/shell/testcases/listing/dumps/0014objects_0.json-nft
 create mode 100644 tests/shell/testcases/listing/dumps/0015dynamic_0.json-nft
 create mode 100644 tests/shell/testcases/listing/dumps/0016anonymous_0.json-nft
 create mode 100644 tests/shell/testcases/listing/dumps/0017objects_0.json-nft
 create mode 100644 tests/shell/testcases/listing/dumps/0018data_0.json-nft
 create mode 100644 tests/shell/testcases/listing/dumps/0019set_0.json-nft
 create mode 100644 tests/shell/testcases/listing/dumps/0020flowtable_0.json-nft
 create mode 100644 tests/shell/testcases/listing/dumps/0021ruleset_json_terse_0.json-nft
 create mode 100644 tests/shell/testcases/listing/dumps/0022terse_0.json-nft
 create mode 100644 tests/shell/testcases/maps/dumps/0003map_add_many_elements_0.json-nft
 create mode 100644 tests/shell/testcases/maps/dumps/0005interval_map_add_many_elements_0.json-nft
 create mode 100644 tests/shell/testcases/maps/dumps/0006interval_map_overlap_0.json-nft
 create mode 100644 tests/shell/testcases/maps/dumps/0007named_ifname_dtype_0.json-nft
 create mode 100644 tests/shell/testcases/maps/dumps/0008interval_map_delete_0.json-nft
 create mode 100644 tests/shell/testcases/maps/dumps/0009vmap_0.json-nft
 create mode 100644 tests/shell/testcases/maps/dumps/0012map_0.json-nft
 create mode 100644 tests/shell/testcases/maps/dumps/0013map_0.json-nft
 create mode 100644 tests/shell/testcases/maps/dumps/0014destroy_0.json-nft
 create mode 100644 tests/shell/testcases/maps/dumps/0016map_leak_0.json-nft
 create mode 100644 tests/shell/testcases/maps/dumps/0017_map_variable_0.json-nft
 create mode 100644 tests/shell/testcases/maps/dumps/0018map_leak_timeout_0.json-nft
 create mode 100644 tests/shell/testcases/maps/dumps/anon_objmap_concat.json-nft
 create mode 100644 tests/shell/testcases/maps/dumps/anonymous_snat_map_0.json-nft
 create mode 100644 tests/shell/testcases/maps/dumps/different_map_types_1.json-nft
 create mode 100644 tests/shell/testcases/maps/dumps/map_catchall_double_deactivate.json-nft
 create mode 100644 tests/shell/testcases/maps/dumps/map_with_flags_0.json-nft
 create mode 100644 tests/shell/testcases/maps/dumps/named_snat_map_0.json-nft
 create mode 100644 tests/shell/testcases/maps/dumps/typeof_maps_add_delete.json-nft
 create mode 100644 tests/shell/testcases/maps/dumps/typeof_maps_update_0.json-nft
 create mode 100644 tests/shell/testcases/netns/dumps/0001nft-f_0.json-nft
 create mode 100644 tests/shell/testcases/netns/dumps/0002loosecommands_0.json-nft
 create mode 100644 tests/shell/testcases/netns/dumps/0003many_0.json-nft
 create mode 100644 tests/shell/testcases/nft-f/dumps/0001define_slash_0.json-nft
 create mode 100644 tests/shell/testcases/nft-f/dumps/0002rollback_rule_0.json-nft
 create mode 100644 tests/shell/testcases/nft-f/dumps/0003rollback_jump_0.json-nft
 create mode 100644 tests/shell/testcases/nft-f/dumps/0004rollback_set_0.json-nft
 create mode 100644 tests/shell/testcases/nft-f/dumps/0005rollback_map_0.json-nft
 create mode 100644 tests/shell/testcases/nft-f/dumps/0006action_object_0.json-nft
 create mode 100644 tests/shell/testcases/nft-f/dumps/0007action_object_set_segfault_1.json-nft
 create mode 100644 tests/shell/testcases/nft-f/dumps/0008split_tables_0.json-nft
 create mode 100644 tests/shell/testcases/nft-f/dumps/0009variable_0.json-nft
 create mode 100644 tests/shell/testcases/nft-f/dumps/0010variable_0.json-nft
 create mode 100644 tests/shell/testcases/nft-f/dumps/0013defines_1.json-nft
 create mode 100644 tests/shell/testcases/nft-f/dumps/0014defines_1.json-nft
 create mode 100644 tests/shell/testcases/nft-f/dumps/0015defines_1.json-nft
 create mode 100644 tests/shell/testcases/nft-f/dumps/0016redefines_1.json-nft
 create mode 100644 tests/shell/testcases/nft-f/dumps/0018ct_expectation_obj_0.json-nft
 create mode 100644 tests/shell/testcases/nft-f/dumps/0018jump_variable_0.json-nft
 create mode 100644 tests/shell/testcases/nft-f/dumps/0019jump_variable_1.json-nft
 create mode 100644 tests/shell/testcases/nft-f/dumps/0020jump_variable_1.json-nft
 create mode 100644 tests/shell/testcases/nft-f/dumps/0021list_ruleset_0.json-nft
 create mode 100644 tests/shell/testcases/nft-f/dumps/0022variables_0.json-nft
 create mode 100644 tests/shell/testcases/nft-f/dumps/0023check_1.json-nft
 create mode 100644 tests/shell/testcases/nft-f/dumps/0025empty_dynset_0.json-nft
 create mode 100644 tests/shell/testcases/nft-f/dumps/0026listing_0.json-nft
 create mode 100644 tests/shell/testcases/nft-f/dumps/0027split_chains_0.json-nft
 create mode 100644 tests/shell/testcases/nft-f/dumps/0028variable_cmdline_0.json-nft
 create mode 100644 tests/shell/testcases/nft-f/dumps/0029split_file_0.json-nft
 create mode 100644 tests/shell/testcases/nft-f/dumps/0030variable_reuse_0.json-nft
 create mode 100644 tests/shell/testcases/nft-f/dumps/0031vmap_string_0.json-nft
 create mode 100644 tests/shell/testcases/nft-f/dumps/0032pknock_0.json-nft
 create mode 100644 tests/shell/testcases/nft-i/dumps/0001define_0.json-nft
 create mode 100644 tests/shell/testcases/optimizations/dumps/dependency_kill.json-nft
 create mode 100644 tests/shell/testcases/optimizations/dumps/merge_nat.json-nft
 create mode 100644 tests/shell/testcases/optimizations/dumps/merge_reject.json-nft
 create mode 100644 tests/shell/testcases/optimizations/dumps/merge_stmts.json-nft
 create mode 100644 tests/shell/testcases/optimizations/dumps/merge_stmts_concat.json-nft
 create mode 100644 tests/shell/testcases/optimizations/dumps/merge_stmts_concat_vmap.json-nft
 create mode 100644 tests/shell/testcases/optimizations/dumps/merge_stmts_vmap.json-nft
 create mode 100644 tests/shell/testcases/optimizations/dumps/merge_vmap_raw.json-nft
 create mode 100644 tests/shell/testcases/optimizations/dumps/merge_vmaps.json-nft
 create mode 100644 tests/shell/testcases/optimizations/dumps/not_mergeable.json-nft
 create mode 100644 tests/shell/testcases/optimizations/dumps/ruleset.json-nft
 create mode 100644 tests/shell/testcases/optimizations/dumps/single_anon_set.json-nft
 create mode 100644 tests/shell/testcases/optimizations/dumps/skip_merge.json-nft
 create mode 100644 tests/shell/testcases/optimizations/dumps/skip_non_eq.json-nft
 create mode 100644 tests/shell/testcases/optimizations/dumps/skip_unsupported.json-nft
 create mode 100644 tests/shell/testcases/optimizations/dumps/variables.json-nft
 create mode 100644 tests/shell/testcases/optionals/dumps/comments_0.json-nft
 create mode 100644 tests/shell/testcases/optionals/dumps/comments_chain_0.json-nft
 create mode 100644 tests/shell/testcases/optionals/dumps/comments_handles_0.json-nft
 create mode 100644 tests/shell/testcases/optionals/dumps/comments_objects_dup_0.json-nft
 create mode 100644 tests/shell/testcases/optionals/dumps/comments_table_0.json-nft
 create mode 100644 tests/shell/testcases/optionals/dumps/delete_object_handles_0.json-nft
 create mode 100644 tests/shell/testcases/optionals/dumps/handles_0.json-nft
 create mode 100644 tests/shell/testcases/optionals/dumps/handles_1.json-nft
 create mode 100644 tests/shell/testcases/optionals/dumps/log_prefix_0.json-nft
 create mode 100644 tests/shell/testcases/optionals/dumps/update_object_handles_0.json-nft
 create mode 100644 tests/shell/testcases/owner/dumps/0001-flowtable-uaf.json-nft
 create mode 100644 tests/shell/testcases/parsing/dumps/describe.json-nft
 create mode 100644 tests/shell/testcases/parsing/dumps/large_rule_pipe.json-nft
 create mode 100644 tests/shell/testcases/parsing/dumps/log.json-nft
 create mode 100644 tests/shell/testcases/parsing/dumps/octal.json-nft
 create mode 100644 tests/shell/testcases/rule_management/dumps/0001addinsertposition_0.json-nft
 create mode 100644 tests/shell/testcases/rule_management/dumps/0002addinsertlocation_1.json-nft
 create mode 100644 tests/shell/testcases/rule_management/dumps/0003insert_0.json-nft
 create mode 100644 tests/shell/testcases/rule_management/dumps/0004replace_0.json-nft
 create mode 100644 tests/shell/testcases/rule_management/dumps/0005replace_1.json-nft
 create mode 100644 tests/shell/testcases/rule_management/dumps/0006replace_1.json-nft
 create mode 100644 tests/shell/testcases/rule_management/dumps/0007delete_0.json-nft
 create mode 100644 tests/shell/testcases/rule_management/dumps/0008delete_1.json-nft
 create mode 100644 tests/shell/testcases/rule_management/dumps/0009delete_1.json-nft
 create mode 100644 tests/shell/testcases/rule_management/dumps/0010replace_0.json-nft
 create mode 100644 tests/shell/testcases/rule_management/dumps/0011reset_0.json-nft
 create mode 100644 tests/shell/testcases/rule_management/dumps/0012destroy_0.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0001named_interval_0.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0002named_interval_automerging_0.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0003named_interval_missing_flag_0.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0004named_interval_shadow_0.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0005named_interval_shadow_0.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0006create_set_0.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0007create_element_0.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0008comments_interval_0.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0009comments_timeout_0.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0010comments_0.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0012add_delete_many_elements_0.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0013add_delete_many_elements_0.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0014malformed_set_is_not_defined_0.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0015rulesetflush_0.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0016element_leak_0.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0017add_after_flush_0.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0018set_check_size_1.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0019set_check_size_0.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0020comments_0.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0021nesting_0.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0022type_selective_flush_0.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0023incomplete_add_set_command_0.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0025anonymous_set_0.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0026named_limit_0.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0027ipv6_maps_ipv4_0.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0028autoselect_0.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0028delete_handle_0.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0032restore_set_simple_0.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0033add_set_simple_flat_0.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0034get_element_0.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0035add_set_elements_flat_0.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0037_set_with_inet_service_0.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0038meter_list_0.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0039delete_interval_0.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0040get_host_endian_elements_0.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0041interval_0.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0042update_set_0.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0043concatenated_ranges_0.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0043concatenated_ranges_1.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0044interval_overlap_1.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0045concat_ipv4_service.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0046netmap_0.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0048set_counters_0.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0049set_define_0.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0050set_define_1.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0051set_interval_counter_0.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0052overlap_0.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0053echo_0.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0054comments_set_0.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0055tcpflags_0.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0056dynamic_limit_0.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0057set_create_fails_0.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0058_setupdate_timeout_0.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0059set_update_multistmt_0.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0060set_multistmt_0.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0060set_multistmt_1.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0061anonymous_automerge_0.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0062set_connlimit_0.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0063set_catchall_0.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0064map_catchall_0.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0065_icmp_postprocessing.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0069interval_merge_0.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0071unclosed_prefix_interval_0.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0072destroy_0.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0073flat_interval_set.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/0074nested_interval_set.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/collapse_elem_0.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/concat_interval_0.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/dynset_missing.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/errors_0.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/exact_overlap_0.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/inner_0.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/set_eval_0.json-nft
 create mode 100644 tests/shell/testcases/sets/dumps/type_set_symbol.json-nft
 create mode 100644 tests/shell/testcases/transactions/dumps/0001table_0.json-nft
 create mode 100644 tests/shell/testcases/transactions/dumps/0002table_0.json-nft
 create mode 100644 tests/shell/testcases/transactions/dumps/0003table_0.json-nft
 create mode 100644 tests/shell/testcases/transactions/dumps/0010chain_0.json-nft
 create mode 100644 tests/shell/testcases/transactions/dumps/0011chain_0.json-nft
 create mode 100644 tests/shell/testcases/transactions/dumps/0012chain_0.json-nft
 create mode 100644 tests/shell/testcases/transactions/dumps/0013chain_0.json-nft
 create mode 100644 tests/shell/testcases/transactions/dumps/0014chain_1.json-nft
 create mode 100644 tests/shell/testcases/transactions/dumps/0015chain_0.json-nft
 create mode 100644 tests/shell/testcases/transactions/dumps/0020rule_0.json-nft
 create mode 100644 tests/shell/testcases/transactions/dumps/0021rule_0.json-nft
 create mode 100644 tests/shell/testcases/transactions/dumps/0022rule_1.json-nft
 create mode 100644 tests/shell/testcases/transactions/dumps/0023rule_1.json-nft
 create mode 100644 tests/shell/testcases/transactions/dumps/0024rule_0.json-nft
 create mode 100644 tests/shell/testcases/transactions/dumps/0025rule_0.json-nft
 create mode 100644 tests/shell/testcases/transactions/dumps/0030set_0.json-nft
 create mode 100644 tests/shell/testcases/transactions/dumps/0031set_0.json-nft
 create mode 100644 tests/shell/testcases/transactions/dumps/0032set_0.json-nft
 create mode 100644 tests/shell/testcases/transactions/dumps/0033set_0.json-nft
 create mode 100644 tests/shell/testcases/transactions/dumps/0034set_0.json-nft
 create mode 100644 tests/shell/testcases/transactions/dumps/0035set_0.json-nft
 create mode 100644 tests/shell/testcases/transactions/dumps/0036set_1.json-nft
 create mode 100644 tests/shell/testcases/transactions/dumps/0037set_0.json-nft
 create mode 100644 tests/shell/testcases/transactions/dumps/0038set_0.json-nft
 create mode 100644 tests/shell/testcases/transactions/dumps/0039set_0.json-nft
 create mode 100644 tests/shell/testcases/transactions/dumps/0040set_0.json-nft
 create mode 100644 tests/shell/testcases/transactions/dumps/0041nat_restore_0.json-nft
 create mode 100644 tests/shell/testcases/transactions/dumps/0042_stateful_expr_0.json-nft
 create mode 100644 tests/shell/testcases/transactions/dumps/0043set_1.json-nft
 create mode 100644 tests/shell/testcases/transactions/dumps/0044rule_0.json-nft
 create mode 100644 tests/shell/testcases/transactions/dumps/0045anon-unbind_0.json-nft
 create mode 100644 tests/shell/testcases/transactions/dumps/0046set_0.json-nft
 create mode 100644 tests/shell/testcases/transactions/dumps/0047set_0.json-nft
 create mode 100644 tests/shell/testcases/transactions/dumps/0048helpers_0.json-nft
 create mode 100644 tests/shell/testcases/transactions/dumps/0049huge_0.json-nft
 create mode 100644 tests/shell/testcases/transactions/dumps/0050rule_1.json-nft
 create mode 100644 tests/shell/testcases/transactions/dumps/30s-stress.json-nft
 create mode 100644 tests/shell/testcases/transactions/dumps/anon_chain_loop.json-nft
 create mode 100644 tests/shell/testcases/transactions/dumps/bad_expression.json-nft
 create mode 100644 tests/shell/testcases/transactions/dumps/table_onoff.json-nft

-- 
2.41.0

[PATCH nft 1/7] json: fix use after free in table_flags_json()

[Thomas Haller]

Valgrind complains about this:

  ==286== Invalid read of size 4
  ==286==    at 0x49B0261: do_dump (dump.c:211)
  ==286==    by 0x49B08B8: do_dump (dump.c:378)
  ==286==    by 0x49B08B8: do_dump (dump.c:378)
  ==286==    by 0x49B04F7: do_dump (dump.c:273)
  ==286==    by 0x49B08B8: do_dump (dump.c:378)
  ==286==    by 0x49B0E84: json_dump_callback (dump.c:465)
  ==286==    by 0x48AF22A: do_command_list_json (json.c:2016)
  ==286==    by 0x48732F1: do_command_list (rule.c:2335)
  ==286==    by 0x48737F5: do_command (rule.c:2605)
  ==286==    by 0x48A867D: nft_netlink (libnftables.c:42)
  ==286==    by 0x48A92B1: nft_run_cmd_from_buffer (libnftables.c:597)
  ==286==    by 0x402CBA: main (main.c:533)
  ==286==  Address 0x5038650 is 0 bytes inside a block of size 32 free'd
  ==286==    at 0x48452AC: free (vg_replace_malloc.c:974)
  ==286==    by 0x49AECDD: UnknownInlinedFun (jansson.h:133)
  ==286==    by 0x49AECDD: UnknownInlinedFun (jansson.h:131)
  ==286==    by 0x49AECDD: UnknownInlinedFun (value.c:398)
  ==286==    by 0x49AECDD: json_delete (value.c:953)
  ==286==    by 0x48A9F61: json_decref (jansson.h:133)
  ==286==    by 0x48AA4AA: table_flags_json (json.c:494)
  ==286==    by 0x48AA52B: table_print_json (json.c:510)
  ==286==    by 0x48ABBAE: table_print_json_full (json.c:1695)
  ==286==    by 0x48ABD48: do_list_ruleset_json (json.c:1739)
  ==286==    by 0x48AF2A0: do_command_list_json (json.c:1962)
  ==286==    by 0x48732F1: do_command_list (rule.c:2335)
  ==286==    by 0x48737F5: do_command (rule.c:2605)
  ==286==    by 0x48A867D: nft_netlink (libnftables.c:42)
  ==286==    by 0x48A92B1: nft_run_cmd_from_buffer (libnftables.c:597)
  ==286==    by 0x402CBA: main (main.c:533)
  ==286==  Block was alloc'd at
  ==286==    at 0x484282F: malloc (vg_replace_malloc.c:431)
  ==286==    by 0x49AE4EA: UnknownInlinedFun (memory.c:27)
  ==286==    by 0x49AE4EA: UnknownInlinedFun (value.c:676)
  ==286==    by 0x49AE4EA: json_stringn_nocheck (value.c:696)
  ==286==    by 0x48AA464: table_flags_json (json.c:482)
  ==286==    by 0x48AA52B: table_print_json (json.c:510)
  ==286==    by 0x48ABBAE: table_print_json_full (json.c:1695)
  ==286==    by 0x48ABD48: do_list_ruleset_json (json.c:1739)
  ==286==    by 0x48AF2A0: do_command_list_json (json.c:1962)
  ==286==    by 0x48732F1: do_command_list (rule.c:2335)
  ==286==    by 0x48737F5: do_command (rule.c:2605)
  ==286==    by 0x48A867D: nft_netlink (libnftables.c:42)
  ==286==    by 0x48A92B1: nft_run_cmd_from_buffer (libnftables.c:597)
  ==286==    by 0x402CBA: main (main.c:533)

Fixes: e70354f53e9f ("libnftables: Implement JSON output support")
Signed-off-by: Thomas Haller <thaller@redhat.com>
---
 src/json.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/json.c b/src/json.c
index 068c423addc7..c0ccf06d85b4 100644
--- a/src/json.c
+++ b/src/json.c
@@ -496,7 +496,7 @@ static json_t *table_flags_json(const struct table *table)
 		json_decref(root);
 		return NULL;
 	case 1:
-		json_unpack(root, "[o]", &tmp);
+		json_unpack(root, "[O]", &tmp);
 		json_decref(root);
 		root = tmp;
 		break;
-- 
2.41.0

[PATCH nft 2/7] json: drop messages "warning: stmt ops chain have no json callback"

[Thomas Haller]

This message purely depends on the internal callbacks and at the program
code. This is not useful. What is the user going to do with this warning?

Maybe there is a bug here, but then we shouldn't print a warning but fix
the bug.

For example, calling `nft -j list ruleset` after test "tests/shell/testcases/chains/0041chain_binding_0"
will trigger messages like:

  warning: stmt ops chain have no json callback
  warning: stmt ops chain have no json callback

Signed-off-by: Thomas Haller <thaller@redhat.com>
---
 src/json.c | 6 ------
 1 file changed, 6 deletions(-)

diff --git a/src/json.c b/src/json.c
index c0ccf06d85b4..c66b58f8c6d5 100644
--- a/src/json.c
+++ b/src/json.c
@@ -52,9 +52,6 @@ static json_t *expr_print_json(const struct expr *expr, struct output_ctx *octx)
 	if (ops->json)
 		return ops->json(expr, octx);
 
-	fprintf(stderr, "warning: expr ops %s have no json callback\n",
-		expr_name(expr));
-
 	fp = octx->output_fp;
 	octx->output_fp = fmemopen(buf, 1024, "w");
 
@@ -95,9 +92,6 @@ static json_t *stmt_print_json(const struct stmt *stmt, struct output_ctx *octx)
 	if (stmt->ops->json)
 		return stmt->ops->json(stmt, octx);
 
-	fprintf(stderr, "warning: stmt ops %s have no json callback\n",
-		stmt->ops->name);
-
 	fp = octx->output_fp;
 	octx->output_fp = fmemopen(buf, 1024, "w");
 
-- 
2.41.0

Re: [PATCH nft 2/7] json: drop messages "warning: stmt ops chain have no json callback"

[Florian Westphal]

Thomas Haller <thaller@redhat.com> wrote:
> This message purely depends on the internal callbacks and at the program
> code. This is not useful. What is the user going to do with this warning?

Report a bug.

> Maybe there is a bug here, but then we shouldn't print a warning but fix
> the bug.

How on earth do we do that, fix a bug before we know its there?

> For example, calling `nft -j list ruleset` after test "tests/shell/testcases/chains/0041chain_binding_0"
> will trigger messages like:
> 
>   warning: stmt ops chain have no json callback
>   warning: stmt ops chain have no json callback

That means that chain ops needs a bug fix as they do not support
json output?

Re: [PATCH nft 2/7] json: drop messages "warning: stmt ops chain have no json callback"

[Thomas Haller]

On Thu, 2023-11-02 at 09:04 +0100, Florian Westphal wrote:
> Thomas Haller <thaller@redhat.com> wrote:
> > This message purely depends on the internal callbacks and at the
> > program
> > code. This is not useful. What is the user going to do with this
> > warning?
> 
> Report a bug.

The way the message is worded ("warning:"), doesn't sound as if a user
should feel invited to report a bug.


> > Maybe there is a bug here, but then we shouldn't print a warning
> > but fix
> > the bug.
> 
> How on earth do we do that, fix a bug before we know its there?

Writing tests that perform basic consistency checks of things that must
hold.

A unit test should check whether ops structs are consistent. This
patchset also enables a test that hits this problem (for "chain" ops).

If there is a bug, then an assertion (or just a plain crash) would be
preferable over printing a "warning" to stderr.

> 
> > For example, calling `nft -j list ruleset` after test
> > "tests/shell/testcases/chains/0041chain_binding_0"
> > will trigger messages like:
> > 
> >   warning: stmt ops chain have no json callback
> >   warning: stmt ops chain have no json callback
> 
> That means that chain ops needs a bug fix as they do not support
> json output?

I don't know. The message might just trying to be helpful, and there is
no actual problem. The condition is attempted to be handled by the
following code.

I don't know which it is. Me sending the patch is basically asking that
question.



Thomas

Re: [PATCH nft 2/7] json: drop messages "warning: stmt ops chain have no json callback"

[Thomas Haller]

On Thu, 2023-11-02 at 09:39 +0100, Thomas Haller wrote:
> On Thu, 2023-11-02 at 09:04 +0100, Florian Westphal wrote:
> 
> 
> I don't know which it is. Me sending the patch is basically asking
> that
> question.


This patch 2/7 is obsoleted/replaced by

	[PATCH nft 1/2] json: implement json() hook for "symbol_expr_ops"/"variabl_expr_ops"
	[PATCH nft 2/2] json: drop handling missing json() hook for "struct expr_ops"


Unit tests for consistency of "struct expr_ops" are still missing (I
will send them, once some groundwork patches are in).


Thomas

[PATCH nft 3/7] tests/shell: check and generate JSON dump files

[Thomas Haller]

The rules after a successful test are good opportunity to test
`nft -j list ruleset` and `nft -j --check`. This quite possibly touches
code paths that are not hit by other tests yet.

The only downside is the increase of the test runtime (which seems
negligible, given the benefits of increasing test coverage).

Note that with some tests the test wrapper now prints:

  "warning: stmt ops chain have no json callback"

That is a bug that will be fixed next.

Future commits will commit those ".json-nft" dump files.

Test "tests/shell/testcases/transactions/table_onoff" will trigger
a valgrind error during `nft -j list ruleset` which causes the test
to fail in valgrind mode. That needs to be fixed.

Signed-off-by: Thomas Haller <thaller@redhat.com>
---
 tests/shell/helpers/test-wrapper.sh | 138 +++++++++++++++++++++-------
 tests/shell/run-tests.sh            |  11 ++-
 2 files changed, 110 insertions(+), 39 deletions(-)

diff --git a/tests/shell/helpers/test-wrapper.sh b/tests/shell/helpers/test-wrapper.sh
index 872a0c56ed54..f56537f601e0 100755
--- a/tests/shell/helpers/test-wrapper.sh
+++ b/tests/shell/helpers/test-wrapper.sh
@@ -15,6 +15,16 @@ array_contains() {
 	return 1
 }
 
+show_file() {
+	local filename="$1"
+	shift
+	local msg="$*"
+
+	printf '%s\n>>>>\n' "$msg"
+	cat "$filename"
+	printf "<<<<\n"
+}
+
 TEST="$1"
 TESTBASE="$(basename "$TEST")"
 TESTDIR="$(dirname "$TEST")"
@@ -109,55 +119,103 @@ if [ "$rc_test" -eq 0 ] ; then
 	"${CMD[@]}" &>> "$NFT_TEST_TESTTMPDIR/testout.log" || rc_test=$?
 fi
 
-$NFT list ruleset > "$NFT_TEST_TESTTMPDIR/ruleset-after"
+rc_chkdump=0
+rc=0
+$NFT list ruleset > "$NFT_TEST_TESTTMPDIR/ruleset-after" 2> "$NFT_TEST_TESTTMPDIR/chkdump" || rc=$?
+if [ "$rc" -ne 0 -o -s "$NFT_TEST_TESTTMPDIR/chkdump" ] ; then
+	show_file "$NFT_TEST_TESTTMPDIR/chkdump" "Command \`$NFT list ruleset\"$DUMPFILE\"\` failed" >> "$NFT_TEST_TESTTMPDIR/rc-failed-chkdump"
+	rc_chkdump=1
+fi
+if [ "$NFT_TEST_HAVE_json" != n ] ; then
+	rc=0
+	$NFT -j list ruleset > "$NFT_TEST_TESTTMPDIR/ruleset-after.json" 2> "$NFT_TEST_TESTTMPDIR/chkdump" || rc=$?
+	if [ "$rc" -ne 0 -o -s "$NFT_TEST_TESTTMPDIR/chkdump" ] ; then
+		show_file "$NFT_TEST_TESTTMPDIR/chkdump" "Command \`$NFT -j list ruleset\"$DUMPFILE\"\` failed" >> "$NFT_TEST_TESTTMPDIR/rc-failed-chkdump"
+		rc_chkdump=1
+	fi
+	# Normalize the version number from the JSON output. Otherwise, we'd
+	# have to regenerate the .json-nft files upon release.
+	sed '1s/\({"nftables": \[{"metainfo": {"version": "\)[0-9.]\+\(", "release_name": "\)[^"]\+\(", "\)/\1VERSION\2RELEASE_NAME\3/' -i "$NFT_TEST_TESTTMPDIR/ruleset-after.json"
+fi
 
 read tainted_after < /proc/sys/kernel/tainted
 
 DUMPPATH="$TESTDIR/dumps"
 DUMPFILE="$DUMPPATH/$TESTBASE.nft"
+JDUMPFILE="$DUMPPATH/$TESTBASE.json-nft"
 NODUMPFILE="$DUMPPATH/$TESTBASE.nodump"
 
-dump_written=
-
-# The caller can request a re-geneating of the dumps, by setting
-# DUMPGEN=y.
+# The caller can request a re-geneating of the .nft, .nodump, .json-nft dump files
+# by setting DUMPGEN=y. In that case, only the existing files will be regenerated
+# (unless all three files are missing, in which case all of them are generated).
 #
-# This only will happen if the command completed with success.
-#
-# It also will only happen for tests, that have a "$DUMPPATH" directory. There
-# might be tests, that don't want to have dumps created. The existence of the
-# directory controls that. Tests that have a "$NODUMPFILE" file, don't get a dump generated.
-if [ "$rc_test" -eq 0 -a "$DUMPGEN" = y -a -d "$DUMPPATH" -a ! -f "$NODUMPFILE" ] ; then
+# By setting DUMPGEN=force, all 3 files are always regenerated.
+dump_written=n
+if [ "$rc_test" -eq 0 -a '(' "$DUMPGEN" = force -o "$DUMPGEN" = y ')' ] ; then
 	dump_written=y
-	if [ ! -f "$DUMPFILE" ] ; then
-		# No dumpfile exists yet. We generate both a .nft and a .nodump
-		# file. The user can pick which one to commit to git.
+	if [ ! -d "$DUMPPATH" ] ; then
+		mkdir "$DUMPPATH"
+	fi
+	if [ "$DUMPGEN" = force ] ; then
+		gen_nodumpfile=y
+		gen_dumpfile=y
+		gen_jdumpfile=y
+	else
+		# by default, only regenerate the files that we already have on disk.
+		gen_nodumpfile=n
+		gen_dumpfile=n
+		gen_jdumpfile=n
+		test -f "$DUMPFILE"  && gen_dumpfile=y
+		test -f "$JDUMPFILE" && gen_jdumpfile=y
+		test -f "$NODUMPFILE" && gen_nodumpfile=y
+		if [ "$gen_dumpfile" = y -a "$gen_jdumpfile" = y -a "$gen_nodumpfile" = y ] ; then
+			# Except, if no files exist. Them generate all files.
+			gen_dumpfile=y
+			gen_jdumpfile=y
+			gen_nodumpfile=y
+		fi
+	fi
+	if [ "$gen_nodumpfile" = y ] ; then
 		: > "$NODUMPFILE"
 	fi
-	cat "$NFT_TEST_TESTTMPDIR/ruleset-after" > "$DUMPFILE"
+	if [ "$gen_dumpfile" = y ] ; then
+		cat "$NFT_TEST_TESTTMPDIR/ruleset-after" > "$DUMPFILE"
+	fi
+	if [ "$NFT_TEST_HAVE_json" != n -a "$gen_jdumpfile" = y ] ; then
+		cat "$NFT_TEST_TESTTMPDIR/ruleset-after.json" > "$JDUMPFILE"
+	fi
 fi
 
 rc_dump=0
-if [ "$rc_test" -ne 77 -a -f "$DUMPFILE" ] ; then
-	if [ "$dump_written" != y ] ; then
+if [ "$rc_test" -ne 77 -a "$dump_written" != y ] ; then
+	if [ -f "$DUMPFILE" ] ; then
 		if ! $DIFF -u "$DUMPFILE" "$NFT_TEST_TESTTMPDIR/ruleset-after" &> "$NFT_TEST_TESTTMPDIR/ruleset-diff" ; then
+			show_file "$NFT_TEST_TESTTMPDIR/ruleset-diff" "Failed \`$DIFF -u \"$DUMPFILE\" \"$NFT_TEST_TESTTMPDIR/ruleset-after\"\`" >> "$NFT_TEST_TESTTMPDIR/rc-failed-dump"
 			rc_dump=1
 		else
 			rm -f "$NFT_TEST_TESTTMPDIR/ruleset-diff"
 		fi
 	fi
-fi
-if [ "$rc_dump" -ne 0 ] ; then
-	echo "$DUMPFILE" > "$NFT_TEST_TESTTMPDIR/rc-failed-dump"
+	if [ "$NFT_TEST_HAVE_json" != n -a -f "$JDUMPFILE" ] ; then
+		if ! $DIFF -u "$JDUMPFILE" "$NFT_TEST_TESTTMPDIR/ruleset-after.json" &> "$NFT_TEST_TESTTMPDIR/ruleset-diff.json" ; then
+			show_file "$NFT_TEST_TESTTMPDIR/ruleset-diff.json" "Failed \`$DIFF -u \"$JDUMPFILE\" \"$NFT_TEST_TESTTMPDIR/ruleset-after.json\"\`" >> "$NFT_TEST_TESTTMPDIR/rc-failed-dump"
+			rc_dump=1
+		else
+			rm -f "$NFT_TEST_TESTTMPDIR/ruleset-diff.json"
+		fi
+	fi
 fi
 
-rc_chkdump=0
 # check that a flush after the test succeeds. We anyway need a clean ruleset
 # for the `nft --check` next.
-$NFT flush ruleset &> "$NFT_TEST_TESTTMPDIR/rc-failed-chkdump" || rc_chkdump=1
+rc=0
+$NFT flush ruleset &> "$NFT_TEST_TESTTMPDIR/chkdump" || rc=1
+if [ "$rc" = 1 -o -s "$NFT_TEST_TESTTMPDIR/chkdump" ] ; then
+	show_file "$NFT_TEST_TESTTMPDIR/chkdump" "Command \`$NFT flush ruleset\` failed" >> "$NFT_TEST_TESTTMPDIR/rc-failed-chkdump"
+	rc_chkdump=1
+fi
+# For the dumpfiles, call `nft --check` to possibly cover new code paths.
 if [ -f "$DUMPFILE" ] ; then
-	# We have a dumpfile. Call `nft --check` to possibly cover new code
-	# paths.
 	if [ "$rc_test" -eq 77 ] ; then
 		# The test was skipped. Possibly we don't have the required
 		# features to process this file. Ignore any output and exit
@@ -165,20 +223,30 @@ if [ -f "$DUMPFILE" ] ; then
 		# issue we hope to find).
 		$NFT --check -f "$DUMPFILE" &>/dev/null || :
 	else
-		$NFT --check -f "$DUMPFILE" &>> "$NFT_TEST_TESTTMPDIR/rc-failed-chkdump" || rc_chkdump=1
+		fail=n
+		$NFT --check -f "$DUMPFILE" &> "$NFT_TEST_TESTTMPDIR/chkdump" || fail=y
+		test -s "$NFT_TEST_TESTTMPDIR/chkdump" && fail=y
+		if [ "$fail" = y ] ; then
+			show_file "$NFT_TEST_TESTTMPDIR/chkdump" "Command \`$NFT --check -f \"$DUMPFILE\"\` failed" >> "$NFT_TEST_TESTTMPDIR/rc-failed-chkdump"
+			rc_chkdump=1
+		fi
+		rm -f "$NFT_TEST_TESTTMPDIR/chkdump"
 	fi
 fi
-if [ -s "$NFT_TEST_TESTTMPDIR/rc-failed-chkdump" ] ; then
-	# Non-empty output? That is wrong.
-	rc_chkdump=1
-elif [ "$rc_chkdump" -eq 0 ] ; then
-	rm -rf "$NFT_TEST_TESTTMPDIR/rc-failed-chkdump"
-fi
-if [ "$rc_chkdump" -ne 0 ] ; then
-	# Ensure we don't have empty output files. Always write something, so
-	# that `grep ^ -R` lists the file.
-	echo -e "<<<<<\n\nCalling \`nft --check\` (or \`nft flush ruleset\`) failed for \"$DUMPFILE\"" >> "$NFT_TEST_TESTTMPDIR/rc-failed-chkdump"
+if [ "$NFT_TEST_HAVE_json" != n -a -f "$JDUMPFILE" ] ; then
+	if [ "$rc_test" -eq 77 ] ; then
+		$NFT -j --check -f "$JDUMPFILE" &>/dev/null || :
+	else
+		fail=n
+		$NFT -j --check -f "$JDUMPFILE" &> "$NFT_TEST_TESTTMPDIR/chkdump" || fail=y
+		test -s "$NFT_TEST_TESTTMPDIR/chkdump" && fail=y
+		if [ "$fail" = y ] ; then
+			show_file "$NFT_TEST_TESTTMPDIR/chkdump" "Command \`$NFT -j --check -f \"$JDUMPFILE\"\` failed" >> "$NFT_TEST_TESTTMPDIR/rc-failed-chkdump"
+			rc_chkdump=1
+		fi
+	fi
 fi
+rm -f "$NFT_TEST_TESTTMPDIR/chkdump"
 
 rc_valgrind=0
 [ -f "$NFT_TEST_TESTTMPDIR/rc-failed-valgrind" ] && rc_valgrind=1
diff --git a/tests/shell/run-tests.sh b/tests/shell/run-tests.sh
index 27a0ec43042a..846de8429e1f 100755
--- a/tests/shell/run-tests.sh
+++ b/tests/shell/run-tests.sh
@@ -184,9 +184,10 @@ usage() {
 	echo " VERBOSE=*|y   : Enable verbose output."
 	echo " NFT_TEST_VERBOSE_TEST=*|y: if true, enable verbose output for tests. For bash scripts, this means"
 	echo "                 to pass \"-x\" to the interpreter."
-	echo " DUMPGEN=*|y   : Regenerate dump files. Dump files are only recreated if the"
-	echo "                 test completes successfully and the \"dumps\" directory for the"
-	echo "                 test exits."
+	echo " DUMPGEN=*|y|force : Regenerate dump files .nft, .json-nft, .nodump. \"DUMPGEN=y\" only regenerates existing"
+	echo "                 files, unless the test has no files (then all three files are generated, and you need to"
+	echo "                 choose those to keep). With \"DUMPGEN=force\" all 3 files are regenerated, regardless"
+	echo "                 whether they existed."
 	echo " VALGRIND=*|y  : Run \$NFT in valgrind."
 	echo " KMEMLEAK=*|y  : Check for kernel memleaks."
 	echo " NFT_TEST_HAS_REALROOT=*|y : To indicate whether the test has real root permissions."
@@ -279,7 +280,9 @@ _NFT_TEST_JOBS_DEFAULT="$(( _NFT_TEST_JOBS_DEFAULT + (_NFT_TEST_JOBS_DEFAULT + 1
 
 VERBOSE="$(bool_y "$VERBOSE")"
 NFT_TEST_VERBOSE_TEST="$(bool_y "$NFT_TEST_VERBOSE_TEST")"
-DUMPGEN="$(bool_y "$DUMPGEN")"
+if [ "$DUMPGEN" != "force" ] ; then
+	DUMPGEN="$(bool_y "$DUMPGEN")"
+fi
 VALGRIND="$(bool_y "$VALGRIND")"
 KMEMLEAK="$(bool_y "$KMEMLEAK")"
 NFT_TEST_KEEP_LOGS="$(bool_y "$NFT_TEST_KEEP_LOGS")"
-- 
2.41.0

[PATCH nft 5/7] tools: simplify error handling in "check-tree.sh" by adding msg_err()/msg_warn()

[Thomas Haller]

msg_err() also sets EXIT_CODE=, so we don't have to duplicate this.

Also add msg_warn() to print non-fatal warnings. Will be used in the
future. As "check-tree.sh" tests the consistency of the source tree, a
warning only makes sense to point something out that really should be
fixed, but is not yet.

Signed-off-by: Thomas Haller <thaller@redhat.com>
---
 tools/check-tree.sh | 37 ++++++++++++++++++++-----------------
 1 file changed, 20 insertions(+), 17 deletions(-)

diff --git a/tools/check-tree.sh b/tools/check-tree.sh
index e3ddf8bdea58..b16d37c4651b 100755
--- a/tools/check-tree.sh
+++ b/tools/check-tree.sh
@@ -25,6 +25,15 @@ cd "$(dirname "$0")/.."
 
 EXIT_CODE=0
 
+msg_err() {
+	printf "ERR:  %s\n" "$*"
+	EXIT_CODE=1
+}
+
+msg_warn() {
+	printf "WARN: %s\n" "$*"
+}
+
 ##############################################################################
 
 check_shell_dumps() {
@@ -37,8 +46,7 @@ check_shell_dumps() {
 	local nodump_name
 
 	if [ ! -d "$dir/dumps/" ] ; then
-		echo "\"$TEST\" has no \"$dir/dumps/\" directory"
-		EXIT_CODE=1
+		msg_err "\"$TEST\" has no \"$dir/dumps/\" directory"
 		return 0
 	fi
 
@@ -49,34 +57,31 @@ check_shell_dumps() {
 	[ -f "$nodump_name" ] && has_nodump=1
 
 	if [ "$has_nft" != 1 -a "$has_nodump" != 1 ] ; then
-		echo "\"$TEST\" has no \"$dir/dumps/$base.{nft,nodump}\" file"
-		EXIT_CODE=1
+		msg_err "\"$TEST\" has no \"$dir/dumps/$base.{nft,nodump}\" file"
 	elif [ "$has_nft" == 1 -a "$has_nodump" == 1 ] ; then
-		echo "\"$TEST\" has both \"$dir/dumps/$base.{nft,nodump}\" files"
-		EXIT_CODE=1
+		msg_err "\"$TEST\" has both \"$dir/dumps/$base.{nft,nodump}\" files"
 	elif [ "$has_nodump" == 1 -a -s "$nodump_name" ] ; then
-		echo "\"$TEST\" has a non-empty \"$dir/dumps/$base.nodump\" file"
-		EXIT_CODE=1
+		msg_err "\"$TEST\" has a non-empty \"$dir/dumps/$base.nodump\" file"
 	fi
 }
 
 SHELL_TESTS=( $(find "tests/shell/testcases/" -type f -executable | sort) )
 
 if [ "${#SHELL_TESTS[@]}" -eq 0 ] ; then
-	echo "No executable tests under \"tests/shell/testcases/\" found"
-	EXIT_CODE=1
+	msg_err "No executable tests under \"tests/shell/testcases/\" found"
 fi
 for t in "${SHELL_TESTS[@]}" ; do
 	check_shell_dumps "$t"
-	head -n 1 "$t" |grep -q  '^#!/bin/sh' && echo "$t uses sh instead of bash" && EXIT_CODE=1
+	if head -n 1 "$t" |grep -q  '^#!/bin/sh' ; then
+		msg_err "$t uses #!/bin/sh instead of /bin/bash"
+	fi
 done
 
 ##############################################################################
 
 SHELL_TESTS2=( $(./tests/shell/run-tests.sh --list-tests) )
 if [ "${SHELL_TESTS[*]}" != "${SHELL_TESTS2[*]}" ] ; then
-	echo "\`./tests/shell/run-tests.sh --list-tests\` does not list the expected tests"
-	EXIT_CODE=1
+	msg_err "\`./tests/shell/run-tests.sh --list-tests\` does not list the expected tests"
 fi
 
 ##############################################################################
@@ -85,8 +90,7 @@ F=( $(find tests/shell/testcases/ -type f | grep '^tests/shell/testcases/[^/]\+/
 IGNORED_FILES=( tests/shell/testcases/bogons/nft-f/* )
 for f in "${F[@]}" ; do
 	if ! array_contains "$f" "${SHELL_TESTS[@]}" "${IGNORED_FILES[@]}" ; then
-		echo "Unexpected file \"$f\""
-		EXIT_CODE=1
+		msg_err "Unexpected file \"$f\""
 	fi
 done
 
@@ -97,8 +101,7 @@ FILES=( $(find "tests/shell/testcases/" -type f | sed -n 's#\(tests/shell/testca
 for f in "${FILES[@]}" ; do
 	f2="$(echo "$f" | sed -n 's#\(tests/shell/testcases\(/.*\)\?/\)dumps/\(.*\)\.\(nft\|nodump\)$#\1\3#p')"
 	if ! array_contains "$f2" "${SHELL_TESTS[@]}" ; then
-		echo "\"$f\" has no test \"$f2\""
-		EXIT_CODE=1
+		msg_err "\"$f\" has no test \"$f2\""
 	fi
 done
 
-- 
2.41.0

[PATCH nft 6/7] tools: check more strictly for bash shebang in "check-tree.sh"

[Thomas Haller]

There is no principle problem to allow any executable/shebang. However,
it also not clear why we would want to use anything except bash.  Unless
we have a good reason, check and reject anything else.

Also not that `./tests/shell/run-tests.sh -x` only works if the shebang
is either "#!/bin/bash" or "#!/bin/bash -e". It probably could also work
with other tests, but it's unclear what they are and how to enable
verbose mode in that case.

Just check that they are all bash scripts. If there is a use-case, we
can always adjust this check.

Signed-off-by: Thomas Haller <thaller@redhat.com>
---
 tools/check-tree.sh | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/tools/check-tree.sh b/tools/check-tree.sh
index b16d37c4651b..4be874fcd85e 100755
--- a/tools/check-tree.sh
+++ b/tools/check-tree.sh
@@ -72,8 +72,13 @@ if [ "${#SHELL_TESTS[@]}" -eq 0 ] ; then
 fi
 for t in "${SHELL_TESTS[@]}" ; do
 	check_shell_dumps "$t"
-	if head -n 1 "$t" |grep -q  '^#!/bin/sh' ; then
-		msg_err "$t uses #!/bin/sh instead of /bin/bash"
+	if ! ( head -n 1 "$t" | grep -q '^#!/bin/bash\( -e\)\?$' ) ; then
+		# Currently all tests only use bash as shebang. That also
+		# works with `./tests/shell/run-tests.sh -x`.
+		#
+		# We could allow other shebangs, but for now there is no need.
+		# Unless you have a good reason, create a bash script.
+		msg_err "$t should use either \"#!/bin/bash\" or \"#!/bin/bash -e\" as shebang"
 	fi
 done
 
-- 
2.41.0

[PATCH nft 7/7] tools: check for consistency of .json-nft dumps in "check-tree.sh"

[Thomas Haller]

Add checks for the newly introduced .json-nft dump files.

Optimally, every test that has a .nft dump should also have a .json-nft
dump, and vice versa.

However, currently some JSON tests fail to validate, and are missing.
Only flag those missing files as warning, without failing the script.
The reason to warn about this, is that we really should fix those tests,
and having a annoying warning increases the pressure and makes it
discoverable.

Signed-off-by: Thomas Haller <thaller@redhat.com>
---
 tools/check-tree.sh | 21 ++++++++++++++++++++-
 1 file changed, 20 insertions(+), 1 deletion(-)

diff --git a/tools/check-tree.sh b/tools/check-tree.sh
index 4be874fcd85e..e358c957857e 100755
--- a/tools/check-tree.sh
+++ b/tools/check-tree.sh
@@ -41,6 +41,7 @@ check_shell_dumps() {
 	local base="$(basename "$TEST")"
 	local dir="$(dirname "$TEST")"
 	local has_nft=0
+	local has_jnft=0
 	local has_nodump=0
 	local nft_name
 	local nodump_name
@@ -51,9 +52,11 @@ check_shell_dumps() {
 	fi
 
 	nft_name="$dir/dumps/$base.nft"
+	jnft_name="$dir/dumps/$base.json-nft"
 	nodump_name="$dir/dumps/$base.nodump"
 
 	[ -f "$nft_name" ] && has_nft=1
+	[ -f "$jnft_name" ] && has_jnft=1
 	[ -f "$nodump_name" ] && has_nodump=1
 
 	if [ "$has_nft" != 1 -a "$has_nodump" != 1 ] ; then
@@ -63,6 +66,22 @@ check_shell_dumps() {
 	elif [ "$has_nodump" == 1 -a -s "$nodump_name" ] ; then
 		msg_err "\"$TEST\" has a non-empty \"$dir/dumps/$base.nodump\" file"
 	fi
+	if [ "$has_jnft" = 1 -a "$has_nft" != 1 ] ; then
+		msg_err "\"$TEST\" has a JSON dump file \"$jnft_name\" but lacks a dump \"$nft_name\""
+	fi
+	if [ "$has_nft" = 1 -a "$has_jnft" != 1 ] ; then
+		# it's currently known that `nft -j --check` cannot parse all dumped rulesets.
+		# Accept having no JSON dump file.
+		#
+		# This should be fixed. Currently this is only a warning.
+		msg_warn "\"$TEST\" has a dump file \"$nft_name\" but lacks a JSON dump \"$jnft_name\""
+	fi
+
+	if [ "$has_jnft" = 1 ] && command -v jq &>/dev/null ; then
+		if ! jq empty < "$jnft_name" &>/dev/null ; then
+			msg_err "\"$TEST\" has a JSON dump file \"$jnft_name\" that does not validate with \`jq empty < \"$jnft_name\"\`"
+		fi
+	fi
 }
 
 SHELL_TESTS=( $(find "tests/shell/testcases/" -type f -executable | sort) )
@@ -91,7 +110,7 @@ fi
 
 ##############################################################################
 #
-F=( $(find tests/shell/testcases/ -type f | grep '^tests/shell/testcases/[^/]\+/dumps/[^/]\+\.\(nft\|nodump\)$' -v | sort) )
+F=( $(find tests/shell/testcases/ -type f | grep '^tests/shell/testcases/[^/]\+/dumps/[^/]\+\.\(json-nft\|nft\|nodump\)$' -v | sort) )
 IGNORED_FILES=( tests/shell/testcases/bogons/nft-f/* )
 for f in "${F[@]}" ; do
 	if ! array_contains "$f" "${SHELL_TESTS[@]}" "${IGNORED_FILES[@]}" ; then
-- 
2.41.0

Re: [PATCH nft 0/7] add and check dump files for JSON in tests/shell

[Thomas Haller]

On Tue, 2023-10-31 at 19:53 +0100, Thomas Haller wrote:
> Like we have .nft dump files to compare the expected result, add
> .json-nft files that compare the JSON output.
> 
> Thomas Haller (7):
>   json: fix use after free in table_flags_json()
>   json: drop messages "warning: stmt ops chain have no json callback"
>   tests/shell: check and generate JSON dump files
>   tests/shell: add JSON dump files
>   tools: simplify error handling in "check-tree.sh" by adding
>     msg_err()/msg_warn()
>   tools: check more strictly for bash shebang in "check-tree.sh"
>   tools: check for consistency of .json-nft dumps in "check-tree.sh"

Hm. Patch 4/7 bounced (too large).

Will see how to resend, after there is some feedback.

The patch is also here:
https://gitlab.freedesktop.org/thaller/nftables/-/commit/6545b31080036e8525be5c80c0103a1509e698e4


Thomas

Re: [PATCH nft 0/7] add and check dump files for JSON in tests/shell

[Pablo Neira Ayuso]

On Tue, Oct 31, 2023 at 08:17:24PM +0100, Thomas Haller wrote:
> On Tue, 2023-10-31 at 19:53 +0100, Thomas Haller wrote:
> > Like we have .nft dump files to compare the expected result, add
> > .json-nft files that compare the JSON output.
> > 
> > Thomas Haller (7):
> >   json: fix use after free in table_flags_json()
> >   json: drop messages "warning: stmt ops chain have no json callback"
> >   tests/shell: check and generate JSON dump files
> >   tests/shell: add JSON dump files
> >   tools: simplify error handling in "check-tree.sh" by adding
> >     msg_err()/msg_warn()
> >   tools: check more strictly for bash shebang in "check-tree.sh"
> >   tools: check for consistency of .json-nft dumps in "check-tree.sh"

If this is improving json support coverage without imposing any extra
restriction other than adding a .nft-json file, then this is very good
to have.

I believe I switfly read on a commit message that this is skipped if
nft is compiled without json support, correct?

> Hm. Patch 4/7 bounced (too large).
>
> Will see how to resend, after there is some feedback.

I suggest you Cc: me so I can apply this.

> The patch is also here:
> https://gitlab.freedesktop.org/thaller/nftables/-/commit/6545b31080036e8525be5c80c0103a1509e698e4

You said:

"Note that for some JSON dumps, `nft -f --check` fails (or prints
something). For those tests no *.json-nft file is added. The bugs needs
to be fixed first."

Do you have a list of tests that are failing? Or maybe include this
list in the commit description? To keep them in the radar, we can
incrementally fix them.

Thanks.

Re: [PATCH nft 0/7] add and check dump files for JSON in tests/shell

[Thomas Haller]

On Wed, 2023-11-01 at 09:28 +0100, Pablo Neira Ayuso wrote:
> On Tue, Oct 31, 2023 at 08:17:24PM +0100, Thomas Haller wrote:
> > On Tue, 2023-10-31 at 19:53 +0100, Thomas Haller wrote:
> > > Like we have .nft dump files to compare the expected result, add
> > > .json-nft files that compare the JSON output.
> > > 
> > > Thomas Haller (7):
> > >   json: fix use after free in table_flags_json()
> > >   json: drop messages "warning: stmt ops chain have no json
> > > callback"
> > >   tests/shell: check and generate JSON dump files
> > >   tests/shell: add JSON dump files
> > >   tools: simplify error handling in "check-tree.sh" by adding
> > >     msg_err()/msg_warn()
> > >   tools: check more strictly for bash shebang in "check-tree.sh"
> > >   tools: check for consistency of .json-nft dumps in "check-
> > > tree.sh"
> 
> If this is improving json support coverage without imposing any extra
> restriction other than adding a .nft-json file, then this is very
> good
> to have.
> 
> I believe I switfly read on a commit message that this is skipped if
> nft is compiled without json support, correct?

Yes, that's how it's supposed to work (and tests correctly for me).

> 
> > Hm. Patch 4/7 bounced (too large).
> > 
> > Will see how to resend, after there is some feedback.
> 
> I suggest you Cc: me so I can apply this.


You could also run

  git fetch https://gitlab.freedesktop.org/thaller/nftables.git 6545b31080036e8525be5c80c0103a1509e698e4 ; git cherry-pick 6545b31080036e8525be5c80c0103a1509e698e4


Anyway. I will CC you on the patch as soon as I get to it!

> 
> > The patch is also here:
> > https://gitlab.freedesktop.org/thaller/nftables/-/commit/6545b31080036e8525be5c80c0103a1509e698e4
> 
> You said:
> 
> "Note that for some JSON dumps, `nft -f --check` fails (or prints
> something). For those tests no *.json-nft file is added. The bugs
> needs
> to be fixed first."

> 
> Do you have a list of tests that are failing? Or maybe include this
> list in the commit description? To keep them in the radar, we can
> incrementally fix them.


You can find them by running ./tools/check-tree.sh, it will print
warnings about the tests that lack the .json-nft file. It's exactly
those.

It prints for example:

 WARN: "tests/shell/testcases/cache/0010_implicit_chain_0" has a dump file "tests/shell/testcases/cache/dumps/0010_implicit_chain_0.nft" but lacks a JSON dump "tests/shell/testcases/cache/dumps/0010_implicit_chain_0.json-nft"


Thomas