All of lore.kernel.org
 help / color / mirror / Atom feed
* [RFC 00/13] objtool: Function validation tracing
@ 2025-06-06 15:34 Alexandre Chartre
  2025-06-06 15:34 ` [RFC 01/13] objtool: Move disassembly functions to a separated file Alexandre Chartre
                   ` (14 more replies)
  0 siblings, 15 replies; 35+ messages in thread
From: Alexandre Chartre @ 2025-06-06 15:34 UTC (permalink / raw)
  To: linux-kernel, mingo, jpoimboe, peterz; +Cc: alexandre.chartre

Hi,

This RFC provides two changes to objtool.

- Disassemble code with libopcodes instead of running objdump

  objtool executes the objdump command to disassemble code. In particular,
  if objtool fails to validate a function then it will use objdump to
  disassemble the entire file which is not very helpful when processing
  a large file (like vmlinux.o).

  Using libopcodes provides more control about the disassembly scope and
  output, and it is possible to disassemble a single instruction or
  a single function. Now when objtool fails to validate a function it
  will disassemble that single function instead of disassembling the
  entire file.

- Add the --trace <function> option to trace function validation

  Figuring out why a function validation has failed can be difficult because
  objtool checks all code flows (including alternatives) and maintains
  instructions states (in particular call frame information).

  The trace option allows to follow the function validation done by objtool
  instruction per instruction, see what objtool is doing and get function
  validation information. An output example is shown below.

Note: some changes are architecture specific (x86, powerpc, loongarch). So far,
I have only tested on x86, and some code might be x86 specific. Any feedback
about the behavior on powerpc and loongarch is welcome.

Thanks,

alex.


-----

Example: Trace the validation of the os_save() function in vmlinux.o

$ ./tools/objtool/objtool --hacks=jump_label --hacks=noinstr --hacks=skylake --ibt --orc --retpoline --rethunk --sls --static-call --uaccess --prefix=16 --link --trace os_xsave -v vmlinux.o
os_xsave: validation begin
 65c20: os_xsave+0x0  push %r12				- state: cfa=rsp+16 r12=(cfa-16) stack_size=16 
 65c22: os_xsave+0x2  mov 0x0(%rip),%eax # alternatives_patched 
 65c28: os_xsave+0x8  push %rbp 			- state: cfa=rsp+24 rbp=(cfa-24) stack_size=24 
 65c29: os_xsave+0x9  mov %rdi,%rbp 
 65c2c: os_xsave+0xc  push %rbx 			- state: cfa=rsp+32 rbx=(cfa-32) stack_size=32 
 65c2d: os_xsave+0xd  mov 0x8(%rdi),%rbx 
 65c31: os_xsave+0x11 mov %rbx,%r12 
 65c34: os_xsave+0x14 shr $0x20,%r12 
 65c38: os_xsave+0x18 test %eax,%eax 
 65c3a: os_xsave+0x1a je 65c6a <os_xsave+0x4a> 		- jump taken
 65c6a: os_xsave+0x4a | ud2 
 65c6c: os_xsave+0x4c | jmp 65c3c <os_xsave+0x1c> 	- unconditional jump
 65c3c: os_xsave+0x1c | xor %edx,%edx 
 65c3e: os_xsave+0x1e | mov %rbx,%rsi 
 65c41: os_xsave+0x21 | mov %rbp,%rdi 
 65c44: os_xsave+0x24 | callq xfd_validate_state 	- call
 65c49: os_xsave+0x29 | mov %ebx,%eax 
 65c4b: os_xsave+0x2b | mov %r12d,%edx 
 65c4e: os_xsave+0x2e | <alternative.65c4e> alt 1/4 begin
 65c55: os_xsave+0x35 | | test %ebx,%ebx 
 65c57: os_xsave+0x37 | | jne 65c6e <os_xsave+0x4e> 	- jump taken
 65c6e: os_xsave+0x4e | | | ud2 
 65c70: os_xsave+0x50 | | | pop %rbx 			- state: cfa=rsp+24 rbx=<undef> stack_size=24 
 65c71: os_xsave+0x51 | | | pop %rbp 			- state: cfa=rsp+16 rbp=<undef> stack_size=16 
 65c72: os_xsave+0x52 | | | pop %r12 			- state: cfa=rsp+8 r12=<undef> stack_size=8 
 65c74: os_xsave+0x54 | | | xor %eax,%eax 
 65c76: os_xsave+0x56 | | | xor %edx,%edx 
 65c78: os_xsave+0x58 | | | xor %esi,%esi 
 65c7a: os_xsave+0x5a | | | xor %edi,%edi 
 65c7c: os_xsave+0x5c | | | jmpq __x86_return_thunk 	- return
 65c57: os_xsave+0x37 | | jne 65c6e <os_xsave+0x4e> 	- jump not taken
 65c59: os_xsave+0x39 | | pop %rbx 			- state: cfa=rsp+24 rbx=<undef> stack_size=24 
 65c5a: os_xsave+0x3a | | pop %rbp 			- state: cfa=rsp+16 rbp=<undef> stack_size=16 
 65c5b: os_xsave+0x3b | | pop %r12 			- state: cfa=rsp+8 r12=<undef> stack_size=8 
 65c5d: os_xsave+0x3d | | xor %eax,%eax 
 65c5f: os_xsave+0x3f | | xor %edx,%edx 
 65c61: os_xsave+0x41 | | xor %esi,%esi 
 65c63: os_xsave+0x43 | | xor %edi,%edi 
 65c65: os_xsave+0x45 | | jmpq __x86_return_thunk - return
                      | <alternative.65c4e> alt 1/4 end
 65c4e: os_xsave+0x2e | <alternative.65c4e> alt 2/4 begin
 1c3d: .altinstr_replacement+0x1c3d | | xsaves64 0x40(%rbp) 
 65c53: os_xsave+0x33 | | xor %ebx,%ebx 
 65c55: os_xsave+0x35 | | test %ebx,%ebx - already visited
                      | <alternative.65c4e> alt 2/4 end
 65c4e: os_xsave+0x2e | <alternative.65c4e> alt 3/4 begin
 1c38: .altinstr_replacement+0x1c38 | | xsavec64 0x40(%rbp) 
 65c53: os_xsave+0x33 | | xor %ebx,%ebx - already visited
                      | <alternative.65c4e> alt 3/4 end
 65c4e: os_xsave+0x2e | <alternative.65c4e> alt 4/4 begin
 1c33: .altinstr_replacement+0x1c33 | | xsaveopt64 0x40(%rbp) 
 65c53: os_xsave+0x33 | | xor %ebx,%ebx - already visited
                      | <alternative.65c4e> alt 4/4 end
 65c4e: os_xsave+0x2e | <alternative.65c4e> alt default
 65c4e: os_xsave+0x2e | xsave64 0x40(%rbp) 
 65c53: os_xsave+0x33 | xor %ebx,%ebx - already visited
 65c3a: os_xsave+0x1a je 65c6a <os_xsave+0x4a> - jump not taken
 65c3c: os_xsave+0x1c xor %edx,%edx - already visited
os_xsave: validation end

-----

Alexandre Chartre (13):
  objtool: Move disassembly functions to a separated file
  objtool: Create disassembly context
  objtool: Disassemble code with libopcodes instead of running objdump
  objtool: Print symbol during disassembly
  objtool: Store instruction disassembly result
  objtool: Disassemble instruction on warning or backtrace
  objtool: Extract code to validate instruction from the validate branch
    loop
  objtool: Record symbol name max length
  objtool: Add option to trace function validation
  objtool: Trace instruction state changes during function validation
  objtool: Improve register reporting during function validation
  objtool: Improve tracing of alternative instructions
  objtool: Do not validate IBT for .return_sites and .call_sites

 tools/objtool/Build                     |   1 +
 tools/objtool/Makefile                  |   2 +-
 tools/objtool/arch/loongarch/decode.c   |  17 +
 tools/objtool/arch/powerpc/decode.c     |  18 +
 tools/objtool/arch/x86/decode.c         |  15 +
 tools/objtool/builtin-check.c           |   1 +
 tools/objtool/check.c                   | 829 ++++++++++++++++--------
 tools/objtool/disas.c                   | 452 +++++++++++++
 tools/objtool/include/objtool/arch.h    |   7 +
 tools/objtool/include/objtool/builtin.h |   1 +
 tools/objtool/include/objtool/check.h   |  29 +
 tools/objtool/include/objtool/warn.h    |  17 +-
 12 files changed, 1122 insertions(+), 267 deletions(-)
 create mode 100644 tools/objtool/disas.c

-- 
2.43.5


^ permalink raw reply	[flat|nested] 35+ messages in thread
* Re: [RFC 03/13] objtool: Disassemble code with libopcodes instead of running objdump
@ 2025-06-06 17:21 kernel test robot
  0 siblings, 0 replies; 35+ messages in thread
From: kernel test robot @ 2025-06-06 17:21 UTC (permalink / raw)
  To: oe-kbuild; +Cc: lkp

:::::: 
:::::: Manual check reason: "only suspicious fbc files changed"
:::::: 

BCC: lkp@intel.com
CC: oe-kbuild-all@lists.linux.dev
In-Reply-To: <20250606153440.865808-4-alexandre.chartre@oracle.com>
References: <20250606153440.865808-4-alexandre.chartre@oracle.com>
TO: Alexandre Chartre <alexandre.chartre@oracle.com>

Hi Alexandre,

[This is a private test report for your RFC patch.]
kernel test robot noticed the following build errors:

[auto build test ERROR on linus/master]
[also build test ERROR on v6.15 next-20250606]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Alexandre-Chartre/objtool-Move-disassembly-functions-to-a-separated-file/20250606-233948
base:   linus/master
patch link:    https://lore.kernel.org/r/20250606153440.865808-4-alexandre.chartre%40oracle.com
patch subject: [RFC 03/13] objtool: Disassemble code with libopcodes instead of running objdump
:::::: branch date: 2 hours ago
:::::: commit date: 2 hours ago
reproduce: (https://download.01.org/0day-ci/archive/20250607/202506070147.yRx7HyNf-lkp@intel.com/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/r/202506070147.yRx7HyNf-lkp@intel.com/

All errors (new ones prefixed by >>):

   1 error generated.
   make[2]: *** [scripts/Makefile.build:98: scripts/mod/devicetable-offsets.s] Error 1
   make[2]: Target 'scripts/mod/' not remade because of errors.
   make[1]: *** [Makefile:1278: prepare0] Error 2
   In file included from disas.c:12:
>> tools/include/tools/dis-asm-compat.h:10:6: error: redefinition of 'disassembler_style'
      10 | enum disassembler_style {DISASSEMBLER_STYLE_NOT_EMPTY};
         |      ^
   /usr/include/dis-asm.h:53:6: note: previous definition is here
      53 | enum disassembler_style
         |      ^
   In file included from disas.c:12:
>> tools/include/tools/dis-asm-compat.h:51:23: error: too few arguments to function call, expected 4, have 3
      50 |         init_disassemble_info(info, stream,
         |         ~~~~~~~~~~~~~~~~~~~~~
      51 |                               unstyled_func);
         |                                            ^
   /usr/include/dis-asm.h:480:13: note: 'init_disassemble_info' declared here


vim +/disassembler_style +10 tools/include/tools/dis-asm-compat.h

a45b3d6926231c Andres Freund 2022-07-31   7  
a45b3d6926231c Andres Freund 2022-07-31   8  /* define types for older binutils version, to centralize ifdef'ery a bit */
a45b3d6926231c Andres Freund 2022-07-31   9  #ifndef DISASM_INIT_STYLED
a45b3d6926231c Andres Freund 2022-07-31 @10  enum disassembler_style {DISASSEMBLER_STYLE_NOT_EMPTY};
a45b3d6926231c Andres Freund 2022-07-31  11  typedef int (*fprintf_styled_ftype) (void *, enum disassembler_style, const char*, ...);
a45b3d6926231c Andres Freund 2022-07-31  12  #endif
a45b3d6926231c Andres Freund 2022-07-31  13  
a45b3d6926231c Andres Freund 2022-07-31  14  /*
a45b3d6926231c Andres Freund 2022-07-31  15   * Trivial fprintf wrapper to be used as the fprintf_styled_func argument to
a45b3d6926231c Andres Freund 2022-07-31  16   * init_disassemble_info_compat() when normal fprintf suffices.
a45b3d6926231c Andres Freund 2022-07-31  17   */
a45b3d6926231c Andres Freund 2022-07-31  18  static inline int fprintf_styled(void *out,
a45b3d6926231c Andres Freund 2022-07-31  19  				 enum disassembler_style style,
a45b3d6926231c Andres Freund 2022-07-31  20  				 const char *fmt, ...)
a45b3d6926231c Andres Freund 2022-07-31  21  {
a45b3d6926231c Andres Freund 2022-07-31  22  	va_list args;
a45b3d6926231c Andres Freund 2022-07-31  23  	int r;
a45b3d6926231c Andres Freund 2022-07-31  24  
a45b3d6926231c Andres Freund 2022-07-31  25  	(void)style;
a45b3d6926231c Andres Freund 2022-07-31  26  
a45b3d6926231c Andres Freund 2022-07-31  27  	va_start(args, fmt);
a45b3d6926231c Andres Freund 2022-07-31  28  	r = vfprintf(out, fmt, args);
a45b3d6926231c Andres Freund 2022-07-31  29  	va_end(args);
a45b3d6926231c Andres Freund 2022-07-31  30  
a45b3d6926231c Andres Freund 2022-07-31  31  	return r;
a45b3d6926231c Andres Freund 2022-07-31  32  }
a45b3d6926231c Andres Freund 2022-07-31  33  
a45b3d6926231c Andres Freund 2022-07-31  34  /*
a45b3d6926231c Andres Freund 2022-07-31  35   * Wrapper for init_disassemble_info() that hides version
a45b3d6926231c Andres Freund 2022-07-31  36   * differences. Depending on binutils version and architecture either
a45b3d6926231c Andres Freund 2022-07-31  37   * fprintf_func or fprintf_styled_func will be called.
a45b3d6926231c Andres Freund 2022-07-31  38   */
a45b3d6926231c Andres Freund 2022-07-31  39  static inline void init_disassemble_info_compat(struct disassemble_info *info,
a45b3d6926231c Andres Freund 2022-07-31  40  						void *stream,
a45b3d6926231c Andres Freund 2022-07-31  41  						fprintf_ftype unstyled_func,
a45b3d6926231c Andres Freund 2022-07-31  42  						fprintf_styled_ftype styled_func)
a45b3d6926231c Andres Freund 2022-07-31  43  {
a45b3d6926231c Andres Freund 2022-07-31  44  #ifdef DISASM_INIT_STYLED
a45b3d6926231c Andres Freund 2022-07-31  45  	init_disassemble_info(info, stream,
a45b3d6926231c Andres Freund 2022-07-31  46  			      unstyled_func,
a45b3d6926231c Andres Freund 2022-07-31  47  			      styled_func);
a45b3d6926231c Andres Freund 2022-07-31  48  #else
a45b3d6926231c Andres Freund 2022-07-31  49  	(void)styled_func;
a45b3d6926231c Andres Freund 2022-07-31  50  	init_disassemble_info(info, stream,
a45b3d6926231c Andres Freund 2022-07-31 @51  			      unstyled_func);
a45b3d6926231c Andres Freund 2022-07-31  52  #endif
a45b3d6926231c Andres Freund 2022-07-31  53  }
a45b3d6926231c Andres Freund 2022-07-31  54  

-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

^ permalink raw reply	[flat|nested] 35+ messages in thread
* Re: [RFC 03/13] objtool: Disassemble code with libopcodes instead of running objdump
@ 2025-06-07  7:20 kernel test robot
  0 siblings, 0 replies; 35+ messages in thread
From: kernel test robot @ 2025-06-07  7:20 UTC (permalink / raw)
  To: oe-kbuild; +Cc: lkp

:::::: 
:::::: Manual check reason: "only suspicious fbc files changed"
:::::: 

BCC: lkp@intel.com
CC: oe-kbuild-all@lists.linux.dev
In-Reply-To: <20250606153440.865808-4-alexandre.chartre@oracle.com>
References: <20250606153440.865808-4-alexandre.chartre@oracle.com>
TO: Alexandre Chartre <alexandre.chartre@oracle.com>

Hi Alexandre,

[This is a private test report for your RFC patch.]
kernel test robot noticed the following build errors:

[auto build test ERROR on linus/master]
[also build test ERROR on v6.15 next-20250606]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Alexandre-Chartre/objtool-Move-disassembly-functions-to-a-separated-file/20250606-233948
base:   linus/master
patch link:    https://lore.kernel.org/r/20250606153440.865808-4-alexandre.chartre%40oracle.com
patch subject: [RFC 03/13] objtool: Disassemble code with libopcodes instead of running objdump
:::::: branch date: 15 hours ago
:::::: commit date: 15 hours ago
config: powerpc-allnoconfig (https://download.01.org/0day-ci/archive/20250607/202506071438.NpC5O1aA-lkp@intel.com/config)
compiler: powerpc-linux-gcc (GCC) 15.1.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250607/202506071438.NpC5O1aA-lkp@intel.com/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/r/202506071438.NpC5O1aA-lkp@intel.com/

All error/warnings (new ones prefixed by >>):

   arch/powerpc/decode.c: In function 'arch_reloc_size':
>> arch/powerpc/decode.c:131:5: warning: declaration of 'arch_disas_info_init' shadows a global declaration [-Wshadow]
     131 | int arch_disas_info_init(struct disassemble_info *dinfo)
         |     ^~~~~~~~~~~~~~~~~~~~
   In file included from tools/objtool/include/objtool/check.h:11,
                    from arch/powerpc/decode.c:5:
   tools/objtool/include/objtool/arch.h:107:5: note: shadowed declaration is here
     107 | int arch_disas_info_init(struct disassemble_info *dinfo);
         |     ^~~~~~~~~~~~~~~~~~~~
>> arch/powerpc/decode.c:131:1: warning: ISO C90 forbids mixed declarations and code [-Wdeclaration-after-statement]
     131 | int arch_disas_info_init(struct disassemble_info *dinfo)
         | ^~~
>> arch/powerpc/decode.c:136:1: error: expected declaration or statement at end of input
     136 | }
         | ^
   arch/powerpc/decode.c: At top level:
>> arch/powerpc/decode.c:131:5: warning: 'arch_disas_info_init' defined but not used [-Wunused-function]
     131 | int arch_disas_info_init(struct disassemble_info *dinfo)
         |     ^~~~~~~~~~~~~~~~~~~~
   make[6]: *** [tools/build/Makefile.build:86: tools/objtool/arch/powerpc/decode.o] Error 1
   In file included from disas.c:12:
   tools/include/tools/dis-asm-compat.h:10:6: error: redeclaration of 'enum disassembler_style'
      10 | enum disassembler_style {DISASSEMBLER_STYLE_NOT_EMPTY};
         |      ^~~~~~~~~~~~~~~~~~
   In file included from tools/objtool/include/objtool/arch.h:10,
                    from disas.c:6:
   /usr/include/dis-asm.h:53:6: note: originally defined here
      53 | enum disassembler_style
         |      ^~~~~~~~~~~~~~~~~~
   tools/include/tools/dis-asm-compat.h: In function 'init_disassemble_info_compat':
   tools/include/tools/dis-asm-compat.h:50:9: error: too few arguments to function 'init_disassemble_info'
      50 |         init_disassemble_info(info, stream,
         |         ^~~~~~~~~~~~~~~~~~~~~
   /usr/include/dis-asm.h:480:13: note: declared here
     480 | extern void init_disassemble_info (struct disassemble_info *dinfo, void *stream,
         |             ^~~~~~~~~~~~~~~~~~~~~
   make[5]: *** [tools/build/Makefile.build:86: tools/objtool/disas.o] Error 1
   make[6]: Target '__build' not remade because of errors.
   make[5]: *** [tools/build/Makefile.build:142: arch/powerpc] Error 2
   make[5]: Target '__build' not remade because of errors.
   make[4]: *** [Makefile:65: tools/objtool/objtool-in.o] Error 2
   make[4]: Target 'all' not remade because of errors.
   make[3]: *** [Makefile:73: objtool] Error 2
   make[2]: *** [Makefile:1448: tools/objtool] Error 2
   make[2]: Target 'prepare' not remade because of errors.
   make[1]: *** [Makefile:248: __sub-make] Error 2
   make[1]: Target 'prepare' not remade because of errors.
   make: *** [Makefile:248: __sub-make] Error 2
   make: Target 'prepare' not remade because of errors.

-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

^ permalink raw reply	[flat|nested] 35+ messages in thread

end of thread, other threads:[~2025-06-11 19:25 UTC | newest]

Thread overview: 35+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-06-06 15:34 [RFC 00/13] objtool: Function validation tracing Alexandre Chartre
2025-06-06 15:34 ` [RFC 01/13] objtool: Move disassembly functions to a separated file Alexandre Chartre
2025-06-06 15:34 ` [RFC 02/13] objtool: Create disassembly context Alexandre Chartre
2025-06-10 21:12   ` Josh Poimboeuf
2025-06-06 15:34 ` [RFC 03/13] objtool: Disassemble code with libopcodes instead of running objdump Alexandre Chartre
2025-06-07  0:45   ` kernel test robot
2025-06-10 21:22   ` Josh Poimboeuf
2025-06-11 12:23   ` Peter Zijlstra
2025-06-11 13:35     ` Alexandre Chartre
2025-06-11 19:25       ` Josh Poimboeuf
2025-06-06 15:34 ` [RFC 04/13] objtool: Print symbol during disassembly Alexandre Chartre
2025-06-10 21:55   ` Josh Poimboeuf
2025-06-06 15:34 ` [RFC 05/13] objtool: Store instruction disassembly result Alexandre Chartre
2025-06-10 22:40   ` Josh Poimboeuf
2025-06-06 15:34 ` [RFC 06/13] objtool: Disassemble instruction on warning or backtrace Alexandre Chartre
2025-06-06 15:34 ` [RFC 07/13] objtool: Extract code to validate instruction from the validate branch loop Alexandre Chartre
2025-06-10 23:31   ` Josh Poimboeuf
2025-06-06 15:34 ` [RFC 08/13] objtool: Record symbol name max length Alexandre Chartre
2025-06-06 15:34 ` [RFC 09/13] objtool: Add option to trace function validation Alexandre Chartre
2025-06-11  0:48   ` Josh Poimboeuf
2025-06-06 15:34 ` [RFC 10/13] objtool: Trace instruction state changes during " Alexandre Chartre
2025-06-11  1:23   ` Josh Poimboeuf
2025-06-06 15:34 ` [RFC 11/13] objtool: Improve register reporting " Alexandre Chartre
2025-06-06 15:34 ` [RFC 12/13] objtool: Improve tracing of alternative instructions Alexandre Chartre
2025-06-06 15:34 ` [RFC 13/13] objtool: Do not validate IBT for .return_sites and .call_sites Alexandre Chartre
2025-06-06 15:58 ` [RFC 00/13] objtool: Function validation tracing Josh Poimboeuf
2025-06-06 19:29   ` Alexandre Chartre
2025-06-09 18:31 ` Josh Poimboeuf
2025-06-10  7:07   ` Alexandre Chartre
2025-06-10 13:00     ` Alexandre Chartre
2025-06-10 21:05       ` Josh Poimboeuf
2025-06-11  6:00         ` Alexandre Chartre
2025-06-11 14:20           ` Peter Zijlstra
  -- strict thread matches above, loose matches on Subject: below --
2025-06-06 17:21 [RFC 03/13] objtool: Disassemble code with libopcodes instead of running objdump kernel test robot
2025-06-07  7:20 kernel test robot

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.