stable.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
To: linux-kernel@vger.kernel.org
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
	stable@vger.kernel.org, Chang Rui <changruinj@gmail.com>,
	Fangrui Song <maskray@google.com>, Leo Yan <leo.yan@linaro.org>,
	Namhyung Kim <namhyung@kernel.org>,
	Alexander Shishkin <alexander.shishkin@linux.intel.com>,
	Ian Rogers <irogers@google.com>, Ingo Molnar <mingo@redhat.com>,
	Jiri Olsa <jolsa@kernel.org>, Mark Rutland <mark.rutland@arm.com>,
	Peter Zijlstra <peterz@infradead.org>,
	Arnaldo Carvalho de Melo <acme@redhat.com>,
	Sasha Levin <sashal@kernel.org>
Subject: [PATCH 5.18 74/88] perf symbol: Correct address for bss symbols
Date: Mon,  1 Aug 2022 13:47:28 +0200	[thread overview]
Message-ID: <20220801114141.409143282@linuxfoundation.org> (raw)
In-Reply-To: <20220801114138.041018499@linuxfoundation.org>

From: Leo Yan <leo.yan@linaro.org>

[ Upstream commit 2d86612aacb7805f72873691a2644d7279ed0630 ]

When using 'perf mem' and 'perf c2c', an issue is observed that tool
reports the wrong offset for global data symbols.  This is a common
issue on both x86 and Arm64 platforms.

Let's see an example, for a test program, below is the disassembly for
its .bss section which is dumped with objdump:

  ...

  Disassembly of section .bss:

  0000000000004040 <completed.0>:
  	...

  0000000000004080 <buf1>:
  	...

  00000000000040c0 <buf2>:
  	...

  0000000000004100 <thread>:
  	...

First we used 'perf mem record' to run the test program and then used
'perf --debug verbose=4 mem report' to observe what's the symbol info
for 'buf1' and 'buf2' structures.

  # ./perf mem record -e ldlat-loads,ldlat-stores -- false_sharing.exe 8
  # ./perf --debug verbose=4 mem report
    ...
    dso__load_sym_internal: adjusting symbol: st_value: 0x40c0 sh_addr: 0x4040 sh_offset: 0x3028
    symbol__new: buf2 0x30a8-0x30e8
    ...
    dso__load_sym_internal: adjusting symbol: st_value: 0x4080 sh_addr: 0x4040 sh_offset: 0x3028
    symbol__new: buf1 0x3068-0x30a8
    ...

The perf tool relies on libelf to parse symbols, in executable and
shared object files, 'st_value' holds a virtual address; 'sh_addr' is
the address at which section's first byte should reside in memory, and
'sh_offset' is the byte offset from the beginning of the file to the
first byte in the section.  The perf tool uses below formula to convert
a symbol's memory address to a file address:

  file_address = st_value - sh_addr + sh_offset
                    ^
                    ` Memory address

We can see the final adjusted address ranges for buf1 and buf2 are
[0x30a8-0x30e8) and [0x3068-0x30a8) respectively, apparently this is
incorrect, in the code, the structure for 'buf1' and 'buf2' specifies
compiler attribute with 64-byte alignment.

The problem happens for 'sh_offset', libelf returns it as 0x3028 which
is not 64-byte aligned, combining with disassembly, it's likely libelf
doesn't respect the alignment for .bss section, therefore, it doesn't
return the aligned value for 'sh_offset'.

Suggested by Fangrui Song, ELF file contains program header which
contains PT_LOAD segments, the fields p_vaddr and p_offset in PT_LOAD
segments contain the execution info.  A better choice for converting
memory address to file address is using the formula:

  file_address = st_value - p_vaddr + p_offset

This patch introduces elf_read_program_header() which returns the
program header based on the passed 'st_value', then it uses the formula
above to calculate the symbol file address; and the debugging log is
updated respectively.

After applying the change:

  # ./perf --debug verbose=4 mem report
    ...
    dso__load_sym_internal: adjusting symbol: st_value: 0x40c0 p_vaddr: 0x3d28 p_offset: 0x2d28
    symbol__new: buf2 0x30c0-0x3100
    ...
    dso__load_sym_internal: adjusting symbol: st_value: 0x4080 p_vaddr: 0x3d28 p_offset: 0x2d28
    symbol__new: buf1 0x3080-0x30c0
    ...

Fixes: f17e04afaff84b5c ("perf report: Fix ELF symbol parsing")
Reported-by: Chang Rui <changruinj@gmail.com>
Suggested-by: Fangrui Song <maskray@google.com>
Signed-off-by: Leo Yan <leo.yan@linaro.org>
Acked-by: Namhyung Kim <namhyung@kernel.org>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Ian Rogers <irogers@google.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: https://lore.kernel.org/r/20220724060013.171050-2-leo.yan@linaro.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Signed-off-by: Sasha Levin <sashal@kernel.org>
---
 tools/perf/util/symbol-elf.c | 45 ++++++++++++++++++++++++++++++++----
 1 file changed, 41 insertions(+), 4 deletions(-)

diff --git a/tools/perf/util/symbol-elf.c b/tools/perf/util/symbol-elf.c
index ecd377938eea..ef6ced5c5746 100644
--- a/tools/perf/util/symbol-elf.c
+++ b/tools/perf/util/symbol-elf.c
@@ -233,6 +233,33 @@ Elf_Scn *elf_section_by_name(Elf *elf, GElf_Ehdr *ep,
 	return NULL;
 }
 
+static int elf_read_program_header(Elf *elf, u64 vaddr, GElf_Phdr *phdr)
+{
+	size_t i, phdrnum;
+	u64 sz;
+
+	if (elf_getphdrnum(elf, &phdrnum))
+		return -1;
+
+	for (i = 0; i < phdrnum; i++) {
+		if (gelf_getphdr(elf, i, phdr) == NULL)
+			return -1;
+
+		if (phdr->p_type != PT_LOAD)
+			continue;
+
+		sz = max(phdr->p_memsz, phdr->p_filesz);
+		if (!sz)
+			continue;
+
+		if (vaddr >= phdr->p_vaddr && (vaddr < phdr->p_vaddr + sz))
+			return 0;
+	}
+
+	/* Not found any valid program header */
+	return -1;
+}
+
 static bool want_demangle(bool is_kernel_sym)
 {
 	return is_kernel_sym ? symbol_conf.demangle_kernel : symbol_conf.demangle;
@@ -1209,6 +1236,7 @@ dso__load_sym_internal(struct dso *dso, struct map *map, struct symsrc *syms_ss,
 					sym.st_value);
 			used_opd = true;
 		}
+
 		/*
 		 * When loading symbols in a data mapping, ABS symbols (which
 		 * has a value of SHN_ABS in its st_shndx) failed at
@@ -1262,11 +1290,20 @@ dso__load_sym_internal(struct dso *dso, struct map *map, struct symsrc *syms_ss,
 				goto out_elf_end;
 		} else if ((used_opd && runtime_ss->adjust_symbols) ||
 			   (!used_opd && syms_ss->adjust_symbols)) {
+			GElf_Phdr phdr;
+
+			if (elf_read_program_header(syms_ss->elf,
+						    (u64)sym.st_value, &phdr)) {
+				pr_warning("%s: failed to find program header for "
+					   "symbol: %s st_value: %#" PRIx64 "\n",
+					   __func__, elf_name, (u64)sym.st_value);
+				continue;
+			}
 			pr_debug4("%s: adjusting symbol: st_value: %#" PRIx64 " "
-				  "sh_addr: %#" PRIx64 " sh_offset: %#" PRIx64 "\n", __func__,
-				  (u64)sym.st_value, (u64)shdr.sh_addr,
-				  (u64)shdr.sh_offset);
-			sym.st_value -= shdr.sh_addr - shdr.sh_offset;
+				  "p_vaddr: %#" PRIx64 " p_offset: %#" PRIx64 "\n",
+				  __func__, (u64)sym.st_value, (u64)phdr.p_vaddr,
+				  (u64)phdr.p_offset);
+			sym.st_value -= phdr.p_vaddr - phdr.p_offset;
 		}
 
 		demangled = demangle_sym(dso, kmodule, elf_name);
-- 
2.35.1




  parent reply	other threads:[~2022-08-01 12:17 UTC|newest]

Thread overview: 102+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-08-01 11:46 [PATCH 5.18 00/88] 5.18.16-rc1 review Greg Kroah-Hartman
2022-08-01 11:46 ` [PATCH 5.18 01/88] Bluetooth: Always set event mask on suspend Greg Kroah-Hartman
2022-08-01 11:46 ` [PATCH 5.18 02/88] Bluetooth: L2CAP: Fix use-after-free caused by l2cap_chan_put Greg Kroah-Hartman
2022-08-01 11:46 ` [PATCH 5.18 03/88] ARM: dts: lan966x: fix sys_clk frequency Greg Kroah-Hartman
2022-08-01 11:46 ` [PATCH 5.18 04/88] ARM: pxa2xx: Fix GPIO descriptor tables Greg Kroah-Hartman
2022-08-01 11:46 ` [PATCH 5.18 05/88] Revert "ocfs2: mount shared volume without ha stack" Greg Kroah-Hartman
2022-08-01 11:46 ` [PATCH 5.18 06/88] userfaultfd: provide properly masked address for huge-pages Greg Kroah-Hartman
2022-08-01 11:46 ` [PATCH 5.18 07/88] ntfs: fix use-after-free in ntfs_ucsncmp() Greg Kroah-Hartman
2022-08-01 11:46 ` [PATCH 5.18 08/88] fs: sendfile handles O_NONBLOCK of out_fd Greg Kroah-Hartman
2022-08-01 11:46 ` [PATCH 5.18 09/88] secretmem: fix unhandled fault in truncate Greg Kroah-Hartman
2022-08-01 11:46 ` [PATCH 5.18 10/88] mm: fix page leak with multiple threads mapping the same page Greg Kroah-Hartman
2022-08-01 11:46 ` [PATCH 5.18 11/88] mm: fix missing wake-up event for FSDAX pages Greg Kroah-Hartman
2022-08-01 11:46 ` [PATCH 5.18 12/88] hugetlb: fix memoryleak in hugetlb_mcopy_atomic_pte Greg Kroah-Hartman
2022-08-01 11:46 ` [PATCH 5.18 13/88] asm-generic: remove a broken and needless ifdef conditional Greg Kroah-Hartman
2022-08-01 11:46 ` [PATCH 5.18 14/88] s390/archrandom: prevent CPACF trng invocations in interrupt context Greg Kroah-Hartman
2022-08-01 11:46 ` [PATCH 5.18 15/88] intel_idle: Fix false positive RCU splats due to incorrect hardirqs state Greg Kroah-Hartman
2022-08-01 11:46 ` [PATCH 5.18 16/88] nouveau/svm: Fix to migrate all requested pages Greg Kroah-Hartman
2022-08-01 11:46 ` [PATCH 5.18 17/88] drm/simpledrm: Fix return type of simpledrm_simple_display_pipe_mode_valid() Greg Kroah-Hartman
2022-08-01 11:46 ` [PATCH 5.18 18/88] watch_queue: Fix missing rcu annotation Greg Kroah-Hartman
2022-08-01 11:46 ` [PATCH 5.18 19/88] watch_queue: Fix missing locking in add_watch_to_object() Greg Kroah-Hartman
2022-08-01 11:46 ` [PATCH 5.18 20/88] tcp: Fix data-races around sysctl_tcp_dsack Greg Kroah-Hartman
2022-08-01 11:46 ` [PATCH 5.18 21/88] tcp: Fix a data-race around sysctl_tcp_app_win Greg Kroah-Hartman
2022-08-01 11:46 ` [PATCH 5.18 22/88] tcp: Fix a data-race around sysctl_tcp_adv_win_scale Greg Kroah-Hartman
2022-08-01 11:46 ` [PATCH 5.18 23/88] tcp: Fix a data-race around sysctl_tcp_frto Greg Kroah-Hartman
2022-08-01 11:46 ` [PATCH 5.18 24/88] tcp: Fix a data-race around sysctl_tcp_nometrics_save Greg Kroah-Hartman
2022-08-01 11:46 ` [PATCH 5.18 25/88] tcp: Fix data-races around sysctl_tcp_no_ssthresh_metrics_save Greg Kroah-Hartman
2022-08-01 11:46 ` [PATCH 5.18 26/88] bridge: Do not send empty IFLA_AF_SPEC attribute Greg Kroah-Hartman
2022-08-01 11:46 ` [PATCH 5.18 27/88] ice: Fix max VLANs available for VF Greg Kroah-Hartman
2022-08-01 11:46 ` [PATCH 5.18 28/88] ice: check (DD | EOF) bits on Rx descriptor rather than (EOP | RS) Greg Kroah-Hartman
2022-08-01 11:46 ` [PATCH 5.18 29/88] ice: do not setup vlan for loopback VSI Greg Kroah-Hartman
2022-08-01 11:46 ` [PATCH 5.18 30/88] ice: Fix VSIs unable to share unicast MAC Greg Kroah-Hartman
2022-08-01 11:46 ` [PATCH 5.18 31/88] scsi: ufs: host: Hold reference returned by of_parse_phandle() Greg Kroah-Hartman
2022-08-01 11:46 ` [PATCH 5.18 32/88] Revert "tcp: change pingpong threshold to 3" Greg Kroah-Hartman
2022-08-01 11:46 ` [PATCH 5.18 33/88] octeontx2-pf: Fix UDP/TCP src and dst port tc filters Greg Kroah-Hartman
2022-08-01 11:46 ` [PATCH 5.18 34/88] tcp: md5: fix IPv4-mapped support Greg Kroah-Hartman
2022-08-01 11:46 ` [PATCH 5.18 35/88] tcp: Fix data-races around sysctl_tcp_moderate_rcvbuf Greg Kroah-Hartman
2022-08-01 11:46 ` [PATCH 5.18 36/88] tcp: Fix a data-race around sysctl_tcp_limit_output_bytes Greg Kroah-Hartman
2022-08-01 11:46 ` [PATCH 5.18 37/88] tcp: Fix a data-race around sysctl_tcp_challenge_ack_limit Greg Kroah-Hartman
2022-08-01 11:46 ` [PATCH 5.18 38/88] scsi: core: Fix warning in scsi_alloc_sgtables() Greg Kroah-Hartman
2022-08-01 11:46 ` [PATCH 5.18 39/88] scsi: mpt3sas: Stop fw fault watchdog work item during system shutdown Greg Kroah-Hartman
2022-08-01 11:46 ` [PATCH 5.18 40/88] net: ping6: Fix memleak in ipv6_renew_options() Greg Kroah-Hartman
2022-08-01 11:46 ` [PATCH 5.18 41/88] ipv6/addrconf: fix a null-ptr-deref bug for ip6_ptr Greg Kroah-Hartman
2022-08-01 11:46 ` [PATCH 5.18 42/88] net/tls: Remove the context from the list in tls_device_down Greg Kroah-Hartman
2022-08-01 11:46 ` [PATCH 5.18 43/88] net: pcs: xpcs: propagate xpcs_read error to xpcs_get_state_c37_sgmii Greg Kroah-Hartman
2022-08-01 11:46 ` [PATCH 5.18 44/88] net: sungem_phy: Add of_node_put() for reference returned by of_get_parent() Greg Kroah-Hartman
2022-08-01 11:46 ` [PATCH 5.18 45/88] mlxsw: spectrum_router: simplify list unwinding Greg Kroah-Hartman
2022-08-01 11:47 ` [PATCH 5.18 46/88] tcp: Fix a data-race around sysctl_tcp_min_tso_segs Greg Kroah-Hartman
2022-08-01 11:47 ` [PATCH 5.18 47/88] tcp: Fix a data-race around sysctl_tcp_tso_rtt_log Greg Kroah-Hartman
2022-08-01 11:47 ` [PATCH 5.18 48/88] tcp: Fix a data-race around sysctl_tcp_min_rtt_wlen Greg Kroah-Hartman
2022-08-01 11:47 ` [PATCH 5.18 49/88] tcp: Fix a data-race around sysctl_tcp_autocorking Greg Kroah-Hartman
2022-08-01 11:47 ` [PATCH 5.18 50/88] tcp: Fix a data-race around sysctl_tcp_invalid_ratelimit Greg Kroah-Hartman
2022-08-01 11:47 ` [PATCH 5.18 51/88] Documentation: fix sctp_wmem in ip-sysctl.rst Greg Kroah-Hartman
2022-08-01 11:47 ` [PATCH 5.18 52/88] macsec: fix NULL deref in macsec_add_rxsa Greg Kroah-Hartman
2022-08-01 11:47 ` [PATCH 5.18 53/88] macsec: fix error message in macsec_add_rxsa and _txsa Greg Kroah-Hartman
2022-08-01 11:47 ` [PATCH 5.18 54/88] macsec: limit replay window size with XPN Greg Kroah-Hartman
2022-08-01 11:47 ` [PATCH 5.18 55/88] macsec: always read MACSEC_SA_ATTR_PN as a u64 Greg Kroah-Hartman
2022-08-01 11:47 ` [PATCH 5.18 56/88] net: macsec: fix potential resource leak in macsec_add_rxsa() and macsec_add_txsa() Greg Kroah-Hartman
2022-08-01 11:47 ` [PATCH 5.18 57/88] net: mld: fix reference count leak in mld_{query | report}_work() Greg Kroah-Hartman
2022-08-01 11:47 ` [PATCH 5.18 58/88] tcp: Fix data-races around sk_pacing_rate Greg Kroah-Hartman
2022-08-01 11:47 ` [PATCH 5.18 59/88] net: Fix data-races around sysctl_[rw]mem(_offset)? Greg Kroah-Hartman
2022-08-01 11:47 ` [PATCH 5.18 60/88] tcp: Fix a data-race around sysctl_tcp_comp_sack_delay_ns Greg Kroah-Hartman
2022-08-01 11:47 ` [PATCH 5.18 61/88] tcp: Fix a data-race around sysctl_tcp_comp_sack_slack_ns Greg Kroah-Hartman
2022-08-01 11:47 ` [PATCH 5.18 62/88] tcp: Fix a data-race around sysctl_tcp_comp_sack_nr Greg Kroah-Hartman
2022-08-01 11:47 ` [PATCH 5.18 63/88] tcp: Fix data-races around sysctl_tcp_reflect_tos Greg Kroah-Hartman
2022-08-01 11:47 ` [PATCH 5.18 64/88] ipv4: Fix data-races around sysctl_fib_notify_on_flag_change Greg Kroah-Hartman
2022-08-01 11:47 ` [PATCH 5.18 65/88] i40e: Fix interface init with MSI interrupts (no MSI-X) Greg Kroah-Hartman
2022-08-01 11:47 ` [PATCH 5.18 66/88] net: dsa: fix reference counting for LAG FDBs Greg Kroah-Hartman
2022-08-01 11:47 ` [PATCH 5.18 67/88] sctp: fix sleep in atomic context bug in timer handlers Greg Kroah-Hartman
2022-08-01 11:47 ` [PATCH 5.18 68/88] octeontx2-pf: cn10k: Fix egress ratelimit configuration Greg Kroah-Hartman
2022-08-01 11:47 ` [PATCH 5.18 69/88] netfilter: nf_queue: do not allow packet truncation below transport header offset Greg Kroah-Hartman
2022-08-01 11:47 ` [PATCH 5.18 70/88] scsi: ufs: Support clearing multiple commands at once Greg Kroah-Hartman
2022-08-01 11:47 ` [PATCH 5.18 71/88] scsi: ufs: core: Fix a race condition related to device management Greg Kroah-Hartman
2022-08-01 11:47 ` [PATCH 5.18 72/88] mptcp: dont send RST for single subflow Greg Kroah-Hartman
2022-08-02 20:52   ` Mat Martineau
2022-08-02 21:21     ` Sasha Levin
2022-08-03  6:56       ` Greg Kroah-Hartman
2022-08-01 11:47 ` [PATCH 5.18 73/88] virtio-net: fix the race between refill work and close Greg Kroah-Hartman
2022-08-01 11:47 ` Greg Kroah-Hartman [this message]
2022-08-01 11:47 ` [PATCH 5.18 75/88] sfc: disable softirqs for ptp TX Greg Kroah-Hartman
2022-08-01 11:47 ` [PATCH 5.18 76/88] sctp: leave the err path free in sctp_stream_init to sctp_stream_free Greg Kroah-Hartman
2022-08-01 11:47 ` [PATCH 5.18 77/88] net/funeth: Fix fun_xdp_tx() and XDP packet reclaim Greg Kroah-Hartman
2022-08-01 11:47 ` [PATCH 5.18 78/88] stmmac: dwmac-mediatek: fix resource leak in probe Greg Kroah-Hartman
2022-08-01 11:47 ` [PATCH 5.18 79/88] mm/hmm: fault non-owner device private entries Greg Kroah-Hartman
2022-08-01 11:47 ` [PATCH 5.18 80/88] page_alloc: fix invalid watermark check on a negative value Greg Kroah-Hartman
2022-08-01 11:47 ` [PATCH 5.18 81/88] tcp: Fix data-races around sysctl_tcp_workaround_signed_windows Greg Kroah-Hartman
2022-08-01 11:47 ` [PATCH 5.18 82/88] ARM: 9216/1: Fix MAX_DMA_ADDRESS overflow Greg Kroah-Hartman
2022-08-01 11:47 ` [PATCH 5.18 83/88] EDAC/ghes: Set the DIMM label unconditionally Greg Kroah-Hartman
2022-08-01 11:47 ` [PATCH 5.18 84/88] EDAC/synopsys: Use the correct register to disable the error interrupt on v3 hw Greg Kroah-Hartman
2022-08-01 11:47 ` [PATCH 5.18 85/88] EDAC/synopsys: Re-enable the error interrupts " Greg Kroah-Hartman
2022-08-01 11:47 ` [PATCH 5.18 86/88] docs/kernel-parameters: Update descriptions for "mitigations=" param with retbleed Greg Kroah-Hartman
2022-08-01 11:47 ` [PATCH 5.18 87/88] locking/rwsem: Allow slowpath writer to ignore handoff bit if not set by first waiter Greg Kroah-Hartman
2022-08-01 11:47 ` [PATCH 5.18 88/88] x86/bugs: Do not enable IBPB at firmware entry when IBPB is not available Greg Kroah-Hartman
2022-08-01 14:26 ` [PATCH 5.18 00/88] 5.18.16-rc1 review Jon Hunter
2022-08-01 19:45 ` Daniel Díaz
2022-08-01 22:09 ` Florian Fainelli
2022-08-01 22:13 ` Shuah Khan
2022-08-01 22:45 ` Zan Aziz
2022-08-02  1:00 ` Ron Economos
2022-08-02  5:29 ` Guenter Roeck
2022-08-02 10:18 ` Justin Forbes
2022-08-02 12:10 ` Bagas Sanjaya
2022-08-02 17:42 ` Sudip Mukherjee (Codethink)

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20220801114141.409143282@linuxfoundation.org \
    --to=gregkh@linuxfoundation.org \
    --cc=acme@redhat.com \
    --cc=alexander.shishkin@linux.intel.com \
    --cc=changruinj@gmail.com \
    --cc=irogers@google.com \
    --cc=jolsa@kernel.org \
    --cc=leo.yan@linaro.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mark.rutland@arm.com \
    --cc=maskray@google.com \
    --cc=mingo@redhat.com \
    --cc=namhyung@kernel.org \
    --cc=peterz@infradead.org \
    --cc=sashal@kernel.org \
    --cc=stable@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).