From: Daniel J Blueman <daniel@numascale.com>
To: Myron Stowe <myron.stowe@gmail.com>
Cc: Borislav Petkov <bp@suse.de>,
Myron Stowe <myron.stowe@redhat.com>,
Bjorn Helgaas <bhelgaas@google.com>,
linux-pci <linux-pci@vger.kernel.org>,
Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>,
Aravind Gopalakrishnan <aravind.gopalakrishnan@amd.com>,
kim.naru@amd.com, Thomas Gleixner <tglx@linutronix.de>,
mingo@redhat.com, hpa@zytor.com, x86 <x86@kernel.org>,
sp@numascale.com,
"linux-acpi@vger.kernel.org" <linux-acpi@vger.kernel.org>,
LKML <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH v2 5/5] PCI: Remove redundant 'quirk_amd_nb_node'
Date: Tue, 22 Apr 2014 00:53:59 +0800 [thread overview]
Message-ID: <53554D27.1010803@numascale.com> (raw)
In-Reply-To: <CAL-B5D1_fns=mu7Cw6jeVHO0XPeKYakUrPv-JgO=ma1rjCoQvg@mail.gmail.com>
Hi Myron,
On 04/20/2014 09:44 PM, Myron Stowe wrote:
> On Sun, Apr 20, 2014 at 4:54 AM, Borislav Petkov <bp@suse.de> wrote:
>> On Fri, Apr 18, 2014 at 08:53:46PM -0600, Myron Stowe wrote:
>>> With the amd_bus.c updates to support additional AMD processors (11h, 12h,
>>> 14h 15h and 16h) 'quirk_amd_nb_node' seems to be redundant. This patch
>>> removes it.
>>>
>>> Signed-off-by: Myron Stowe <myron.stowe@redhat.com>
>>
>> Good idea. Has this been tested, though? There are a bunch of places
>> using dev_to_node() which this quirk corrects, especially 'local_cpus'
>> in a bunch of sysfs nodes in the PCI hierarchy /sys/devices/pci*.
>
> Borislav:
>
> Thanks for the review, suggestions, and comments.
>
> Just and FYI - I'll be gone until late in this coming week on a
> backpacking trip in remote Utah. I'll work your suggestions when I
> get back.
>
> With respect to this patch and testing. Yes, I did test and it worked
> for my situation. I would like to hear from Daniel as I'm not exactly
> sure what his last change was specific to.
>
> Daniel: with the previous patches in this series applied do you still
> need this quirk for your situation?
It's fine enough for us, since we'll always emit a PXM method for any
PCI host bridges (but roots), which gets correctly preserved with your
patches [1]. I don't have reason to suspect northbridges at bus 0,
devices 0x19 and later in each PCI domain having the wrong NUMA node is
going to cause any issues.
We could improve things by changing probe_pci_hostbridge to iterate all
PCI bus domains, but it's probably not a big deal to bail with one
warning, as we currently see multiple (which would become a lot on most
of our systems):
[ 3.155974] AMD-Bus: TOM: 00000000d8000000 aka 3456M
[ 3.160003]
[ 3.161496] AMD-Bus: WARNING: Failed to find root info for node 0x4,
link 0x0
[ 3.168625]
[ 3.170002]
[ 3.171492] AMD-Bus: WARNING: Failed to find root info for node 0x4,
link 0x0
[ 3.178614] AMD-Bus: WARNING: Failed to find root info for node 0x4,
link 0x0
[ 3.180003] AMD-Bus: TOM2: 0000001900000000 aka 102400M
Thanks,
Daniel
-- [1] (two server NumaConnect system)
# lspci
0000:00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890
Northbridge only dual slot (2x16) PCI-e GFX Hydra part (rev 02)
0000:00:0d.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890
PCI to PCI bridge (external gfx1 port B)
0000:00:11.0 SATA controller: Advanced Micro Devices, Inc. [AMD/ATI]
SB7x0/SB8x0/SB9x0 SATA Controller [IDE mode]
0000:00:12.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI]
SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
0000:00:12.1 USB controller: Advanced Micro Devices, Inc. [AMD/ATI]
SB7x0 USB OHCI1 Controller
0000:00:12.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI]
SB7x0/SB8x0/SB9x0 USB EHCI Controller
0000:00:13.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI]
SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
0000:00:13.1 USB controller: Advanced Micro Devices, Inc. [AMD/ATI]
SB7x0 USB OHCI1 Controller
0000:00:13.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI]
SB7x0/SB8x0/SB9x0 USB EHCI Controller
0000:00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 SMBus
Controller (rev 3d)
0000:00:14.1 IDE interface: Advanced Micro Devices, Inc. [AMD/ATI]
SB7x0/SB8x0/SB9x0 IDE Controller
0000:00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD/ATI]
SB7x0/SB8x0/SB9x0 LPC host controller
0000:00:14.4 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SBx00
PCI to PCI Bridge
0000:00:14.5 USB controller: Advanced Micro Devices, Inc. [AMD/ATI]
SB7x0/SB8x0/SB9x0 USB OHCI2 Controller
0000:00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h
Processor Function 0
0000:00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h
Processor Function 1
0000:00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h
Processor Function 2
0000:00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h
Processor Function 3
0000:00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h
Processor Function 4
0000:00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h
Processor Function 5
0000:00:19.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h
Processor Function 0
0000:00:19.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h
Processor Function 1
0000:00:19.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h
Processor Function 2
0000:00:19.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h
Processor Function 3
0000:00:19.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h
Processor Function 4
0000:00:19.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h
Processor Function 5
0000:00:1a.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h
Processor Function 0
0000:00:1a.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h
Processor Function 1
0000:00:1a.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h
Processor Function 2
0000:00:1a.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h
Processor Function 3
0000:00:1a.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h
Processor Function 4
0000:00:1a.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h
Processor Function 5
0000:00:1b.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h
Processor Function 0
0000:00:1b.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h
Processor Function 1
0000:00:1b.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h
Processor Function 2
0000:00:1b.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h
Processor Function 3
0000:00:1b.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h
Processor Function 4
0000:00:1b.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h
Processor Function 5
0000:00:1c.0 Host bridge: Numascale AS NumaChip N601 (rev 02)
0000:00:1c.1 Host bridge: Numascale AS NumaChip N602 (rev 02)
0000:01:04.0 VGA compatible controller: Matrox Electronics Systems Ltd.
MGA G200eW WPCM450 (rev 0a)
0000:02:00.0 Ethernet controller: Intel Corporation 82576 Gigabit
Network Connection (rev 01)
0000:02:00.1 Ethernet controller: Intel Corporation 82576 Gigabit
Network Connection (rev 01)
0001:00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890
Northbridge only dual slot (2x16) PCI-e GFX Hydra part (rev 02)
0001:00:0d.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890
PCI to PCI bridge (external gfx1 port B)
0001:00:11.0 SATA controller: Advanced Micro Devices, Inc. [AMD/ATI]
SB7x0/SB8x0/SB9x0 SATA Controller [IDE mode]
0001:00:14.0 ISA bridge: Advanced Micro Devices, Inc. [AMD/ATI] SBx00
SMBus Controller (rev 3d)
0001:00:14.4 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SBx00
PCI to PCI Bridge
0001:00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h
Processor Function 0
0001:00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h
Processor Function 1
0001:00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h
Processor Function 2
0001:00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h
Processor Function 3
0001:00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h
Processor Function 4
0001:00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h
Processor Function 5
0001:00:19.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h
Processor Function 0
0001:00:19.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h
Processor Function 1
0001:00:19.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h
Processor Function 2
0001:00:19.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h
Processor Function 3
0001:00:19.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h
Processor Function 4
0001:00:19.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h
Processor Function 5
0001:00:1a.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h
Processor Function 0
0001:00:1a.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h
Processor Function 1
0001:00:1a.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h
Processor Function 2
0001:00:1a.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h
Processor Function 3
0001:00:1a.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h
Processor Function 4
0001:00:1a.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h
Processor Function 5
0001:00:1b.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h
Processor Function 0
0001:00:1b.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h
Processor Function 1
0001:00:1b.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h
Processor Function 2
0001:00:1b.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h
Processor Function 3
0001:00:1b.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h
Processor Function 4
0001:00:1b.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h
Processor Function 5
0001:00:1c.0 Host bridge: Numascale AS NumaChip N601 (rev 02)
0001:00:1c.1 Host bridge: Numascale AS NumaChip N602 (rev 02)
0001:02:00.0 Ethernet controller: Intel Corporation 82576 Gigabit
Network Connection (rev 01)
0001:02:00.1 Ethernet controller: Intel Corporation 82576 Gigabit
Network Connection (rev 01)
# find /sys/devices/pci* -name numa_node -print -exec cat {} \;
/sys/devices/pci0000:00/0000:00:00.0/numa_node
0
/sys/devices/pci0000:00/0000:00:11.0/numa_node
0
/sys/devices/pci0000:00/0000:00:12.0/numa_node
0
/sys/devices/pci0000:00/0000:00:12.1/numa_node
0
/sys/devices/pci0000:00/0000:00:12.2/numa_node
0
/sys/devices/pci0000:00/0000:00:13.0/numa_node
0
/sys/devices/pci0000:00/0000:00:13.1/numa_node
0
/sys/devices/pci0000:00/0000:00:13.2/numa_node
0
/sys/devices/pci0000:00/0000:00:14.0/numa_node
0
/sys/devices/pci0000:00/0000:00:14.1/numa_node
0
/sys/devices/pci0000:00/0000:00:14.3/numa_node
0
/sys/devices/pci0000:00/0000:00:14.4/0000:01:04.0/numa_node
0
/sys/devices/pci0000:00/0000:00:14.4/numa_node
0
/sys/devices/pci0000:00/0000:00:14.5/numa_node
0
/sys/devices/pci0000:00/0000:00:18.0/numa_node
0
/sys/devices/pci0000:00/0000:00:18.1/numa_node
0
/sys/devices/pci0000:00/0000:00:18.2/numa_node
0
/sys/devices/pci0000:00/0000:00:18.3/numa_node
0
/sys/devices/pci0000:00/0000:00:18.4/numa_node
0
/sys/devices/pci0000:00/0000:00:18.5/numa_node
0
/sys/devices/pci0000:00/0000:00:19.0/numa_node
0
/sys/devices/pci0000:00/0000:00:19.1/numa_node
0
/sys/devices/pci0000:00/0000:00:19.2/numa_node
0
/sys/devices/pci0000:00/0000:00:19.3/numa_node
0
/sys/devices/pci0000:00/0000:00:19.4/numa_node
0
/sys/devices/pci0000:00/0000:00:19.5/numa_node
0
/sys/devices/pci0000:00/0000:00:0d.0/0000:02:00.0/numa_node
0
/sys/devices/pci0000:00/0000:00:0d.0/0000:02:00.1/numa_node
0
/sys/devices/pci0000:00/0000:00:0d.0/numa_node
0
/sys/devices/pci0000:00/0000:00:1a.0/numa_node
0
/sys/devices/pci0000:00/0000:00:1a.1/numa_node
0
/sys/devices/pci0000:00/0000:00:1a.2/numa_node
0
/sys/devices/pci0000:00/0000:00:1a.3/numa_node
0
/sys/devices/pci0000:00/0000:00:1a.4/numa_node
0
/sys/devices/pci0000:00/0000:00:1a.5/numa_node
0
/sys/devices/pci0000:00/0000:00:1b.0/numa_node
0
/sys/devices/pci0000:00/0000:00:1b.1/numa_node
0
/sys/devices/pci0000:00/0000:00:1b.2/numa_node
0
/sys/devices/pci0000:00/0000:00:1b.3/numa_node
0
/sys/devices/pci0000:00/0000:00:1b.4/numa_node
0
/sys/devices/pci0000:00/0000:00:1b.5/numa_node
0
/sys/devices/pci0000:00/0000:00:1c.0/numa_node
0
/sys/devices/pci0000:00/0000:00:1c.1/numa_node
0
/sys/devices/pci0001:00/0001:00:00.0/numa_node
4
/sys/devices/pci0001:00/0001:00:11.0/numa_node
4
/sys/devices/pci0001:00/0001:00:14.0/numa_node
4
/sys/devices/pci0001:00/0001:00:14.4/numa_node
4
/sys/devices/pci0001:00/0001:00:18.0/numa_node
4
/sys/devices/pci0001:00/0001:00:18.1/numa_node
4
/sys/devices/pci0001:00/0001:00:18.2/numa_node
4
/sys/devices/pci0001:00/0001:00:18.3/numa_node
4
/sys/devices/pci0001:00/0001:00:18.4/numa_node
4
/sys/devices/pci0001:00/0001:00:18.5/numa_node
4
/sys/devices/pci0001:00/0001:00:19.0/numa_node
4
/sys/devices/pci0001:00/0001:00:19.1/numa_node
4
/sys/devices/pci0001:00/0001:00:19.2/numa_node
4
/sys/devices/pci0001:00/0001:00:19.3/numa_node
4
/sys/devices/pci0001:00/0001:00:19.4/numa_node
4
/sys/devices/pci0001:00/0001:00:19.5/numa_node
4
/sys/devices/pci0001:00/0001:00:0d.0/0001:02:00.0/numa_node
4
/sys/devices/pci0001:00/0001:00:0d.0/0001:02:00.1/numa_node
4
/sys/devices/pci0001:00/0001:00:0d.0/numa_node
4
/sys/devices/pci0001:00/0001:00:1a.0/numa_node
4
/sys/devices/pci0001:00/0001:00:1a.1/numa_node
4
/sys/devices/pci0001:00/0001:00:1a.2/numa_node
4
/sys/devices/pci0001:00/0001:00:1a.3/numa_node
4
/sys/devices/pci0001:00/0001:00:1a.4/numa_node
4
/sys/devices/pci0001:00/0001:00:1a.5/numa_node
4
/sys/devices/pci0001:00/0001:00:1b.0/numa_node
4
/sys/devices/pci0001:00/0001:00:1b.1/numa_node
4
/sys/devices/pci0001:00/0001:00:1b.2/numa_node
4
/sys/devices/pci0001:00/0001:00:1b.3/numa_node
4
/sys/devices/pci0001:00/0001:00:1b.4/numa_node
4
/sys/devices/pci0001:00/0001:00:1b.5/numa_node
4
/sys/devices/pci0001:00/0001:00:1c.0/numa_node
4
/sys/devices/pci0001:00/0001:00:1c.1/numa_node
4
--
Daniel J Blueman
Principal Software Engineer, Numascale
next prev parent reply other threads:[~2014-04-21 16:53 UTC|newest]
Thread overview: 58+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-03-05 21:06 [PATCH 0/3] amd/pci: Add AMD hostbridge supports for newer AMD systems suravee.suthikulpanit
2014-03-05 21:06 ` [PATCH 1/3] amd/pci: Add supports for generic AMD hostbridges suravee.suthikulpanit
2014-03-05 21:06 ` [PATCH 2/3] amd/pci: Support additional MMIO ranges capabilities suravee.suthikulpanit
2014-03-20 17:33 ` Bjorn Helgaas
2014-03-05 21:06 ` [PATCH 3/3] amd/pci: Miscellaneous code clean up for early_fillup_mp_bus_info suravee.suthikulpanit
2014-03-05 21:24 ` [PATCH 0/3] amd/pci: Add AMD hostbridge supports for newer AMD systems Bjorn Helgaas
2014-03-06 2:13 ` Suravee Suthikulanit
2014-03-06 6:30 ` Suravee Suthikulpanit
2014-03-06 17:40 ` Bjorn Helgaas
2014-03-06 20:03 ` Suravee Suthikulpanit
2014-03-11 18:12 ` Bjorn Helgaas
2014-03-12 21:13 ` Bjorn Helgaas
2014-03-13 1:30 ` Myron Stowe
2014-03-14 2:06 ` Suravee Suthikulpanit
2014-03-17 17:18 ` Bjorn Helgaas
2014-03-20 17:42 ` Bjorn Helgaas
2014-04-19 2:53 ` [PATCH v2 0/5] x86/PCI: Add AMD hostbridge support " Myron Stowe
2014-04-19 2:53 ` [PATCH v2 1/5] x86/PCI: Add support for generic AMD hostbridges Myron Stowe
2014-04-19 11:31 ` Borislav Petkov
2014-04-28 21:10 ` Myron Stowe
2014-04-19 2:53 ` [PATCH v2 2/5] x86/PCI: Support additional MMIO range capabilities Myron Stowe
2014-04-19 13:52 ` Borislav Petkov
2014-04-20 7:59 ` Borislav Petkov
2014-04-25 22:24 ` Myron Stowe
2014-04-26 9:10 ` Borislav Petkov
2014-04-28 20:50 ` Bjorn Helgaas
2014-04-28 21:40 ` Borislav Petkov
2014-04-29 7:33 ` Andreas Herrmann
2014-04-29 10:20 ` Borislav Petkov
2014-04-29 13:07 ` Steffen Persvold
2014-04-29 15:16 ` Suravee Suthikulanit
2014-04-29 19:14 ` Borislav Petkov
2014-04-29 21:40 ` Myron Stowe
2014-04-30 7:00 ` Robert Richter
2014-04-30 7:50 ` Suravee Suthikulpanit
2014-04-30 9:51 ` Robert Richter
2014-04-30 23:03 ` Myron Stowe
2014-04-29 11:19 ` Robert Richter
2014-04-29 7:06 ` Jan Beulich
2014-04-29 3:04 ` Suravee Suthikulanit
2014-04-28 21:19 ` Myron Stowe
2014-04-29 2:47 ` Suravee Suthikulanit
2014-04-29 17:17 ` Robert Richter
2014-04-30 6:41 ` Robert Richter
2014-04-19 2:53 ` [PATCH v2 3/5] x86/PCI: Miscellaneous code clean up for early_fillup_mp_bus_info Myron Stowe
2014-04-20 8:02 ` Borislav Petkov
2014-04-28 21:21 ` Myron Stowe
2014-04-19 2:53 ` [PATCH v2 4/5] ACPI/PCI: Warn if we have to "guess" host bridge node information Myron Stowe
2014-04-20 10:21 ` Borislav Petkov
2014-04-28 21:24 ` Myron Stowe
2014-04-29 19:16 ` Borislav Petkov
2014-04-19 2:53 ` [PATCH v2 5/5] PCI: Remove redundant 'quirk_amd_nb_node' Myron Stowe
2014-04-20 10:54 ` Borislav Petkov
2014-04-20 13:44 ` Myron Stowe
2014-04-21 16:53 ` Daniel J Blueman [this message]
2014-04-29 2:02 ` Suravee Suthikulanit
2014-04-29 19:29 ` Bjorn Helgaas
2014-04-28 21:28 ` Myron Stowe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=53554D27.1010803@numascale.com \
--to=daniel@numascale.com \
--cc=aravind.gopalakrishnan@amd.com \
--cc=bhelgaas@google.com \
--cc=bp@suse.de \
--cc=hpa@zytor.com \
--cc=kim.naru@amd.com \
--cc=linux-acpi@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-pci@vger.kernel.org \
--cc=mingo@redhat.com \
--cc=myron.stowe@gmail.com \
--cc=myron.stowe@redhat.com \
--cc=sp@numascale.com \
--cc=suravee.suthikulpanit@amd.com \
--cc=tglx@linutronix.de \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).