From: Andrey Ryabinin <a.ryabinin@samsung.com>
To: Sasha Levin <sasha.levin@oracle.com>,
"Kirill A. Shutemov" <kirill.shutemov@linux.intel.com>,
Linus Torvalds <torvalds@linux-foundation.org>,
Andrew Morton <akpm@linux-foundation.org>,
Mel Gorman <mgorman@suse.de>, Rik van Riel <riel@redhat.com>
Cc: Andi Kleen <ak@linux.intel.com>,
Matthew Wilcox <matthew.r.wilcox@intel.com>,
Dave Hansen <dave.hansen@linux.intel.com>,
Alexander Viro <viro@zeniv.linux.org.uk>,
Dave Chinner <david@fromorbit.com>, Ning Qu <quning@gmail.com>,
linux-mm@kvack.org, linux-fsdevel@vger.kernel.org,
linux-kernel@vger.kernel.org, Dave Jones <davej@redhat.com>,
Konstantin Khlebnikov <koct9i@gmail.com>
Subject: Re: [PATCHv3 1/2] mm: introduce vm_ops->map_pages()
Date: Thu, 24 Jul 2014 10:53:50 +0400 [thread overview]
Message-ID: <53D0AD7E.3050705@samsung.com> (raw)
In-Reply-To: <53D07E96.5000006@oracle.com>
On 07/24/14 07:33, Sasha Levin wrote:
> On 02/27/2014 02:53 PM, Kirill A. Shutemov wrote:
>> The patch introduces new vm_ops callback ->map_pages() and uses it for
>> mapping easy accessible pages around fault address.
>>
>> On read page fault, if filesystem provides ->map_pages(), we try to map
>> up to FAULT_AROUND_PAGES pages around page fault address in hope to
>> reduce number of minor page faults.
>>
>> We call ->map_pages first and use ->fault() as fallback if page by the
>> offset is not ready to be mapped (cold page cache or something).
>>
>> Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
>> ---
>
> Hi all,
>
> This patch triggers use-after-free when fuzzing using trinity and the KASAN
> patchset.
>
I think this should be fixed already by following patch:
From: Konstantin Khlebnikov <koct9i@gmail.com>
Subject: mm: do not call do_fault_around for non-linear fault
Ingo Korb reported that "repeated mapping of the same file on tmpfs using
remap_file_pages sometimes triggers a BUG at mm/filemap.c:202 when the
process exits". He bisected the bug to d7c1755179b82d ("mm: implement
->map_pages for shmem/tmpfs"), although the bug was actually added by
8c6e50b0290c4 ("mm: introduce vm_ops->map_pages()").
Problem is caused by calling do_fault_around for _non-linear_ faiult. In
this case pgoff is shifted and might become negative during calculation.
Faulting around non-linear page-fault has no sense and breaks logic in
do_fault_around because pgoff is shifted.
Signed-off-by: Konstantin Khlebnikov <koct9i@gmail.com>
Reported-by: Ingo Korb <ingo.korb@tu-dortmund.de>
Tested-by: Ingo Korb <ingo.korb@tu-dortmund.de>
Cc: Hugh Dickins <hughd@google.com>
Cc: Sasha Levin <sasha.levin@oracle.com>
Cc: Dave Jones <davej@redhat.com>
Cc: Ning Qu <quning@google.com>
Cc: "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com>
Cc: <stable@vger.kernel.org> [3.15.x]
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
mm/memory.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff -puN mm/memory.c~mm-do-not-call-do_fault_around-for-non-linear-fault mm/memory.c
--- a/mm/memory.c~mm-do-not-call-do_fault_around-for-non-linear-fault
+++ a/mm/memory.c
@@ -2882,7 +2882,8 @@ static int do_read_fault(struct mm_struc
* if page by the offset is not ready to be mapped (cold cache or
* something).
*/
- if (vma->vm_ops->map_pages && fault_around_pages() > 1) {
+ if (vma->vm_ops->map_pages && !(flags & FAULT_FLAG_NONLINEAR) &&
+ fault_around_pages() > 1) {
pte = pte_offset_map_lock(mm, pmd, address, &ptl);
do_fault_around(vma, address, pte, pgoff, flags);
if (!pte_same(*pte, orig_pte))
> KASAN's report is:
>
> [ 663.269187] AddressSanitizer: use after free in do_read_fault.isra.40+0x3c2/0x510 at addr ffff88048a733110
> [ 663.275260] page:ffffea001229ccc0 count:0 mapcount:0 mapping: (null) index:0x0
> [ 663.277061] page flags: 0xafffff80008000(tail)
> [ 663.278759] page dumped because: kasan error
> [ 663.280645] CPU: 6 PID: 9262 Comm: trinity-c104 Not tainted 3.16.0-rc6-next-20140723-sasha-00047-g289342b-dirty #929
> [ 663.282898] 00000000000000fb 0000000000000000 ffffea001229ccc0 ffff88038ac0fb78
> [ 663.288759] ffffffffa5e40903 ffff88038ac0fc48 ffff88038ac0fc38 ffffffffa142acfc
> [ 663.291496] 0000000000000001 ffff880509ff5aa8 ffff88038ac10038 ffff88038ac0fbb0
> [ 663.294379] Call Trace:
> [ 663.294806] dump_stack (lib/dump_stack.c:52)
> [ 663.300665] kasan_report_error (mm/kasan/report.c:98 mm/kasan/report.c:166)
> [ 663.301659] ? debug_smp_processor_id (lib/smp_processor_id.c:57)
> [ 663.304645] ? preempt_count_sub (kernel/sched/core.c:2606)
> [ 663.305800] ? put_lock_stats.isra.13 (./arch/x86/include/asm/preempt.h:98 kernel/locking/lockdep.c:254)
> [ 663.306839] ? do_read_fault.isra.40 (mm/memory.c:2784 mm/memory.c:2849 mm/memory.c:2898)
> [ 663.307515] __asan_load8 (mm/kasan/kasan.c:364)
> [ 663.308038] ? do_read_fault.isra.40 (mm/memory.c:2864 mm/memory.c:2898)
> [ 663.309158] do_read_fault.isra.40 (mm/memory.c:2864 mm/memory.c:2898)
> [ 663.310311] ? _raw_spin_unlock (./arch/x86/include/asm/preempt.h:98 include/linux/spinlock_api_smp.h:152 kernel/locking/spinlock.c:183)
> [ 663.311282] ? __pte_alloc (mm/memory.c:598)
> [ 663.312331] handle_mm_fault (mm/memory.c:3092 mm/memory.c:3225 mm/memory.c:3345 mm/memory.c:3374)
> [ 663.313895] ? pud_huge (./arch/x86/include/asm/paravirt.h:611 arch/x86/mm/hugetlbpage.c:76)
> [ 663.314793] __get_user_pages (mm/gup.c:286 mm/gup.c:478)
> [ 663.315775] __mlock_vma_pages_range (mm/mlock.c:262)
> [ 663.316879] __mm_populate (mm/mlock.c:710)
> [ 663.317813] SyS_remap_file_pages (mm/mmap.c:2653 mm/mmap.c:2593)
> [ 663.318848] tracesys (arch/x86/kernel/entry_64.S:541)
> [ 663.319683] Read of size 8 by thread T9262:
> [ 663.320580] Memory state around the buggy address:
> [ 663.321392] ffff88048a732e80: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> [ 663.322573] ffff88048a732f00: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> [ 663.323802] ffff88048a732f80: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> [ 663.325080] ffff88048a733000: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> [ 663.326327] ffff88048a733080: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> [ 663.327572] >ffff88048a733100: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> [ 663.328840] ^
> [ 663.329487] ffff88048a733180: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> [ 663.330762] ffff88048a733200: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> [ 663.331994] ffff88048a733280: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> [ 663.333262] ffff88048a733300: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> [ 663.334488] ffff88048a733380: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
>
> This also proceeds with the traditional:
>
> [ 663.474532] BUG: unable to handle kernel paging request at ffff88048a635de8
> [ 663.474548] IP: do_read_fault.isra.40 (mm/memory.c:2864 mm/memory.c:2898)
>
> But the rest of it got scrambled between the KASAN prints and the other BUG info + trace (who
> broke printk?!).
>
>
> Thanks,
> Sasha
>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
WARNING: multiple messages have this Message-ID (diff)
From: Andrey Ryabinin <a.ryabinin@samsung.com>
To: Sasha Levin <sasha.levin@oracle.com>,
"Kirill A. Shutemov" <kirill.shutemov@linux.intel.com>,
Linus Torvalds <torvalds@linux-foundation.org>,
Andrew Morton <akpm@linux-foundation.org>,
Mel Gorman <mgorman@suse.de>, Rik van Riel <riel@redhat.com>
Cc: Andi Kleen <ak@linux.intel.com>,
Matthew Wilcox <matthew.r.wilcox@intel.com>,
Dave Hansen <dave.hansen@linux.intel.com>,
Alexander Viro <viro@zeniv.linux.org.uk>,
Dave Chinner <david@fromorbit.com>, Ning Qu <quning@gmail.com>,
linux-mm@kvack.org, linux-fsdevel@vger.kernel.org,
linux-kernel@vger.kernel.org, Dave Jones <davej@redhat.com>,
Konstantin Khlebnikov <koct9i@gmail.com>
Subject: Re: [PATCHv3 1/2] mm: introduce vm_ops->map_pages()
Date: Thu, 24 Jul 2014 10:53:50 +0400 [thread overview]
Message-ID: <53D0AD7E.3050705@samsung.com> (raw)
In-Reply-To: <53D07E96.5000006@oracle.com>
On 07/24/14 07:33, Sasha Levin wrote:
> On 02/27/2014 02:53 PM, Kirill A. Shutemov wrote:
>> The patch introduces new vm_ops callback ->map_pages() and uses it for
>> mapping easy accessible pages around fault address.
>>
>> On read page fault, if filesystem provides ->map_pages(), we try to map
>> up to FAULT_AROUND_PAGES pages around page fault address in hope to
>> reduce number of minor page faults.
>>
>> We call ->map_pages first and use ->fault() as fallback if page by the
>> offset is not ready to be mapped (cold page cache or something).
>>
>> Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
>> ---
>
> Hi all,
>
> This patch triggers use-after-free when fuzzing using trinity and the KASAN
> patchset.
>
I think this should be fixed already by following patch:
From: Konstantin Khlebnikov <koct9i@gmail.com>
Subject: mm: do not call do_fault_around for non-linear fault
Ingo Korb reported that "repeated mapping of the same file on tmpfs using
remap_file_pages sometimes triggers a BUG at mm/filemap.c:202 when the
process exits". He bisected the bug to d7c1755179b82d ("mm: implement
->map_pages for shmem/tmpfs"), although the bug was actually added by
8c6e50b0290c4 ("mm: introduce vm_ops->map_pages()").
Problem is caused by calling do_fault_around for _non-linear_ faiult. In
this case pgoff is shifted and might become negative during calculation.
Faulting around non-linear page-fault has no sense and breaks logic in
do_fault_around because pgoff is shifted.
Signed-off-by: Konstantin Khlebnikov <koct9i@gmail.com>
Reported-by: Ingo Korb <ingo.korb@tu-dortmund.de>
Tested-by: Ingo Korb <ingo.korb@tu-dortmund.de>
Cc: Hugh Dickins <hughd@google.com>
Cc: Sasha Levin <sasha.levin@oracle.com>
Cc: Dave Jones <davej@redhat.com>
Cc: Ning Qu <quning@google.com>
Cc: "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com>
Cc: <stable@vger.kernel.org> [3.15.x]
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
mm/memory.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff -puN mm/memory.c~mm-do-not-call-do_fault_around-for-non-linear-fault mm/memory.c
--- a/mm/memory.c~mm-do-not-call-do_fault_around-for-non-linear-fault
+++ a/mm/memory.c
@@ -2882,7 +2882,8 @@ static int do_read_fault(struct mm_struc
* if page by the offset is not ready to be mapped (cold cache or
* something).
*/
- if (vma->vm_ops->map_pages && fault_around_pages() > 1) {
+ if (vma->vm_ops->map_pages && !(flags & FAULT_FLAG_NONLINEAR) &&
+ fault_around_pages() > 1) {
pte = pte_offset_map_lock(mm, pmd, address, &ptl);
do_fault_around(vma, address, pte, pgoff, flags);
if (!pte_same(*pte, orig_pte))
> KASAN's report is:
>
> [ 663.269187] AddressSanitizer: use after free in do_read_fault.isra.40+0x3c2/0x510 at addr ffff88048a733110
> [ 663.275260] page:ffffea001229ccc0 count:0 mapcount:0 mapping: (null) index:0x0
> [ 663.277061] page flags: 0xafffff80008000(tail)
> [ 663.278759] page dumped because: kasan error
> [ 663.280645] CPU: 6 PID: 9262 Comm: trinity-c104 Not tainted 3.16.0-rc6-next-20140723-sasha-00047-g289342b-dirty #929
> [ 663.282898] 00000000000000fb 0000000000000000 ffffea001229ccc0 ffff88038ac0fb78
> [ 663.288759] ffffffffa5e40903 ffff88038ac0fc48 ffff88038ac0fc38 ffffffffa142acfc
> [ 663.291496] 0000000000000001 ffff880509ff5aa8 ffff88038ac10038 ffff88038ac0fbb0
> [ 663.294379] Call Trace:
> [ 663.294806] dump_stack (lib/dump_stack.c:52)
> [ 663.300665] kasan_report_error (mm/kasan/report.c:98 mm/kasan/report.c:166)
> [ 663.301659] ? debug_smp_processor_id (lib/smp_processor_id.c:57)
> [ 663.304645] ? preempt_count_sub (kernel/sched/core.c:2606)
> [ 663.305800] ? put_lock_stats.isra.13 (./arch/x86/include/asm/preempt.h:98 kernel/locking/lockdep.c:254)
> [ 663.306839] ? do_read_fault.isra.40 (mm/memory.c:2784 mm/memory.c:2849 mm/memory.c:2898)
> [ 663.307515] __asan_load8 (mm/kasan/kasan.c:364)
> [ 663.308038] ? do_read_fault.isra.40 (mm/memory.c:2864 mm/memory.c:2898)
> [ 663.309158] do_read_fault.isra.40 (mm/memory.c:2864 mm/memory.c:2898)
> [ 663.310311] ? _raw_spin_unlock (./arch/x86/include/asm/preempt.h:98 include/linux/spinlock_api_smp.h:152 kernel/locking/spinlock.c:183)
> [ 663.311282] ? __pte_alloc (mm/memory.c:598)
> [ 663.312331] handle_mm_fault (mm/memory.c:3092 mm/memory.c:3225 mm/memory.c:3345 mm/memory.c:3374)
> [ 663.313895] ? pud_huge (./arch/x86/include/asm/paravirt.h:611 arch/x86/mm/hugetlbpage.c:76)
> [ 663.314793] __get_user_pages (mm/gup.c:286 mm/gup.c:478)
> [ 663.315775] __mlock_vma_pages_range (mm/mlock.c:262)
> [ 663.316879] __mm_populate (mm/mlock.c:710)
> [ 663.317813] SyS_remap_file_pages (mm/mmap.c:2653 mm/mmap.c:2593)
> [ 663.318848] tracesys (arch/x86/kernel/entry_64.S:541)
> [ 663.319683] Read of size 8 by thread T9262:
> [ 663.320580] Memory state around the buggy address:
> [ 663.321392] ffff88048a732e80: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> [ 663.322573] ffff88048a732f00: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> [ 663.323802] ffff88048a732f80: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> [ 663.325080] ffff88048a733000: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> [ 663.326327] ffff88048a733080: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> [ 663.327572] >ffff88048a733100: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> [ 663.328840] ^
> [ 663.329487] ffff88048a733180: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> [ 663.330762] ffff88048a733200: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> [ 663.331994] ffff88048a733280: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> [ 663.333262] ffff88048a733300: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> [ 663.334488] ffff88048a733380: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
>
> This also proceeds with the traditional:
>
> [ 663.474532] BUG: unable to handle kernel paging request at ffff88048a635de8
> [ 663.474548] IP: do_read_fault.isra.40 (mm/memory.c:2864 mm/memory.c:2898)
>
> But the rest of it got scrambled between the KASAN prints and the other BUG info + trace (who
> broke printk?!).
>
>
> Thanks,
> Sasha
>
next prev parent reply other threads:[~2014-07-24 6:53 UTC|newest]
Thread overview: 73+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-02-27 19:53 [PATCHv3 0/2] mm: map few pages around fault address if they are in page cache Kirill A. Shutemov
2014-02-27 19:53 ` Kirill A. Shutemov
2014-02-27 19:53 ` [PATCHv3 1/2] mm: introduce vm_ops->map_pages() Kirill A. Shutemov
2014-02-27 19:53 ` Kirill A. Shutemov
2014-02-27 21:59 ` Dave Hansen
2014-02-27 21:59 ` Dave Hansen
2014-02-27 22:06 ` Linus Torvalds
2014-02-27 22:06 ` Linus Torvalds
2014-02-27 22:34 ` Dave Hansen
2014-02-27 22:34 ` Dave Hansen
2014-02-28 0:18 ` Kirill A. Shutemov
2014-02-28 0:18 ` Kirill A. Shutemov
2014-02-28 11:50 ` Benjamin Herrenschmidt
2014-02-28 11:50 ` Benjamin Herrenschmidt
2014-02-27 22:08 ` Andrew Morton
2014-02-27 22:08 ` Andrew Morton
2014-03-03 23:16 ` Andrew Morton
2014-03-04 1:26 ` Kirill A. Shutemov
2014-03-04 1:26 ` Kirill A. Shutemov
2014-03-04 1:26 ` Kirill A. Shutemov
2014-03-05 0:04 ` Rusty Russell
2014-03-05 0:04 ` Rusty Russell
2014-03-05 20:02 ` Andrew Morton
2014-03-05 20:02 ` Andrew Morton
2014-07-24 3:33 ` Sasha Levin
2014-07-24 3:33 ` Sasha Levin
2014-07-24 6:53 ` Andrey Ryabinin [this message]
2014-07-24 6:53 ` Andrey Ryabinin
2014-07-24 12:48 ` Sasha Levin
2014-07-24 12:48 ` Sasha Levin
2014-07-24 13:05 ` Sasha Levin
2014-07-24 13:05 ` Sasha Levin
2014-07-24 13:30 ` Konstantin Khlebnikov
2014-07-24 13:30 ` Konstantin Khlebnikov
2014-07-28 7:43 ` [PATCH] mm: don't allow fault_around_bytes to be 0 Andrey Ryabinin
2014-07-28 7:43 ` Andrey Ryabinin
2014-07-28 7:47 ` Andrey Ryabinin
2014-07-28 7:47 ` Andrey Ryabinin
2014-07-28 9:36 ` Kirill A. Shutemov
2014-07-28 9:36 ` Kirill A. Shutemov
2014-07-28 9:36 ` Kirill A. Shutemov
2014-07-28 10:27 ` Andrey Ryabinin
2014-07-28 10:27 ` Andrey Ryabinin
2014-07-28 10:27 ` Andrey Ryabinin
2014-07-28 10:27 ` Andrey Ryabinin
2014-07-28 10:52 ` Kirill A. Shutemov
2014-07-28 10:52 ` Kirill A. Shutemov
2014-07-28 12:32 ` Sasha Levin
2014-07-28 12:32 ` Sasha Levin
2014-07-28 22:43 ` David Rientjes
2014-07-28 22:43 ` David Rientjes
2014-07-28 15:26 ` Dave Hansen
2014-07-28 15:26 ` Dave Hansen
2014-02-27 19:53 ` [PATCHv3 2/2] mm: implement ->map_pages for page cache Kirill A. Shutemov
2014-02-27 19:53 ` Kirill A. Shutemov
2014-02-27 21:47 ` Andrew Morton
2014-02-27 21:47 ` Andrew Morton
2014-02-28 0:31 ` Kirill A. Shutemov
2014-02-28 0:31 ` Kirill A. Shutemov
2014-04-02 18:03 ` Konstantin Khlebnikov
2014-04-02 18:03 ` Konstantin Khlebnikov
2014-04-02 19:07 ` Kirill A. Shutemov
2014-04-02 19:07 ` Kirill A. Shutemov
2014-02-27 21:28 ` [PATCHv3 0/2] mm: map few pages around fault address if they are in " Linus Torvalds
2014-02-27 21:28 ` Linus Torvalds
2014-02-28 0:10 ` Kirill A. Shutemov
2014-02-28 0:10 ` Kirill A. Shutemov
2014-02-28 3:52 ` Wilcox, Matthew R
2014-02-28 3:52 ` Wilcox, Matthew R
2014-02-28 23:08 ` Linus Torvalds
2014-02-28 23:08 ` Linus Torvalds
2014-03-12 14:22 ` Kirill A. Shutemov
2014-03-12 14:22 ` Kirill A. Shutemov
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=53D0AD7E.3050705@samsung.com \
--to=a.ryabinin@samsung.com \
--cc=ak@linux.intel.com \
--cc=akpm@linux-foundation.org \
--cc=dave.hansen@linux.intel.com \
--cc=davej@redhat.com \
--cc=david@fromorbit.com \
--cc=kirill.shutemov@linux.intel.com \
--cc=koct9i@gmail.com \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=matthew.r.wilcox@intel.com \
--cc=mgorman@suse.de \
--cc=quning@gmail.com \
--cc=riel@redhat.com \
--cc=sasha.levin@oracle.com \
--cc=torvalds@linux-foundation.org \
--cc=viro@zeniv.linux.org.uk \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.