From: Bagas Sanjaya <bagasdotme@gmail.com>
To: Oleksandr Natalenko <oleksandr@natalenko.name>,
Linux Kernel Mailing List <linux-kernel@vger.kernel.org>
Cc: "Thomas Hellström" <thomas.hellstrom@linux.intel.com>,
"Daniel Vetter" <daniel@ffwll.ch>,
"David Airlie" <airlied@gmail.com>,
"Linux Intel Graphics" <intel-gfx@lists.freedesktop.org>,
"Linux DRI Development" <dri-devel@lists.freedesktop.org>,
"Chris Wilson" <chris@chris-wilson.co.uk>,
"Nathan Chancellor" <nathan@kernel.org>,
"Matthew Auld" <matthew.auld@intel.com>,
"Andrzej Hajda" <andrzej.hajda@intel.com>,
"Rodrigo Vivi" <rodrigo.vivi@intel.com>,
"Matt Roper" <matthew.d.roper@intel.com>,
"Linux Regressions" <regressions@lists.linux.dev>
Subject: Re: [Intel-gfx] [REGRESSION] Panic in gen8_ggtt_insert_entries() with v6.5
Date: Tue, 19 Sep 2023 21:08:07 +0700 [thread overview]
Message-ID: <ZQmrR7FVKRdVP53q@debian.me> (raw)
In-Reply-To: <4857570.31r3eYUQgx@natalenko.name>
[-- Attachment #1: Type: text/plain, Size: 3998 bytes --]
On Sat, Sep 02, 2023 at 06:14:12PM +0200, Oleksandr Natalenko wrote:
> Hello.
>
> Since v6.5 kernel the following HW:
>
> * Lenovo T460s laptop with Skylake GT2 [HD Graphics 520] (rev 07)
> * Lenovo T490s laptop with WhiskeyLake-U GT2 [UHD Graphics 620] (rev 02)
>
> is affected by the following crash once KDE on either X11 or Wayland is started:
>
> i915 0000:00:02.0: enabling device (0006 -> 0007)
> i915 0000:00:02.0: vgaarb: deactivate vga console
> i915 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=mem
> i915 0000:00:02.0: [drm] Finished loading DMC firmware i915/skl_dmc_ver1_27.bin (v1.27)
> [drm] Initialized i915 1.6.0 20201103 for 0000:00:02.0 on minor 1
> fbcon: i915drmfb (fb0) is primary device
> i915 0000:00:02.0: [drm] fb0: i915drmfb frame buffer device
> …
> memfd_create() without MFD_EXEC nor MFD_NOEXEC_SEAL, pid=674 'kwin_wayland'
> BUG: unable to handle page fault for address: ffffb422c2800000
> #PF: supervisor write access in kernel mode
> #PF: error_code(0x0002) - not-present page
> PGD 100000067 P4D 100000067 PUD 1001df067 PMD 10d1cf067 PTE 0
> Oops: 0002 [#1] PREEMPT SMP PTI
> CPU: 1 PID: 674 Comm: kwin_wayland Not tainted 6.5.0-pf1 #1 a6c58ff41a7b8bb16a19f5af9e0e9bce20f9f38d
> Hardware name: LENOVO 20FAS2BM0F/20FAS2BM0F, BIOS N1CET90W (1.58 ) 11/15/2022
> RIP: 0010:gen8_ggtt_insert_entries+0xc2/0x140 [i915]
> …
> Call Trace:
> <TASK>
> intel_ggtt_bind_vma+0x3e/0x60 [i915 a83fdc6539431252dba13053979a8b680af86836]
> i915_vma_bind+0x216/0x4b0 [i915 a83fdc6539431252dba13053979a8b680af86836]
> i915_vma_pin_ww+0x405/0xa80 [i915 a83fdc6539431252dba13053979a8b680af86836]
> __i915_ggtt_pin+0x5a/0x130 [i915 a83fdc6539431252dba13053979a8b680af86836]
> i915_ggtt_pin+0x78/0x1f0 [i915 a83fdc6539431252dba13053979a8b680af86836]
> __intel_context_do_pin_ww+0x312/0x700 [i915 a83fdc6539431252dba13053979a8b680af86836]
> i915_gem_do_execbuffer+0xfc6/0x2720 [i915 a83fdc6539431252dba13053979a8b680af86836]
> i915_gem_execbuffer2_ioctl+0x111/0x260 [i915 a83fdc6539431252dba13053979a8b680af86836]
> drm_ioctl_kernel+0xca/0x170
> drm_ioctl+0x30f/0x580
> __x64_sys_ioctl+0x94/0xd0
> do_syscall_64+0x5d/0x90
> entry_SYSCALL_64_after_hwframe+0x6e/0xd8
> …
> note: kwin_wayland[674] exited with irqs disabled
>
> RIP seems to translate into this:
>
> $ scripts/faddr2line drivers/gpu/drm/i915/gt/intel_ggtt.o gen8_ggtt_insert_entries+0xc2
> gen8_ggtt_insert_entries+0xc2/0x150:
> writeq at /home/pf/work/devel/own/pf-kernel/linux/./arch/x86/include/asm/io.h:99
> (inlined by) gen8_set_pte at /home/pf/work/devel/own/pf-kernel/linux/drivers/gpu/drm/i915/gt/intel_ggtt.c:257
> (inlined by) gen8_ggtt_insert_entries at /home/pf/work/devel/own/pf-kernel/linux/drivers/gpu/drm/i915/gt/intel_ggtt.c:300
>
> Probably, recent PTE-related changes are relevant:
>
> $ git log --oneline --no-merges v6.4..v6.5 -- drivers/gpu/drm/i915/gt/intel_ggtt.c
> 3532e75dfadcf drm/i915/uc: perma-pin firmwares
> 4722e2ebe6f21 drm/i915/gt: Fix second parameter type of pre-gen8 pte_encode callbacks
> 9275277d53248 drm/i915: use pat_index instead of cache_level
> 5e352e32aec23 drm/i915: preparation for using PAT index
> 341ad0e8e2542 drm/i915/mtl: Add PTE encode function
>
> Also note Lenovo T14s laptop with TigerLake-LP GT2 [Iris Xe Graphics] (rev 01) is not affected by this issue.
>
> Full dmesg with DRM debug enabled is available in the bugreport I've reported earlier [1]. I'm sending this email to make the issue more visible.
>
> [1] https://gitlab.freedesktop.org/drm/intel/-/issues/9256
>
Thanks for the regression report. I'm adding it to regzbot:
#regzbot ^introduced: 0b62af28f249b9
#regzbot title: gen8_ggtt_insert_entries() panic on Lenovo T14s (Tiger Lake) due to folio_batch() on shmem_sg_free_table()
#regzbot link: https://gitlab.freedesktop.org/drm/intel/-/issues/9256
--
An old man doll... just what I always wanted! - Clara
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]
WARNING: multiple messages have this Message-ID (diff)
From: Bagas Sanjaya <bagasdotme@gmail.com>
To: Oleksandr Natalenko <oleksandr@natalenko.name>,
Linux Kernel Mailing List <linux-kernel@vger.kernel.org>
Cc: "Thomas Hellström" <thomas.hellstrom@linux.intel.com>,
"Matt Roper" <matthew.d.roper@intel.com>,
"Linux Intel Graphics" <intel-gfx@lists.freedesktop.org>,
"Chris Wilson" <chris@chris-wilson.co.uk>,
"Nathan Chancellor" <nathan@kernel.org>,
"Andrzej Hajda" <andrzej.hajda@intel.com>,
"Linux DRI Development" <dri-devel@lists.freedesktop.org>,
"Daniel Vetter" <daniel@ffwll.ch>,
"Rodrigo Vivi" <rodrigo.vivi@intel.com>,
"David Airlie" <airlied@gmail.com>,
"Matthew Auld" <matthew.auld@intel.com>,
"Linux Regressions" <regressions@lists.linux.dev>
Subject: Re: [REGRESSION] Panic in gen8_ggtt_insert_entries() with v6.5
Date: Tue, 19 Sep 2023 21:08:07 +0700 [thread overview]
Message-ID: <ZQmrR7FVKRdVP53q@debian.me> (raw)
In-Reply-To: <4857570.31r3eYUQgx@natalenko.name>
[-- Attachment #1: Type: text/plain, Size: 3998 bytes --]
On Sat, Sep 02, 2023 at 06:14:12PM +0200, Oleksandr Natalenko wrote:
> Hello.
>
> Since v6.5 kernel the following HW:
>
> * Lenovo T460s laptop with Skylake GT2 [HD Graphics 520] (rev 07)
> * Lenovo T490s laptop with WhiskeyLake-U GT2 [UHD Graphics 620] (rev 02)
>
> is affected by the following crash once KDE on either X11 or Wayland is started:
>
> i915 0000:00:02.0: enabling device (0006 -> 0007)
> i915 0000:00:02.0: vgaarb: deactivate vga console
> i915 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=mem
> i915 0000:00:02.0: [drm] Finished loading DMC firmware i915/skl_dmc_ver1_27.bin (v1.27)
> [drm] Initialized i915 1.6.0 20201103 for 0000:00:02.0 on minor 1
> fbcon: i915drmfb (fb0) is primary device
> i915 0000:00:02.0: [drm] fb0: i915drmfb frame buffer device
> …
> memfd_create() without MFD_EXEC nor MFD_NOEXEC_SEAL, pid=674 'kwin_wayland'
> BUG: unable to handle page fault for address: ffffb422c2800000
> #PF: supervisor write access in kernel mode
> #PF: error_code(0x0002) - not-present page
> PGD 100000067 P4D 100000067 PUD 1001df067 PMD 10d1cf067 PTE 0
> Oops: 0002 [#1] PREEMPT SMP PTI
> CPU: 1 PID: 674 Comm: kwin_wayland Not tainted 6.5.0-pf1 #1 a6c58ff41a7b8bb16a19f5af9e0e9bce20f9f38d
> Hardware name: LENOVO 20FAS2BM0F/20FAS2BM0F, BIOS N1CET90W (1.58 ) 11/15/2022
> RIP: 0010:gen8_ggtt_insert_entries+0xc2/0x140 [i915]
> …
> Call Trace:
> <TASK>
> intel_ggtt_bind_vma+0x3e/0x60 [i915 a83fdc6539431252dba13053979a8b680af86836]
> i915_vma_bind+0x216/0x4b0 [i915 a83fdc6539431252dba13053979a8b680af86836]
> i915_vma_pin_ww+0x405/0xa80 [i915 a83fdc6539431252dba13053979a8b680af86836]
> __i915_ggtt_pin+0x5a/0x130 [i915 a83fdc6539431252dba13053979a8b680af86836]
> i915_ggtt_pin+0x78/0x1f0 [i915 a83fdc6539431252dba13053979a8b680af86836]
> __intel_context_do_pin_ww+0x312/0x700 [i915 a83fdc6539431252dba13053979a8b680af86836]
> i915_gem_do_execbuffer+0xfc6/0x2720 [i915 a83fdc6539431252dba13053979a8b680af86836]
> i915_gem_execbuffer2_ioctl+0x111/0x260 [i915 a83fdc6539431252dba13053979a8b680af86836]
> drm_ioctl_kernel+0xca/0x170
> drm_ioctl+0x30f/0x580
> __x64_sys_ioctl+0x94/0xd0
> do_syscall_64+0x5d/0x90
> entry_SYSCALL_64_after_hwframe+0x6e/0xd8
> …
> note: kwin_wayland[674] exited with irqs disabled
>
> RIP seems to translate into this:
>
> $ scripts/faddr2line drivers/gpu/drm/i915/gt/intel_ggtt.o gen8_ggtt_insert_entries+0xc2
> gen8_ggtt_insert_entries+0xc2/0x150:
> writeq at /home/pf/work/devel/own/pf-kernel/linux/./arch/x86/include/asm/io.h:99
> (inlined by) gen8_set_pte at /home/pf/work/devel/own/pf-kernel/linux/drivers/gpu/drm/i915/gt/intel_ggtt.c:257
> (inlined by) gen8_ggtt_insert_entries at /home/pf/work/devel/own/pf-kernel/linux/drivers/gpu/drm/i915/gt/intel_ggtt.c:300
>
> Probably, recent PTE-related changes are relevant:
>
> $ git log --oneline --no-merges v6.4..v6.5 -- drivers/gpu/drm/i915/gt/intel_ggtt.c
> 3532e75dfadcf drm/i915/uc: perma-pin firmwares
> 4722e2ebe6f21 drm/i915/gt: Fix second parameter type of pre-gen8 pte_encode callbacks
> 9275277d53248 drm/i915: use pat_index instead of cache_level
> 5e352e32aec23 drm/i915: preparation for using PAT index
> 341ad0e8e2542 drm/i915/mtl: Add PTE encode function
>
> Also note Lenovo T14s laptop with TigerLake-LP GT2 [Iris Xe Graphics] (rev 01) is not affected by this issue.
>
> Full dmesg with DRM debug enabled is available in the bugreport I've reported earlier [1]. I'm sending this email to make the issue more visible.
>
> [1] https://gitlab.freedesktop.org/drm/intel/-/issues/9256
>
Thanks for the regression report. I'm adding it to regzbot:
#regzbot ^introduced: 0b62af28f249b9
#regzbot title: gen8_ggtt_insert_entries() panic on Lenovo T14s (Tiger Lake) due to folio_batch() on shmem_sg_free_table()
#regzbot link: https://gitlab.freedesktop.org/drm/intel/-/issues/9256
--
An old man doll... just what I always wanted! - Clara
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]
WARNING: multiple messages have this Message-ID (diff)
From: Bagas Sanjaya <bagasdotme@gmail.com>
To: Oleksandr Natalenko <oleksandr@natalenko.name>,
Linux Kernel Mailing List <linux-kernel@vger.kernel.org>
Cc: "Thomas Hellström" <thomas.hellstrom@linux.intel.com>,
"Linux Intel Graphics" <intel-gfx@lists.freedesktop.org>,
"Linux DRI Development" <dri-devel@lists.freedesktop.org>,
"Chris Wilson" <chris@chris-wilson.co.uk>,
"Nathan Chancellor" <nathan@kernel.org>,
"Matthew Auld" <matthew.auld@intel.com>,
"Andrzej Hajda" <andrzej.hajda@intel.com>,
"Rodrigo Vivi" <rodrigo.vivi@intel.com>,
"Matt Roper" <matthew.d.roper@intel.com>,
"Linux Regressions" <regressions@lists.linux.dev>
Subject: Re: [REGRESSION] Panic in gen8_ggtt_insert_entries() with v6.5
Date: Tue, 19 Sep 2023 21:08:07 +0700 [thread overview]
Message-ID: <ZQmrR7FVKRdVP53q@debian.me> (raw)
In-Reply-To: <4857570.31r3eYUQgx@natalenko.name>
[-- Attachment #1: Type: text/plain, Size: 3998 bytes --]
On Sat, Sep 02, 2023 at 06:14:12PM +0200, Oleksandr Natalenko wrote:
> Hello.
>
> Since v6.5 kernel the following HW:
>
> * Lenovo T460s laptop with Skylake GT2 [HD Graphics 520] (rev 07)
> * Lenovo T490s laptop with WhiskeyLake-U GT2 [UHD Graphics 620] (rev 02)
>
> is affected by the following crash once KDE on either X11 or Wayland is started:
>
> i915 0000:00:02.0: enabling device (0006 -> 0007)
> i915 0000:00:02.0: vgaarb: deactivate vga console
> i915 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=mem
> i915 0000:00:02.0: [drm] Finished loading DMC firmware i915/skl_dmc_ver1_27.bin (v1.27)
> [drm] Initialized i915 1.6.0 20201103 for 0000:00:02.0 on minor 1
> fbcon: i915drmfb (fb0) is primary device
> i915 0000:00:02.0: [drm] fb0: i915drmfb frame buffer device
> …
> memfd_create() without MFD_EXEC nor MFD_NOEXEC_SEAL, pid=674 'kwin_wayland'
> BUG: unable to handle page fault for address: ffffb422c2800000
> #PF: supervisor write access in kernel mode
> #PF: error_code(0x0002) - not-present page
> PGD 100000067 P4D 100000067 PUD 1001df067 PMD 10d1cf067 PTE 0
> Oops: 0002 [#1] PREEMPT SMP PTI
> CPU: 1 PID: 674 Comm: kwin_wayland Not tainted 6.5.0-pf1 #1 a6c58ff41a7b8bb16a19f5af9e0e9bce20f9f38d
> Hardware name: LENOVO 20FAS2BM0F/20FAS2BM0F, BIOS N1CET90W (1.58 ) 11/15/2022
> RIP: 0010:gen8_ggtt_insert_entries+0xc2/0x140 [i915]
> …
> Call Trace:
> <TASK>
> intel_ggtt_bind_vma+0x3e/0x60 [i915 a83fdc6539431252dba13053979a8b680af86836]
> i915_vma_bind+0x216/0x4b0 [i915 a83fdc6539431252dba13053979a8b680af86836]
> i915_vma_pin_ww+0x405/0xa80 [i915 a83fdc6539431252dba13053979a8b680af86836]
> __i915_ggtt_pin+0x5a/0x130 [i915 a83fdc6539431252dba13053979a8b680af86836]
> i915_ggtt_pin+0x78/0x1f0 [i915 a83fdc6539431252dba13053979a8b680af86836]
> __intel_context_do_pin_ww+0x312/0x700 [i915 a83fdc6539431252dba13053979a8b680af86836]
> i915_gem_do_execbuffer+0xfc6/0x2720 [i915 a83fdc6539431252dba13053979a8b680af86836]
> i915_gem_execbuffer2_ioctl+0x111/0x260 [i915 a83fdc6539431252dba13053979a8b680af86836]
> drm_ioctl_kernel+0xca/0x170
> drm_ioctl+0x30f/0x580
> __x64_sys_ioctl+0x94/0xd0
> do_syscall_64+0x5d/0x90
> entry_SYSCALL_64_after_hwframe+0x6e/0xd8
> …
> note: kwin_wayland[674] exited with irqs disabled
>
> RIP seems to translate into this:
>
> $ scripts/faddr2line drivers/gpu/drm/i915/gt/intel_ggtt.o gen8_ggtt_insert_entries+0xc2
> gen8_ggtt_insert_entries+0xc2/0x150:
> writeq at /home/pf/work/devel/own/pf-kernel/linux/./arch/x86/include/asm/io.h:99
> (inlined by) gen8_set_pte at /home/pf/work/devel/own/pf-kernel/linux/drivers/gpu/drm/i915/gt/intel_ggtt.c:257
> (inlined by) gen8_ggtt_insert_entries at /home/pf/work/devel/own/pf-kernel/linux/drivers/gpu/drm/i915/gt/intel_ggtt.c:300
>
> Probably, recent PTE-related changes are relevant:
>
> $ git log --oneline --no-merges v6.4..v6.5 -- drivers/gpu/drm/i915/gt/intel_ggtt.c
> 3532e75dfadcf drm/i915/uc: perma-pin firmwares
> 4722e2ebe6f21 drm/i915/gt: Fix second parameter type of pre-gen8 pte_encode callbacks
> 9275277d53248 drm/i915: use pat_index instead of cache_level
> 5e352e32aec23 drm/i915: preparation for using PAT index
> 341ad0e8e2542 drm/i915/mtl: Add PTE encode function
>
> Also note Lenovo T14s laptop with TigerLake-LP GT2 [Iris Xe Graphics] (rev 01) is not affected by this issue.
>
> Full dmesg with DRM debug enabled is available in the bugreport I've reported earlier [1]. I'm sending this email to make the issue more visible.
>
> [1] https://gitlab.freedesktop.org/drm/intel/-/issues/9256
>
Thanks for the regression report. I'm adding it to regzbot:
#regzbot ^introduced: 0b62af28f249b9
#regzbot title: gen8_ggtt_insert_entries() panic on Lenovo T14s (Tiger Lake) due to folio_batch() on shmem_sg_free_table()
#regzbot link: https://gitlab.freedesktop.org/drm/intel/-/issues/9256
--
An old man doll... just what I always wanted! - Clara
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]
next prev parent reply other threads:[~2023-09-19 14:08 UTC|newest]
Thread overview: 37+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-09-02 16:14 [Intel-gfx] [REGRESSION] Panic in gen8_ggtt_insert_entries() with v6.5 Oleksandr Natalenko
2023-09-02 16:14 ` Oleksandr Natalenko
2023-09-02 16:14 ` Oleksandr Natalenko
2023-09-19 8:26 ` [Intel-gfx] [REGRESSION] [BISECTED] " Oleksandr Natalenko
2023-09-19 8:26 ` Oleksandr Natalenko
2023-09-19 8:26 ` Oleksandr Natalenko
2023-09-19 13:23 ` [Intel-gfx] " Oleksandr Natalenko
2023-09-19 13:23 ` Oleksandr Natalenko
2023-09-19 13:23 ` Oleksandr Natalenko
2023-09-19 14:03 ` [Intel-gfx] " Bagas Sanjaya
2023-09-19 14:03 ` Bagas Sanjaya
2023-09-19 14:03 ` Bagas Sanjaya
2023-09-19 14:14 ` Oleksandr Natalenko
2023-09-19 14:14 ` Oleksandr Natalenko
2023-09-19 14:14 ` Oleksandr Natalenko
2023-09-19 15:43 ` Matthew Wilcox
2023-09-19 15:43 ` Matthew Wilcox
2023-09-19 15:43 ` Matthew Wilcox
2023-09-19 16:02 ` [Intel-gfx] " Matthew Wilcox
2023-09-19 16:02 ` Matthew Wilcox
2023-09-19 16:02 ` Matthew Wilcox
2023-09-19 18:11 ` [Intel-gfx] " Oleksandr Natalenko
2023-09-19 18:11 ` Oleksandr Natalenko
2023-09-19 18:11 ` Oleksandr Natalenko
2023-09-19 19:15 ` [Intel-gfx] " Matthew Wilcox
2023-09-19 19:15 ` Matthew Wilcox
2023-09-19 19:15 ` Matthew Wilcox
2023-09-19 14:08 ` Bagas Sanjaya [this message]
2023-09-19 14:08 ` [REGRESSION] " Bagas Sanjaya
2023-09-19 14:08 ` Bagas Sanjaya
2023-09-29 13:45 ` [Intel-gfx] " Linux regression tracking #update (Thorsten Leemhuis)
2023-09-29 13:45 ` Linux regression tracking #update (Thorsten Leemhuis)
2023-09-29 13:45 ` Linux regression tracking #update (Thorsten Leemhuis)
2023-09-19 16:12 ` [Intel-gfx] ✗ Fi.CI.CHECKPATCH: warning for " Patchwork
2023-09-19 16:23 ` Matthew Wilcox
2023-09-19 16:12 ` [Intel-gfx] ✗ Fi.CI.SPARSE: " Patchwork
2023-09-19 16:30 ` [Intel-gfx] ✗ Fi.CI.BAT: failure " Patchwork
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=ZQmrR7FVKRdVP53q@debian.me \
--to=bagasdotme@gmail.com \
--cc=airlied@gmail.com \
--cc=andrzej.hajda@intel.com \
--cc=chris@chris-wilson.co.uk \
--cc=daniel@ffwll.ch \
--cc=dri-devel@lists.freedesktop.org \
--cc=intel-gfx@lists.freedesktop.org \
--cc=linux-kernel@vger.kernel.org \
--cc=matthew.auld@intel.com \
--cc=matthew.d.roper@intel.com \
--cc=nathan@kernel.org \
--cc=oleksandr@natalenko.name \
--cc=regressions@lists.linux.dev \
--cc=rodrigo.vivi@intel.com \
--cc=thomas.hellstrom@linux.intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.