All of lore.kernel.org
 help / color / mirror / Atom feed
From: "A. Wilcox" <AWilcox@Wilcox-Tech.com>
To: Paolo Bonzini <pbonzini@redhat.com>
Cc: Arnd Bergmann <arnd@arndb.de>, Arnd Bergmann <arnd@kernel.org>,
	kvm@vger.kernel.org,
	Thomas Bogendoerfer <tsbogend@alpha.franken.de>,
	Huacai Chen <chenhuacai@kernel.org>,
	Jiaxun Yang <jiaxun.yang@flygoat.com>,
	Michael Ellerman <mpe@ellerman.id.au>,
	Nicholas Piggin <npiggin@gmail.com>,
	Christophe Leroy <christophe.leroy@csgroup.eu>,
	Naveen N Rao <naveen@kernel.org>,
	Madhavan Srinivasan <maddy@linux.ibm.com>,
	Alexander Graf <graf@amazon.com>,
	Crystal Wood <crwood@redhat.com>,
	Anup Patel <anup@brainfault.org>,
	Atish Patra <atishp@atishpatra.org>,
	Paul Walmsley <paul.walmsley@sifive.com>,
	Palmer Dabbelt <palmer@dabbelt.com>,
	Albert Ou <aou@eecs.berkeley.edu>,
	Sean Christopherson <seanjc@google.com>,
	Thomas Gleixner <tglx@linutronix.de>,
	Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>,
	Dave Hansen <dave.hansen@linux.intel.com>,
	x86@kernel.org, "H. Peter Anvin" <hpa@zytor.com>,
	Vitaly Kuznetsov <vkuznets@redhat.com>,
	David Woodhouse <dwmw2@infradead.org>,
	Paul Durrant <paul@xen.org>, Marc Zyngier <maz@kernel.org>,
	linux-kernel@vger.kernel.org, linux-mips@vger.kernel.org,
	linuxppc-dev@lists.ozlabs.org, kvm-riscv@lists.infradead.org,
	linux-riscv@lists.infradead.org
Subject: Re: [RFC 0/5] KVM: drop 32-bit host support on all architectures
Date: Fri, 13 Dec 2024 02:42:41 -0600	[thread overview]
Message-ID: <CE1F96B2-7213-4352-B80F-6E669F5EED97@Wilcox-Tech.com> (raw)
In-Reply-To: <79b9abfe-cfb8-4ef0-8a4b-7b87787e6549@redhat.com>

On Dec 13, 2024, at 2:20 AM, Paolo Bonzini <pbonzini@redhat.com> wrote:
> 
> On 12/13/24 09:03, Arnd Bergmann wrote:
>> On Fri, Dec 13, 2024, at 04:51, A. Wilcox wrote:
>>> On Dec 12, 2024, at 6:55 AM, Arnd Bergmann <arnd@kernel.org> wrote:
>>>> From: Arnd Bergmann <arnd@arndb.de>
>>>> 
>>>> I submitted a patch to remove KVM support for x86-32 hosts earlier
>>>> this month, but there were still concerns that this might be useful for
>>>> testing 32-bit host in general, as that remains supported on three other
>>>> architectures. I have gone through those three now and prepared similar
>>>> patches, as all of them seem to be equally obsolete.
>>>> 
>>>> Support for 32-bit KVM host on Arm hardware was dropped back in 2020
>>>> because of lack of users, despite Cortex-A7/A15/A17 based SoCs being
>>>> much more widely deployed than the other virtualization capable 32-bit
>>>> CPUs (Intel Core Duo/Silverthorne, PowerPC e300/e500/e600, MIPS P5600)
>>>> combined.
>>> 
>>> 
>>> I do use 32-bit KVM on a Core Duo “Yonah” and a Power Mac G4 (MDD), for
>>> purposes of bisecting kernel issues without having to reboot the host
>>> machine (when it can be duplicated in a KVM environment).
>>> 
>>> I suppose it would still be possible to run the hosts on 6.12 LTS for
>>> some time with newer guests, but it would be unfortunate.
>> Would it be an option for you to just test those kernels on 64-bit
>> machines? I assume you prefer to do native builds on 32-bit hardware
>> because that fits your workflow, but once you get into debugging
>> in a virtual machine, the results should generally be the same when
>> building and running on a 64-bit host for both x86-32 and ppc32-classic,
>> right?
> 
> Certainly for x86-32; ppc32 should be able to use PR-state (aka
> trap and emulate) KVM on a 64-bit host but it's a bit more picky.
> Another possibility for ppc32 is just emulation with QEMU.
> 
> Paolo


Most of the reason I use KVM instead of emulation is because I don’t
trust QEMU emulation at all.  There was even a kernel bug that was
introduced affecting 32-bit x86 in the 4.0 cycle that only happened
because QEMU wasn’t emulating writes to %cr4 properly[1].  And PPC32
emulation is far worse than x86_32.  However, I probably could end
up doing x86_32 testing on a combination of bare metal machines and
KVM on x86_64, sure.

As for Power: I will admit I haven’t tested lately, but well into
the 5 series (5.4, at least), you couldn’t boot a ppc32 Linux kernel
on any 64-bit capable hardware.  It would throw what I believe was an
alignment error while quiescing OpenFirmware and toss you back to an
‘ok >’ prompt.  Unfortunately I can’t find any of the bug reports
or ML threads from the time - it was a known bug in the 2.6 days - but
the answer was always “why are you booting a ppc32 kernel on that
hardware anyway?  It’s a ppc64 machine!”  Is this a case where
that would be accepted as a legitimate bug now?  It would be lovely
to use my largely-SMT 3.0 GHz Power9 box for more of my kernel testing
(where possible) instead of relying on a 933 MHz single-thread G4.

-arw

[1]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=a833581e372a;
It had some form of security impact on Pentium-class machines, too,
as RDPMC became available to non-root even when /sys/devices/cpu/rdpmc
was 0.
-- 
kvm-riscv mailing list
kvm-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kvm-riscv

WARNING: multiple messages have this Message-ID (diff)
From: "A. Wilcox" <AWilcox@Wilcox-Tech.com>
To: Paolo Bonzini <pbonzini@redhat.com>
Cc: Arnd Bergmann <arnd@arndb.de>, Arnd Bergmann <arnd@kernel.org>,
	kvm@vger.kernel.org,
	Thomas Bogendoerfer <tsbogend@alpha.franken.de>,
	Huacai Chen <chenhuacai@kernel.org>,
	Jiaxun Yang <jiaxun.yang@flygoat.com>,
	Michael Ellerman <mpe@ellerman.id.au>,
	Nicholas Piggin <npiggin@gmail.com>,
	Christophe Leroy <christophe.leroy@csgroup.eu>,
	Naveen N Rao <naveen@kernel.org>,
	Madhavan Srinivasan <maddy@linux.ibm.com>,
	Alexander Graf <graf@amazon.com>,
	Crystal Wood <crwood@redhat.com>,
	Anup Patel <anup@brainfault.org>,
	Atish Patra <atishp@atishpatra.org>,
	Paul Walmsley <paul.walmsley@sifive.com>,
	Palmer Dabbelt <palmer@dabbelt.com>,
	Albert Ou <aou@eecs.berkeley.edu>,
	Sean Christopherson <seanjc@google.com>,
	Thomas Gleixner <tglx@linutronix.de>,
	Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>,
	Dave Hansen <dave.hansen@linux.intel.com>,
	x86@kernel.org, "H. Peter Anvin" <hpa@zytor.com>,
	Vitaly Kuznetsov <vkuznets@redhat.com>,
	David Woodhouse <dwmw2@infradead.org>,
	Paul Durrant <paul@xen.org>, Marc Zyngier <maz@kernel.org>,
	linux-kernel@vger.kernel.org, linux-mips@vger.kernel.org,
	linuxppc-dev@lists.ozlabs.org, kvm-riscv@lists.infradead.org,
	linux-riscv@lists.infradead.org
Subject: Re: [RFC 0/5] KVM: drop 32-bit host support on all architectures
Date: Fri, 13 Dec 2024 02:42:41 -0600	[thread overview]
Message-ID: <CE1F96B2-7213-4352-B80F-6E669F5EED97@Wilcox-Tech.com> (raw)
In-Reply-To: <79b9abfe-cfb8-4ef0-8a4b-7b87787e6549@redhat.com>

On Dec 13, 2024, at 2:20 AM, Paolo Bonzini <pbonzini@redhat.com> wrote:
> 
> On 12/13/24 09:03, Arnd Bergmann wrote:
>> On Fri, Dec 13, 2024, at 04:51, A. Wilcox wrote:
>>> On Dec 12, 2024, at 6:55 AM, Arnd Bergmann <arnd@kernel.org> wrote:
>>>> From: Arnd Bergmann <arnd@arndb.de>
>>>> 
>>>> I submitted a patch to remove KVM support for x86-32 hosts earlier
>>>> this month, but there were still concerns that this might be useful for
>>>> testing 32-bit host in general, as that remains supported on three other
>>>> architectures. I have gone through those three now and prepared similar
>>>> patches, as all of them seem to be equally obsolete.
>>>> 
>>>> Support for 32-bit KVM host on Arm hardware was dropped back in 2020
>>>> because of lack of users, despite Cortex-A7/A15/A17 based SoCs being
>>>> much more widely deployed than the other virtualization capable 32-bit
>>>> CPUs (Intel Core Duo/Silverthorne, PowerPC e300/e500/e600, MIPS P5600)
>>>> combined.
>>> 
>>> 
>>> I do use 32-bit KVM on a Core Duo “Yonah” and a Power Mac G4 (MDD), for
>>> purposes of bisecting kernel issues without having to reboot the host
>>> machine (when it can be duplicated in a KVM environment).
>>> 
>>> I suppose it would still be possible to run the hosts on 6.12 LTS for
>>> some time with newer guests, but it would be unfortunate.
>> Would it be an option for you to just test those kernels on 64-bit
>> machines? I assume you prefer to do native builds on 32-bit hardware
>> because that fits your workflow, but once you get into debugging
>> in a virtual machine, the results should generally be the same when
>> building and running on a 64-bit host for both x86-32 and ppc32-classic,
>> right?
> 
> Certainly for x86-32; ppc32 should be able to use PR-state (aka
> trap and emulate) KVM on a 64-bit host but it's a bit more picky.
> Another possibility for ppc32 is just emulation with QEMU.
> 
> Paolo


Most of the reason I use KVM instead of emulation is because I don’t
trust QEMU emulation at all.  There was even a kernel bug that was
introduced affecting 32-bit x86 in the 4.0 cycle that only happened
because QEMU wasn’t emulating writes to %cr4 properly[1].  And PPC32
emulation is far worse than x86_32.  However, I probably could end
up doing x86_32 testing on a combination of bare metal machines and
KVM on x86_64, sure.

As for Power: I will admit I haven’t tested lately, but well into
the 5 series (5.4, at least), you couldn’t boot a ppc32 Linux kernel
on any 64-bit capable hardware.  It would throw what I believe was an
alignment error while quiescing OpenFirmware and toss you back to an
‘ok >’ prompt.  Unfortunately I can’t find any of the bug reports
or ML threads from the time - it was a known bug in the 2.6 days - but
the answer was always “why are you booting a ppc32 kernel on that
hardware anyway?  It’s a ppc64 machine!”  Is this a case where
that would be accepted as a legitimate bug now?  It would be lovely
to use my largely-SMT 3.0 GHz Power9 box for more of my kernel testing
(where possible) instead of relying on a 933 MHz single-thread G4.

-arw

[1]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=a833581e372a;
It had some form of security impact on Pentium-class machines, too,
as RDPMC became available to non-root even when /sys/devices/cpu/rdpmc
was 0.

WARNING: multiple messages have this Message-ID (diff)
From: "A. Wilcox" <AWilcox@Wilcox-Tech.com>
To: Paolo Bonzini <pbonzini@redhat.com>
Cc: Arnd Bergmann <arnd@arndb.de>, Arnd Bergmann <arnd@kernel.org>,
	kvm@vger.kernel.org,
	Thomas Bogendoerfer <tsbogend@alpha.franken.de>,
	Huacai Chen <chenhuacai@kernel.org>,
	Jiaxun Yang <jiaxun.yang@flygoat.com>,
	Michael Ellerman <mpe@ellerman.id.au>,
	Nicholas Piggin <npiggin@gmail.com>,
	Christophe Leroy <christophe.leroy@csgroup.eu>,
	Naveen N Rao <naveen@kernel.org>,
	Madhavan Srinivasan <maddy@linux.ibm.com>,
	Alexander Graf <graf@amazon.com>,
	Crystal Wood <crwood@redhat.com>,
	Anup Patel <anup@brainfault.org>,
	Atish Patra <atishp@atishpatra.org>,
	Paul Walmsley <paul.walmsley@sifive.com>,
	Palmer Dabbelt <palmer@dabbelt.com>,
	Albert Ou <aou@eecs.berkeley.edu>,
	Sean Christopherson <seanjc@google.com>,
	Thomas Gleixner <tglx@linutronix.de>,
	Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>,
	Dave Hansen <dave.hansen@linux.intel.com>,
	x86@kernel.org, "H. Peter Anvin" <hpa@zytor.com>,
	Vitaly Kuznetsov <vkuznets@redhat.com>,
	David Woodhouse <dwmw2@infradead.org>,
	Paul Durrant <paul@xen.org>, Marc Zyngier <maz@kernel.org>,
	linux-kernel@vger.kernel.org, linux-mips@vger.kernel.org,
	linuxppc-dev@lists.ozlabs.org, kvm-riscv@lists.infradead.org,
	linux-riscv@lists.infradead.org
Subject: Re: [RFC 0/5] KVM: drop 32-bit host support on all architectures
Date: Fri, 13 Dec 2024 02:42:41 -0600	[thread overview]
Message-ID: <CE1F96B2-7213-4352-B80F-6E669F5EED97@Wilcox-Tech.com> (raw)
In-Reply-To: <79b9abfe-cfb8-4ef0-8a4b-7b87787e6549@redhat.com>

On Dec 13, 2024, at 2:20 AM, Paolo Bonzini <pbonzini@redhat.com> wrote:
> 
> On 12/13/24 09:03, Arnd Bergmann wrote:
>> On Fri, Dec 13, 2024, at 04:51, A. Wilcox wrote:
>>> On Dec 12, 2024, at 6:55 AM, Arnd Bergmann <arnd@kernel.org> wrote:
>>>> From: Arnd Bergmann <arnd@arndb.de>
>>>> 
>>>> I submitted a patch to remove KVM support for x86-32 hosts earlier
>>>> this month, but there were still concerns that this might be useful for
>>>> testing 32-bit host in general, as that remains supported on three other
>>>> architectures. I have gone through those three now and prepared similar
>>>> patches, as all of them seem to be equally obsolete.
>>>> 
>>>> Support for 32-bit KVM host on Arm hardware was dropped back in 2020
>>>> because of lack of users, despite Cortex-A7/A15/A17 based SoCs being
>>>> much more widely deployed than the other virtualization capable 32-bit
>>>> CPUs (Intel Core Duo/Silverthorne, PowerPC e300/e500/e600, MIPS P5600)
>>>> combined.
>>> 
>>> 
>>> I do use 32-bit KVM on a Core Duo “Yonah” and a Power Mac G4 (MDD), for
>>> purposes of bisecting kernel issues without having to reboot the host
>>> machine (when it can be duplicated in a KVM environment).
>>> 
>>> I suppose it would still be possible to run the hosts on 6.12 LTS for
>>> some time with newer guests, but it would be unfortunate.
>> Would it be an option for you to just test those kernels on 64-bit
>> machines? I assume you prefer to do native builds on 32-bit hardware
>> because that fits your workflow, but once you get into debugging
>> in a virtual machine, the results should generally be the same when
>> building and running on a 64-bit host for both x86-32 and ppc32-classic,
>> right?
> 
> Certainly for x86-32; ppc32 should be able to use PR-state (aka
> trap and emulate) KVM on a 64-bit host but it's a bit more picky.
> Another possibility for ppc32 is just emulation with QEMU.
> 
> Paolo


Most of the reason I use KVM instead of emulation is because I don’t
trust QEMU emulation at all.  There was even a kernel bug that was
introduced affecting 32-bit x86 in the 4.0 cycle that only happened
because QEMU wasn’t emulating writes to %cr4 properly[1].  And PPC32
emulation is far worse than x86_32.  However, I probably could end
up doing x86_32 testing on a combination of bare metal machines and
KVM on x86_64, sure.

As for Power: I will admit I haven’t tested lately, but well into
the 5 series (5.4, at least), you couldn’t boot a ppc32 Linux kernel
on any 64-bit capable hardware.  It would throw what I believe was an
alignment error while quiescing OpenFirmware and toss you back to an
‘ok >’ prompt.  Unfortunately I can’t find any of the bug reports
or ML threads from the time - it was a known bug in the 2.6 days - but
the answer was always “why are you booting a ppc32 kernel on that
hardware anyway?  It’s a ppc64 machine!”  Is this a case where
that would be accepted as a legitimate bug now?  It would be lovely
to use my largely-SMT 3.0 GHz Power9 box for more of my kernel testing
(where possible) instead of relying on a 933 MHz single-thread G4.

-arw

[1]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=a833581e372a;
It had some form of security impact on Pentium-class machines, too,
as RDPMC became available to non-root even when /sys/devices/cpu/rdpmc
was 0.
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

  reply	other threads:[~2024-12-13  8:44 UTC|newest]

Thread overview: 72+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2024-12-12 12:55 [RFC 0/5] KVM: drop 32-bit host support on all architectures Arnd Bergmann
2024-12-12 12:55 ` Arnd Bergmann
2024-12-12 12:55 ` Arnd Bergmann
2024-12-12 12:55 ` [RFC 1/5] mips: kvm: drop support for 32-bit hosts Arnd Bergmann
2024-12-12 12:55   ` Arnd Bergmann
2024-12-12 12:55   ` Arnd Bergmann
2024-12-12 13:20   ` Andreas Schwab
2024-12-12 13:20     ` Andreas Schwab
2024-12-12 13:20     ` Andreas Schwab
2024-12-13  9:23     ` Arnd Bergmann
2024-12-13  9:23       ` Arnd Bergmann
2024-12-13  9:23       ` Arnd Bergmann
2024-12-12 12:55 ` [RFC 2/5] powerpc: kvm: drop 32-bit booke Arnd Bergmann
2024-12-12 12:55   ` Arnd Bergmann
2024-12-12 12:55   ` Arnd Bergmann
2024-12-12 18:35   ` Christophe Leroy
2024-12-12 18:35     ` Christophe Leroy
2024-12-12 18:35     ` Christophe Leroy
2024-12-12 21:08     ` Arnd Bergmann
2024-12-12 21:08       ` Arnd Bergmann
2024-12-12 21:08       ` Arnd Bergmann
2024-12-13  6:25       ` Christophe Leroy
2024-12-13  6:25         ` Christophe Leroy
2024-12-13  6:25         ` Christophe Leroy
2024-12-13 10:20         ` Arnd Bergmann
2024-12-13 10:20           ` Arnd Bergmann
2024-12-13 10:20           ` Arnd Bergmann
2024-12-12 12:55 ` [RFC 3/5] powerpc: kvm: drop 32-bit book3s Arnd Bergmann
2024-12-12 12:55   ` Arnd Bergmann
2024-12-12 12:55   ` Arnd Bergmann
2024-12-12 18:34   ` Christophe Leroy
2024-12-12 18:34     ` Christophe Leroy
2024-12-12 18:34     ` Christophe Leroy
2024-12-13 10:04     ` Arnd Bergmann
2024-12-13 10:04       ` Arnd Bergmann
2024-12-13 10:04       ` Arnd Bergmann
2024-12-13 10:27       ` Christophe Leroy
2024-12-13 10:27         ` Christophe Leroy
2024-12-13 10:27         ` Christophe Leroy
2024-12-13 10:39         ` Arnd Bergmann
2024-12-13 10:39           ` Arnd Bergmann
2024-12-13 10:39           ` Arnd Bergmann
2024-12-13  8:02   ` Christophe Leroy
2024-12-13  8:02     ` Christophe Leroy
2024-12-13  8:02     ` Christophe Leroy
2024-12-12 12:55 ` [RFC 4/5] riscv: kvm: drop 32-bit host support Arnd Bergmann
2024-12-12 12:55   ` Arnd Bergmann
2024-12-12 12:55   ` Arnd Bergmann
2024-12-12 12:55 ` [RFC 5/5] x86: kvm " Arnd Bergmann
2024-12-12 12:55   ` Arnd Bergmann
2024-12-12 12:55   ` Arnd Bergmann
2024-12-12 16:27   ` Paolo Bonzini
2024-12-12 16:27     ` Paolo Bonzini
2024-12-12 16:27     ` Paolo Bonzini
2024-12-13  9:22     ` Arnd Bergmann
2024-12-13  9:22       ` Arnd Bergmann
2024-12-13  9:22       ` Arnd Bergmann
2024-12-13  3:51 ` [RFC 0/5] KVM: drop 32-bit host support on all architectures A. Wilcox
2024-12-13  3:51   ` A. Wilcox
2024-12-13  3:51   ` A. Wilcox
2024-12-13  8:03   ` Arnd Bergmann
2024-12-13  8:03     ` Arnd Bergmann
2024-12-13  8:03     ` Arnd Bergmann
2024-12-13  8:20     ` Paolo Bonzini
2024-12-13  8:20       ` Paolo Bonzini
2024-12-13  8:20       ` Paolo Bonzini
2024-12-13  8:42       ` A. Wilcox [this message]
2024-12-13  8:42         ` A. Wilcox
2024-12-13  8:42         ` A. Wilcox
2024-12-13  9:01         ` Arnd Bergmann
2024-12-13  9:01           ` Arnd Bergmann
2024-12-13  9:01           ` Arnd Bergmann

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=CE1F96B2-7213-4352-B80F-6E669F5EED97@Wilcox-Tech.com \
    --to=awilcox@wilcox-tech.com \
    --cc=anup@brainfault.org \
    --cc=aou@eecs.berkeley.edu \
    --cc=arnd@arndb.de \
    --cc=arnd@kernel.org \
    --cc=atishp@atishpatra.org \
    --cc=bp@alien8.de \
    --cc=chenhuacai@kernel.org \
    --cc=christophe.leroy@csgroup.eu \
    --cc=crwood@redhat.com \
    --cc=dave.hansen@linux.intel.com \
    --cc=dwmw2@infradead.org \
    --cc=graf@amazon.com \
    --cc=hpa@zytor.com \
    --cc=jiaxun.yang@flygoat.com \
    --cc=kvm-riscv@lists.infradead.org \
    --cc=kvm@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-mips@vger.kernel.org \
    --cc=linux-riscv@lists.infradead.org \
    --cc=linuxppc-dev@lists.ozlabs.org \
    --cc=maddy@linux.ibm.com \
    --cc=maz@kernel.org \
    --cc=mingo@redhat.com \
    --cc=mpe@ellerman.id.au \
    --cc=naveen@kernel.org \
    --cc=npiggin@gmail.com \
    --cc=palmer@dabbelt.com \
    --cc=paul.walmsley@sifive.com \
    --cc=paul@xen.org \
    --cc=pbonzini@redhat.com \
    --cc=seanjc@google.com \
    --cc=tglx@linutronix.de \
    --cc=tsbogend@alpha.franken.de \
    --cc=vkuznets@redhat.com \
    --cc=x86@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.