From: hkran <hkran@linux.vnet.ibm.com>
To: Gleb Natapov <gleb@redhat.com>
Cc: Stefan Hajnoczi <stefanha@gmail.com>,
Michael Tokarev <mjt@tls.msk.ru>,
qemu-devel <qemu-devel@nongnu.org>,
"Mars.Cao" <caobingbu@gmail.com>
Subject: Re: [Qemu-devel] Windows 7 shutdown causes BSOD
Date: Thu, 17 Nov 2011 18:55:14 +0800 [thread overview]
Message-ID: <4EC4E812.2010403@linux.vnet.ibm.com> (raw)
In-Reply-To: <20111117063715.GB2933@redhat.com>
On 11/17/2011 02:37 PM, Gleb Natapov wrote:
> On Thu, Nov 17, 2011 at 02:29:47PM +0800, hkran wrote:
>> On 11/16/2011 06:51 PM, Gleb Natapov wrote:
>>> On Wed, Nov 16, 2011 at 10:48:15AM +0000, Stefan Hajnoczi wrote:
>>>> On Wed, Nov 16, 2011 at 10:14 AM, hkran<hkran@linux.vnet.ibm.com> wrote:
>>>>> On 11/15/2011 09:17 PM, Stefan Hajnoczi wrote:
>>>>>> On Fri, Nov 4, 2011 at 11:25 AM, Stefan Hajnoczi<stefanha@gmail.com>
>>>>>> wrote:
>>>>>>> On Fri, Nov 4, 2011 at 10:48 AM, Stefan Hajnoczi<stefanha@gmail.com>
>>>>>>> wrote:
>>>>>>>> Windows 7 32-bit guest blue screens when I shut it down properly with
>>>>>>>> Start | Shut Down. The blue screen is only displayed for a split
>>>>>>>> second before the guest reboots so I am not able to easily tell what
>>>>>>>> it says. My guess is that Windows is triple-faulting or soft
>>>>>>>> rebooting - note that I told Windows to shut down, not reboot.
>>>>>>>>
>>>>>>>> This issue happens on qemu.git/master (and Debian kvm 0.14.1+dfsg-3).
>>>>>>>> Here is the QEMU command-line:
>>>>>>>>
>>>>>>>> x86_64-softmmu/qemu-system-x86_64 -L pc-bios -cpu qemu32 -enable-kvm
>>>>>>>> -m 1024 -rtc base=localtime -drive
>>>>>>>> file=win7.img,if=none,id=drive-ide0-0-0,format=raw -device
>>>>>>>> ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1
>>>>>>>>
>>>>>>>> Questions:
>>>>>>>>
>>>>>>>> Is anyone else experiencing this?
>>>>>>>>
>>>>>>>> Is anyone fixing this?
>>>>>>>>
>>>>>>>> If not I will play with it. Disabling ACPI might reveal the source of
>>>>>>>> the problem. If that turns up nothing I will try to get the BSOD or
>>>>>>>> WinDbg output.
>>>>>>> Thanks to Andreas Faerber and Michael Tokarev I found out the
>>>>>>> automatic reboot can be disabled in Windows. Here is the BSOD
>>>>>>> information:
>>>>>>>
>>>>>>> IRQL_NOT_LESS_OR_EQUAL
>>>>>>> STOP: 0x0000000A (0x00000000,0x000000FF,0x00000001,0x828B7220)
>>>>>> This decodes to:
>>>>>> "Windows or a kernel-mode driver accessed paged memory at
>>>>>> DISPATCH_LEVEL or above."
>>>>>>
>>>>>> Memory referenced: 0x00000000
>>>>>> IRQL: 0xff
>>>>>> Read/write: Write (1)
>>>>>> Address which referenced memory: 0x828B7220
>>>>>>
>>>>>> http://msdn.microsoft.com/en-us/library/ff560129%28v=VS.85%29.aspx
>>>>>>
>>>>>> Looks like a NULL pointer reference or maybe a deliberate "we should
>>>>>> never get here" failure.
>>>>>>
>>>>>> Stefan
>>>>>>
>>>>> I can reproduce this bug in my environment and found out that it has
>>>>> something with the type of "CPU".
>>>>> I tried the command line args as the same as Stefan's and definitely casue
>>>>> the BSOD.
>>>>> If i change the "-cpu qemu32" to "-cpu qemu64" or "-cpu core2duo" or
>>>>> nothing. it will shutdown as expected, that means something?
>>>> Thanks for sharing. The guest is definitely sees a differed CPUID and
>>>> can therefore take different code paths. I'm not sure what
>>>> specifically could have changed.
>>>>
>>> Try adding/removing individual cpuid bits.
>>>
>>> --
>>> Gleb.
>>>
>> It seems that the .model = 3 for "qemu32" type in struct
>> builtin_x86_defs in the file target-i386/cpuid.c make it failed.
>> if I changed it to "2" which is the same as "qemu64". it will be OK.
> Enable tracing like this:
> # echo kvm:kvm_msr> /sys/kernel/debug/tracing/set_event
> and then reboot windows with qemu32. Look for strange things in the log.
> Like msr read/write that caused #GP.
>
> --
> Gleb.
>
the trace for kvm:kvm_msr, if it is not enough, I can enable more kvm
tracing.
# tracer: nop
#
# TASK-PID CPU# TIMESTAMP FUNCTION
# | | | | |
qemu-system-x86-14634 [002] 30288.217803: kvm_msr: msr_write 8b = 0x0
qemu-system-x86-14634 [002] 30288.217808: kvm_msr: msr_read 8b = 0x0
qemu-system-x86-14634 [002] 30288.217842: kvm_msr: msr_write 8b = 0x0
qemu-system-x86-14634 [002] 30288.217844: kvm_msr: msr_read 8b = 0x0
qemu-system-x86-14634 [002] 30288.217846: kvm_msr: msr_write 8b = 0x0
qemu-system-x86-14634 [002] 30288.217849: kvm_msr: msr_read 8b = 0x0
qemu-system-x86-14634 [002] 30288.218326: kvm_msr: msr_write 10 = 0x0
qemu-system-x86-14634 [002] 30290.891908: kvm_msr: msr_write 277 =
0x7010600070106
qemu-system-x86-14634 [003] 30290.978139: kvm_msr: msr_read 179 = 0x20
qemu-system-x86-14634 [002] 30295.672706: kvm_msr: msr_read 179 = 0x20
qemu-system-x86-14634 [002] 30295.672709: kvm_msr: msr_read 401 = 0x0
qemu-system-x86-14634 [002] 30295.672710: kvm_msr: msr_read 405 = 0x0
qemu-system-x86-14634 [002] 30295.672711: kvm_msr: msr_read 409 = 0x0
qemu-system-x86-14634 [002] 30295.672712: kvm_msr: msr_read 40d = 0x0
qemu-system-x86-14634 [002] 30295.672713: kvm_msr: msr_read 411 = 0x0
qemu-system-x86-14634 [002] 30295.672714: kvm_msr: msr_read 415 = 0x0
qemu-system-x86-14634 [002] 30295.672715: kvm_msr: msr_read 419 = 0x0
qemu-system-x86-14634 [002] 30295.672716: kvm_msr: msr_read 41d = 0x0
qemu-system-x86-14634 [002] 30295.672717: kvm_msr: msr_read 421 = 0x0
qemu-system-x86-14634 [002] 30295.672718: kvm_msr: msr_read 425 = 0x0
qemu-system-x86-14634 [002] 30295.672719: kvm_msr: msr_read 429 = 0x0
qemu-system-x86-14634 [002] 30295.672720: kvm_msr: msr_read 42d = 0x0
qemu-system-x86-14634 [002] 30295.672721: kvm_msr: msr_read 431 = 0x0
qemu-system-x86-14634 [002] 30295.672722: kvm_msr: msr_read 435 = 0x0
qemu-system-x86-14634 [002] 30295.672723: kvm_msr: msr_read 439 = 0x0
qemu-system-x86-14634 [002] 30295.672724: kvm_msr: msr_read 43d = 0x0
qemu-system-x86-14634 [002] 30295.672725: kvm_msr: msr_read 441 = 0x0
qemu-system-x86-14634 [002] 30295.672726: kvm_msr: msr_read 445 = 0x0
qemu-system-x86-14634 [002] 30295.672727: kvm_msr: msr_read 449 = 0x0
qemu-system-x86-14634 [002] 30295.672728: kvm_msr: msr_read 44d = 0x0
qemu-system-x86-14634 [002] 30295.672729: kvm_msr: msr_read 451 = 0x0
qemu-system-x86-14634 [002] 30295.672729: kvm_msr: msr_read 455 = 0x0
qemu-system-x86-14634 [002] 30295.672730: kvm_msr: msr_read 459 = 0x0
qemu-system-x86-14634 [002] 30295.672731: kvm_msr: msr_read 45d = 0x0
qemu-system-x86-14634 [002] 30295.672732: kvm_msr: msr_read 461 = 0x0
qemu-system-x86-14634 [002] 30295.672733: kvm_msr: msr_read 465 = 0x0
qemu-system-x86-14634 [002] 30295.672734: kvm_msr: msr_read 469 = 0x0
qemu-system-x86-14634 [002] 30295.672735: kvm_msr: msr_read 46d = 0x0
qemu-system-x86-14634 [002] 30295.672736: kvm_msr: msr_read 471 = 0x0
qemu-system-x86-14634 [002] 30295.672737: kvm_msr: msr_read 475 = 0x0
qemu-system-x86-14634 [002] 30295.672738: kvm_msr: msr_read 479 = 0x0
qemu-system-x86-14634 [002] 30295.672739: kvm_msr: msr_read 47d = 0x0
qemu-system-x86-14634 [002] 30295.880773: kvm_msr: msr_write 8b = 0x0
qemu-system-x86-14634 [002] 30295.880778: kvm_msr: msr_read 8b = 0x0
qemu-system-x86-14634 [002] 30298.757975: kvm_msr: msr_write 8b = 0x0
qemu-system-x86-14634 [002] 30298.757980: kvm_msr: msr_read 8b = 0x0
qemu-system-x86-14634 [000] 30342.901013: kvm_msr: msr_read 401 = 0x0
qemu-system-x86-14634 [000] 30342.901017: kvm_msr: msr_read 405 = 0x0
qemu-system-x86-14634 [000] 30342.901019: kvm_msr: msr_read 409 = 0x0
qemu-system-x86-14634 [000] 30342.901020: kvm_msr: msr_read 40d = 0x0
qemu-system-x86-14634 [000] 30342.901021: kvm_msr: msr_read 411 = 0x0
qemu-system-x86-14634 [000] 30342.901022: kvm_msr: msr_read 415 = 0x0
qemu-system-x86-14634 [000] 30342.901023: kvm_msr: msr_read 419 = 0x0
qemu-system-x86-14634 [000] 30342.901024: kvm_msr: msr_read 41d = 0x0
qemu-system-x86-14634 [000] 30342.901026: kvm_msr: msr_read 421 = 0x0
qemu-system-x86-14634 [000] 30342.901027: kvm_msr: msr_read 425 = 0x0
qemu-system-x86-14634 [000] 30342.901028: kvm_msr: msr_read 429 = 0x0
qemu-system-x86-14634 [000] 30342.901029: kvm_msr: msr_read 42d = 0x0
qemu-system-x86-14634 [000] 30342.901030: kvm_msr: msr_read 431 = 0x0
qemu-system-x86-14634 [000] 30342.901031: kvm_msr: msr_read 435 = 0x0
qemu-system-x86-14634 [000] 30342.901032: kvm_msr: msr_read 439 = 0x0
qemu-system-x86-14634 [000] 30342.901034: kvm_msr: msr_read 43d = 0x0
qemu-system-x86-14634 [000] 30342.901035: kvm_msr: msr_read 441 = 0x0
qemu-system-x86-14634 [000] 30342.901036: kvm_msr: msr_read 445 = 0x0
qemu-system-x86-14634 [000] 30342.901037: kvm_msr: msr_read 449 = 0x0
qemu-system-x86-14634 [000] 30342.901038: kvm_msr: msr_read 44d = 0x0
qemu-system-x86-14634 [000] 30342.901039: kvm_msr: msr_read 451 = 0x0
qemu-system-x86-14634 [000] 30342.901041: kvm_msr: msr_read 455 = 0x0
qemu-system-x86-14634 [000] 30342.901042: kvm_msr: msr_read 459 = 0x0
qemu-system-x86-14634 [000] 30342.901043: kvm_msr: msr_read 45d = 0x0
qemu-system-x86-14634 [000] 30342.901044: kvm_msr: msr_read 461 = 0x0
qemu-system-x86-14634 [000] 30342.901046: kvm_msr: msr_read 465 = 0x0
qemu-system-x86-14634 [000] 30342.901047: kvm_msr: msr_read 469 = 0x0
qemu-system-x86-14634 [000] 30342.901048: kvm_msr: msr_read 46d = 0x0
qemu-system-x86-14634 [000] 30342.901049: kvm_msr: msr_read 471 = 0x0
qemu-system-x86-14634 [000] 30342.901050: kvm_msr: msr_read 475 = 0x0
qemu-system-x86-14634 [000] 30342.901052: kvm_msr: msr_read 479 = 0x0
qemu-system-x86-14634 [000] 30342.901053: kvm_msr: msr_read 47d = 0x0
qemu-system-x86-14634 [003] 30389.843468: kvm_msr: msr_read 401 = 0x0
qemu-system-x86-14634 [003] 30389.843474: kvm_msr: msr_read 405 = 0x0
qemu-system-x86-14634 [003] 30389.843477: kvm_msr: msr_read 409 = 0x0
qemu-system-x86-14634 [003] 30389.843480: kvm_msr: msr_read 40d = 0x0
qemu-system-x86-14634 [003] 30389.843483: kvm_msr: msr_read 411 = 0x0
qemu-system-x86-14634 [003] 30389.843486: kvm_msr: msr_read 415 = 0x0
qemu-system-x86-14634 [003] 30389.843489: kvm_msr: msr_read 419 = 0x0
qemu-system-x86-14634 [003] 30389.843492: kvm_msr: msr_read 41d = 0x0
qemu-system-x86-14634 [003] 30389.843495: kvm_msr: msr_read 421 = 0x0
qemu-system-x86-14634 [003] 30389.843498: kvm_msr: msr_read 425 = 0x0
qemu-system-x86-14634 [003] 30389.843501: kvm_msr: msr_read 429 = 0x0
qemu-system-x86-14634 [003] 30389.843504: kvm_msr: msr_read 42d = 0x0
qemu-system-x86-14634 [003] 30389.843507: kvm_msr: msr_read 431 = 0x0
qemu-system-x86-14634 [003] 30389.843510: kvm_msr: msr_read 435 = 0x0
qemu-system-x86-14634 [003] 30389.843513: kvm_msr: msr_read 439 = 0x0
qemu-system-x86-14634 [003] 30389.843516: kvm_msr: msr_read 43d = 0x0
qemu-system-x86-14634 [003] 30389.843519: kvm_msr: msr_read 441 = 0x0
qemu-system-x86-14634 [003] 30389.843522: kvm_msr: msr_read 445 = 0x0
qemu-system-x86-14634 [003] 30389.843525: kvm_msr: msr_read 449 = 0x0
qemu-system-x86-14634 [003] 30389.843528: kvm_msr: msr_read 44d = 0x0
qemu-system-x86-14634 [003] 30389.843531: kvm_msr: msr_read 451 = 0x0
qemu-system-x86-14634 [003] 30389.843534: kvm_msr: msr_read 455 = 0x0
qemu-system-x86-14634 [003] 30389.843537: kvm_msr: msr_read 459 = 0x0
qemu-system-x86-14634 [003] 30389.843540: kvm_msr: msr_read 45d = 0x0
qemu-system-x86-14634 [003] 30389.843543: kvm_msr: msr_read 461 = 0x0
qemu-system-x86-14634 [003] 30389.843546: kvm_msr: msr_read 465 = 0x0
qemu-system-x86-14634 [003] 30389.843548: kvm_msr: msr_read 469 = 0x0
qemu-system-x86-14634 [003] 30389.843551: kvm_msr: msr_read 46d = 0x0
qemu-system-x86-14634 [003] 30389.843555: kvm_msr: msr_read 471 = 0x0
qemu-system-x86-14634 [003] 30389.843558: kvm_msr: msr_read 475 = 0x0
qemu-system-x86-14634 [003] 30389.843561: kvm_msr: msr_read 479 = 0x0
qemu-system-x86-14634 [003] 30389.843564: kvm_msr: msr_read 47d = 0x0
qemu-system-x86-14634 [002] 30436.593753: kvm_msr: msr_read 401 = 0x0
qemu-system-x86-14634 [002] 30436.593758: kvm_msr: msr_read 405 = 0x0
qemu-system-x86-14634 [002] 30436.593761: kvm_msr: msr_read 409 = 0x0
qemu-system-x86-14634 [002] 30436.593763: kvm_msr: msr_read 40d = 0x0
qemu-system-x86-14634 [002] 30436.593765: kvm_msr: msr_read 411 = 0x0
qemu-system-x86-14634 [002] 30436.593767: kvm_msr: msr_read 415 = 0x0
qemu-system-x86-14634 [002] 30436.593770: kvm_msr: msr_read 419 = 0x0
qemu-system-x86-14634 [002] 30436.593772: kvm_msr: msr_read 41d = 0x0
qemu-system-x86-14634 [002] 30436.593774: kvm_msr: msr_read 421 = 0x0
qemu-system-x86-14634 [002] 30436.593776: kvm_msr: msr_read 425 = 0x0
qemu-system-x86-14634 [002] 30436.593779: kvm_msr: msr_read 429 = 0x0
qemu-system-x86-14634 [002] 30436.593781: kvm_msr: msr_read 42d = 0x0
qemu-system-x86-14634 [002] 30436.593783: kvm_msr: msr_read 431 = 0x0
qemu-system-x86-14634 [002] 30436.593785: kvm_msr: msr_read 435 = 0x0
qemu-system-x86-14634 [002] 30436.593787: kvm_msr: msr_read 439 = 0x0
qemu-system-x86-14634 [002] 30436.593790: kvm_msr: msr_read 43d = 0x0
qemu-system-x86-14634 [002] 30436.593792: kvm_msr: msr_read 441 = 0x0
qemu-system-x86-14634 [002] 30436.593794: kvm_msr: msr_read 445 = 0x0
qemu-system-x86-14634 [002] 30436.593797: kvm_msr: msr_read 449 = 0x0
qemu-system-x86-14634 [002] 30436.593800: kvm_msr: msr_read 44d = 0x0
qemu-system-x86-14634 [002] 30436.593803: kvm_msr: msr_read 451 = 0x0
qemu-system-x86-14634 [002] 30436.593806: kvm_msr: msr_read 455 = 0x0
qemu-system-x86-14634 [002] 30436.593809: kvm_msr: msr_read 459 = 0x0
qemu-system-x86-14634 [002] 30436.593812: kvm_msr: msr_read 45d = 0x0
qemu-system-x86-14634 [002] 30436.593816: kvm_msr: msr_read 461 = 0x0
qemu-system-x86-14634 [002] 30436.593818: kvm_msr: msr_read 465 = 0x0
qemu-system-x86-14634 [002] 30436.593821: kvm_msr: msr_read 469 = 0x0
qemu-system-x86-14634 [002] 30436.593824: kvm_msr: msr_read 46d = 0x0
qemu-system-x86-14634 [002] 30436.593827: kvm_msr: msr_read 471 = 0x0
qemu-system-x86-14634 [002] 30436.593830: kvm_msr: msr_read 475 = 0x0
qemu-system-x86-14634 [002] 30436.593834: kvm_msr: msr_read 479 = 0x0
qemu-system-x86-14634 [002] 30436.593837: kvm_msr: msr_read 47d = 0x0
qemu-system-x86-14634 [002] 30483.401291: kvm_msr: msr_read 401 = 0x0
qemu-system-x86-14634 [002] 30483.401296: kvm_msr: msr_read 405 = 0x0
qemu-system-x86-14634 [002] 30483.401298: kvm_msr: msr_read 409 = 0x0
qemu-system-x86-14634 [002] 30483.401301: kvm_msr: msr_read 40d = 0x0
qemu-system-x86-14634 [002] 30483.401303: kvm_msr: msr_read 411 = 0x0
qemu-system-x86-14634 [002] 30483.401305: kvm_msr: msr_read 415 = 0x0
qemu-system-x86-14634 [002] 30483.401308: kvm_msr: msr_read 419 = 0x0
qemu-system-x86-14634 [002] 30483.401310: kvm_msr: msr_read 41d = 0x0
qemu-system-x86-14634 [002] 30483.401312: kvm_msr: msr_read 421 = 0x0
qemu-system-x86-14634 [002] 30483.401314: kvm_msr: msr_read 425 = 0x0
qemu-system-x86-14634 [002] 30483.401316: kvm_msr: msr_read 429 = 0x0
qemu-system-x86-14634 [002] 30483.401319: kvm_msr: msr_read 42d = 0x0
qemu-system-x86-14634 [002] 30483.401321: kvm_msr: msr_read 431 = 0x0
qemu-system-x86-14634 [002] 30483.401323: kvm_msr: msr_read 435 = 0x0
qemu-system-x86-14634 [002] 30483.401325: kvm_msr: msr_read 439 = 0x0
qemu-system-x86-14634 [002] 30483.401328: kvm_msr: msr_read 43d = 0x0
qemu-system-x86-14634 [002] 30483.401330: kvm_msr: msr_read 441 = 0x0
qemu-system-x86-14634 [002] 30483.401332: kvm_msr: msr_read 445 = 0x0
qemu-system-x86-14634 [002] 30483.401334: kvm_msr: msr_read 449 = 0x0
qemu-system-x86-14634 [002] 30483.401336: kvm_msr: msr_read 44d = 0x0
qemu-system-x86-14634 [002] 30483.401339: kvm_msr: msr_read 451 = 0x0
qemu-system-x86-14634 [002] 30483.401341: kvm_msr: msr_read 455 = 0x0
qemu-system-x86-14634 [002] 30483.401343: kvm_msr: msr_read 459 = 0x0
qemu-system-x86-14634 [002] 30483.401345: kvm_msr: msr_read 45d = 0x0
qemu-system-x86-14634 [002] 30483.401347: kvm_msr: msr_read 461 = 0x0
qemu-system-x86-14634 [002] 30483.401350: kvm_msr: msr_read 465 = 0x0
qemu-system-x86-14634 [002] 30483.401352: kvm_msr: msr_read 469 = 0x0
qemu-system-x86-14634 [002] 30483.401354: kvm_msr: msr_read 46d = 0x0
qemu-system-x86-14634 [002] 30483.401356: kvm_msr: msr_read 471 = 0x0
qemu-system-x86-14634 [002] 30483.401359: kvm_msr: msr_read 475 = 0x0
qemu-system-x86-14634 [002] 30483.401361: kvm_msr: msr_read 479 = 0x0
qemu-system-x86-14634 [002] 30483.401363: kvm_msr: msr_read 47d = 0x0
BSOD
next prev parent reply other threads:[~2011-11-17 11:49 UTC|newest]
Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-11-04 10:48 [Qemu-devel] Windows 7 shutdown causes BSOD Stefan Hajnoczi
2011-11-04 11:25 ` Stefan Hajnoczi
2011-11-15 13:17 ` Stefan Hajnoczi
2011-11-16 10:14 ` hkran
2011-11-16 10:48 ` Stefan Hajnoczi
2011-11-16 10:51 ` Gleb Natapov
[not found] ` <4EC4A9DB.2060406@linux.vnet.ibm.com>
2011-11-17 6:37 ` Gleb Natapov
2011-11-17 10:55 ` hkran [this message]
2011-11-17 11:08 ` Gleb Natapov
2011-11-18 9:11 ` hkran
2011-11-22 8:45 ` Gleb Natapov
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4EC4E812.2010403@linux.vnet.ibm.com \
--to=hkran@linux.vnet.ibm.com \
--cc=caobingbu@gmail.com \
--cc=gleb@redhat.com \
--cc=mjt@tls.msk.ru \
--cc=qemu-devel@nongnu.org \
--cc=stefanha@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.