From: hkran <hkran@linux.vnet.ibm.com>
To: Gleb Natapov <gleb@redhat.com>
Cc: Stefan Hajnoczi <stefanha@gmail.com>,
Michael Tokarev <mjt@tls.msk.ru>,
qemu-devel <qemu-devel@nongnu.org>,
"Mars.Cao" <caobingbu@gmail.com>
Subject: Re: [Qemu-devel] Windows 7 shutdown causes BSOD
Date: Thu, 17 Nov 2011 18:55:14 +0800 [thread overview]
Message-ID: <4EC4E812.2010403@linux.vnet.ibm.com> (raw)
In-Reply-To: <20111117063715.GB2933@redhat.com>
On 11/17/2011 02:37 PM, Gleb Natapov wrote:
> On Thu, Nov 17, 2011 at 02:29:47PM +0800, hkran wrote:
>> On 11/16/2011 06:51 PM, Gleb Natapov wrote:
>>> On Wed, Nov 16, 2011 at 10:48:15AM +0000, Stefan Hajnoczi wrote:
>>>> On Wed, Nov 16, 2011 at 10:14 AM, hkran<hkran@linux.vnet.ibm.com> wrote:
>>>>> On 11/15/2011 09:17 PM, Stefan Hajnoczi wrote:
>>>>>> On Fri, Nov 4, 2011 at 11:25 AM, Stefan Hajnoczi<stefanha@gmail.com>
>>>>>> wrote:
>>>>>>> On Fri, Nov 4, 2011 at 10:48 AM, Stefan Hajnoczi<stefanha@gmail.com>
>>>>>>> wrote:
>>>>>>>> Windows 7 32-bit guest blue screens when I shut it down properly with
>>>>>>>> Start | Shut Down. The blue screen is only displayed for a split
>>>>>>>> second before the guest reboots so I am not able to easily tell what
>>>>>>>> it says. My guess is that Windows is triple-faulting or soft
>>>>>>>> rebooting - note that I told Windows to shut down, not reboot.
>>>>>>>>
>>>>>>>> This issue happens on qemu.git/master (and Debian kvm 0.14.1+dfsg-3).
>>>>>>>> Here is the QEMU command-line:
>>>>>>>>
>>>>>>>> x86_64-softmmu/qemu-system-x86_64 -L pc-bios -cpu qemu32 -enable-kvm
>>>>>>>> -m 1024 -rtc base=localtime -drive
>>>>>>>> file=win7.img,if=none,id=drive-ide0-0-0,format=raw -device
>>>>>>>> ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1
>>>>>>>>
>>>>>>>> Questions:
>>>>>>>>
>>>>>>>> Is anyone else experiencing this?
>>>>>>>>
>>>>>>>> Is anyone fixing this?
>>>>>>>>
>>>>>>>> If not I will play with it. Disabling ACPI might reveal the source of
>>>>>>>> the problem. If that turns up nothing I will try to get the BSOD or
>>>>>>>> WinDbg output.
>>>>>>> Thanks to Andreas Faerber and Michael Tokarev I found out the
>>>>>>> automatic reboot can be disabled in Windows. Here is the BSOD
>>>>>>> information:
>>>>>>>
>>>>>>> IRQL_NOT_LESS_OR_EQUAL
>>>>>>> STOP: 0x0000000A (0x00000000,0x000000FF,0x00000001,0x828B7220)
>>>>>> This decodes to:
>>>>>> "Windows or a kernel-mode driver accessed paged memory at
>>>>>> DISPATCH_LEVEL or above."
>>>>>>
>>>>>> Memory referenced: 0x00000000
>>>>>> IRQL: 0xff
>>>>>> Read/write: Write (1)
>>>>>> Address which referenced memory: 0x828B7220
>>>>>>
>>>>>> http://msdn.microsoft.com/en-us/library/ff560129%28v=VS.85%29.aspx
>>>>>>
>>>>>> Looks like a NULL pointer reference or maybe a deliberate "we should
>>>>>> never get here" failure.
>>>>>>
>>>>>> Stefan
>>>>>>
>>>>> I can reproduce this bug in my environment and found out that it has
>>>>> something with the type of "CPU".
>>>>> I tried the command line args as the same as Stefan's and definitely casue
>>>>> the BSOD.
>>>>> If i change the "-cpu qemu32" to "-cpu qemu64" or "-cpu core2duo" or
>>>>> nothing. it will shutdown as expected, that means something?
>>>> Thanks for sharing. The guest is definitely sees a differed CPUID and
>>>> can therefore take different code paths. I'm not sure what
>>>> specifically could have changed.
>>>>
>>> Try adding/removing individual cpuid bits.
>>>
>>> --
>>> Gleb.
>>>
>> It seems that the .model = 3 for "qemu32" type in struct
>> builtin_x86_defs in the file target-i386/cpuid.c make it failed.
>> if I changed it to "2" which is the same as "qemu64". it will be OK.
> Enable tracing like this:
> # echo kvm:kvm_msr> /sys/kernel/debug/tracing/set_event
> and then reboot windows with qemu32. Look for strange things in the log.
> Like msr read/write that caused #GP.
>
> --
> Gleb.
>
the trace for kvm:kvm_msr, if it is not enough, I can enable more kvm
tracing.
# tracer: nop
#
# TASK-PID CPU# TIMESTAMP FUNCTION
# | | | | |
qemu-system-x86-14634 [002] 30288.217803: kvm_msr: msr_write 8b = 0x0
qemu-system-x86-14634 [002] 30288.217808: kvm_msr: msr_read 8b = 0x0
qemu-system-x86-14634 [002] 30288.217842: kvm_msr: msr_write 8b = 0x0
qemu-system-x86-14634 [002] 30288.217844: kvm_msr: msr_read 8b = 0x0
qemu-system-x86-14634 [002] 30288.217846: kvm_msr: msr_write 8b = 0x0
qemu-system-x86-14634 [002] 30288.217849: kvm_msr: msr_read 8b = 0x0
qemu-system-x86-14634 [002] 30288.218326: kvm_msr: msr_write 10 = 0x0
qemu-system-x86-14634 [002] 30290.891908: kvm_msr: msr_write 277 =
0x7010600070106
qemu-system-x86-14634 [003] 30290.978139: kvm_msr: msr_read 179 = 0x20
qemu-system-x86-14634 [002] 30295.672706: kvm_msr: msr_read 179 = 0x20
qemu-system-x86-14634 [002] 30295.672709: kvm_msr: msr_read 401 = 0x0
qemu-system-x86-14634 [002] 30295.672710: kvm_msr: msr_read 405 = 0x0
qemu-system-x86-14634 [002] 30295.672711: kvm_msr: msr_read 409 = 0x0
qemu-system-x86-14634 [002] 30295.672712: kvm_msr: msr_read 40d = 0x0
qemu-system-x86-14634 [002] 30295.672713: kvm_msr: msr_read 411 = 0x0
qemu-system-x86-14634 [002] 30295.672714: kvm_msr: msr_read 415 = 0x0
qemu-system-x86-14634 [002] 30295.672715: kvm_msr: msr_read 419 = 0x0
qemu-system-x86-14634 [002] 30295.672716: kvm_msr: msr_read 41d = 0x0
qemu-system-x86-14634 [002] 30295.672717: kvm_msr: msr_read 421 = 0x0
qemu-system-x86-14634 [002] 30295.672718: kvm_msr: msr_read 425 = 0x0
qemu-system-x86-14634 [002] 30295.672719: kvm_msr: msr_read 429 = 0x0
qemu-system-x86-14634 [002] 30295.672720: kvm_msr: msr_read 42d = 0x0
qemu-system-x86-14634 [002] 30295.672721: kvm_msr: msr_read 431 = 0x0
qemu-system-x86-14634 [002] 30295.672722: kvm_msr: msr_read 435 = 0x0
qemu-system-x86-14634 [002] 30295.672723: kvm_msr: msr_read 439 = 0x0
qemu-system-x86-14634 [002] 30295.672724: kvm_msr: msr_read 43d = 0x0
qemu-system-x86-14634 [002] 30295.672725: kvm_msr: msr_read 441 = 0x0
qemu-system-x86-14634 [002] 30295.672726: kvm_msr: msr_read 445 = 0x0
qemu-system-x86-14634 [002] 30295.672727: kvm_msr: msr_read 449 = 0x0
qemu-system-x86-14634 [002] 30295.672728: kvm_msr: msr_read 44d = 0x0
qemu-system-x86-14634 [002] 30295.672729: kvm_msr: msr_read 451 = 0x0
qemu-system-x86-14634 [002] 30295.672729: kvm_msr: msr_read 455 = 0x0
qemu-system-x86-14634 [002] 30295.672730: kvm_msr: msr_read 459 = 0x0
qemu-system-x86-14634 [002] 30295.672731: kvm_msr: msr_read 45d = 0x0
qemu-system-x86-14634 [002] 30295.672732: kvm_msr: msr_read 461 = 0x0
qemu-system-x86-14634 [002] 30295.672733: kvm_msr: msr_read 465 = 0x0
qemu-system-x86-14634 [002] 30295.672734: kvm_msr: msr_read 469 = 0x0
qemu-system-x86-14634 [002] 30295.672735: kvm_msr: msr_read 46d = 0x0
qemu-system-x86-14634 [002] 30295.672736: kvm_msr: msr_read 471 = 0x0
qemu-system-x86-14634 [002] 30295.672737: kvm_msr: msr_read 475 = 0x0
qemu-system-x86-14634 [002] 30295.672738: kvm_msr: msr_read 479 = 0x0
qemu-system-x86-14634 [002] 30295.672739: kvm_msr: msr_read 47d = 0x0
qemu-system-x86-14634 [002] 30295.880773: kvm_msr: msr_write 8b = 0x0
qemu-system-x86-14634 [002] 30295.880778: kvm_msr: msr_read 8b = 0x0
qemu-system-x86-14634 [002] 30298.757975: kvm_msr: msr_write 8b = 0x0
qemu-system-x86-14634 [002] 30298.757980: kvm_msr: msr_read 8b = 0x0
qemu-system-x86-14634 [000] 30342.901013: kvm_msr: msr_read 401 = 0x0
qemu-system-x86-14634 [000] 30342.901017: kvm_msr: msr_read 405 = 0x0
qemu-system-x86-14634 [000] 30342.901019: kvm_msr: msr_read 409 = 0x0
qemu-system-x86-14634 [000] 30342.901020: kvm_msr: msr_read 40d = 0x0
qemu-system-x86-14634 [000] 30342.901021: kvm_msr: msr_read 411 = 0x0
qemu-system-x86-14634 [000] 30342.901022: kvm_msr: msr_read 415 = 0x0
qemu-system-x86-14634 [000] 30342.901023: kvm_msr: msr_read 419 = 0x0
qemu-system-x86-14634 [000] 30342.901024: kvm_msr: msr_read 41d = 0x0
qemu-system-x86-14634 [000] 30342.901026: kvm_msr: msr_read 421 = 0x0
qemu-system-x86-14634 [000] 30342.901027: kvm_msr: msr_read 425 = 0x0
qemu-system-x86-14634 [000] 30342.901028: kvm_msr: msr_read 429 = 0x0
qemu-system-x86-14634 [000] 30342.901029: kvm_msr: msr_read 42d = 0x0
qemu-system-x86-14634 [000] 30342.901030: kvm_msr: msr_read 431 = 0x0
qemu-system-x86-14634 [000] 30342.901031: kvm_msr: msr_read 435 = 0x0
qemu-system-x86-14634 [000] 30342.901032: kvm_msr: msr_read 439 = 0x0
qemu-system-x86-14634 [000] 30342.901034: kvm_msr: msr_read 43d = 0x0
qemu-system-x86-14634 [000] 30342.901035: kvm_msr: msr_read 441 = 0x0
qemu-system-x86-14634 [000] 30342.901036: kvm_msr: msr_read 445 = 0x0
qemu-system-x86-14634 [000] 30342.901037: kvm_msr: msr_read 449 = 0x0
qemu-system-x86-14634 [000] 30342.901038: kvm_msr: msr_read 44d = 0x0
qemu-system-x86-14634 [000] 30342.901039: kvm_msr: msr_read 451 = 0x0
qemu-system-x86-14634 [000] 30342.901041: kvm_msr: msr_read 455 = 0x0
qemu-system-x86-14634 [000] 30342.901042: kvm_msr: msr_read 459 = 0x0
qemu-system-x86-14634 [000] 30342.901043: kvm_msr: msr_read 45d = 0x0
qemu-system-x86-14634 [000] 30342.901044: kvm_msr: msr_read 461 = 0x0
qemu-system-x86-14634 [000] 30342.901046: kvm_msr: msr_read 465 = 0x0
qemu-system-x86-14634 [000] 30342.901047: kvm_msr: msr_read 469 = 0x0
qemu-system-x86-14634 [000] 30342.901048: kvm_msr: msr_read 46d = 0x0
qemu-system-x86-14634 [000] 30342.901049: kvm_msr: msr_read 471 = 0x0
qemu-system-x86-14634 [000] 30342.901050: kvm_msr: msr_read 475 = 0x0
qemu-system-x86-14634 [000] 30342.901052: kvm_msr: msr_read 479 = 0x0
qemu-system-x86-14634 [000] 30342.901053: kvm_msr: msr_read 47d = 0x0
qemu-system-x86-14634 [003] 30389.843468: kvm_msr: msr_read 401 = 0x0
qemu-system-x86-14634 [003] 30389.843474: kvm_msr: msr_read 405 = 0x0
qemu-system-x86-14634 [003] 30389.843477: kvm_msr: msr_read 409 = 0x0
qemu-system-x86-14634 [003] 30389.843480: kvm_msr: msr_read 40d = 0x0
qemu-system-x86-14634 [003] 30389.843483: kvm_msr: msr_read 411 = 0x0
qemu-system-x86-14634 [003] 30389.843486: kvm_msr: msr_read 415 = 0x0
qemu-system-x86-14634 [003] 30389.843489: kvm_msr: msr_read 419 = 0x0
qemu-system-x86-14634 [003] 30389.843492: kvm_msr: msr_read 41d = 0x0
qemu-system-x86-14634 [003] 30389.843495: kvm_msr: msr_read 421 = 0x0
qemu-system-x86-14634 [003] 30389.843498: kvm_msr: msr_read 425 = 0x0
qemu-system-x86-14634 [003] 30389.843501: kvm_msr: msr_read 429 = 0x0
qemu-system-x86-14634 [003] 30389.843504: kvm_msr: msr_read 42d = 0x0
qemu-system-x86-14634 [003] 30389.843507: kvm_msr: msr_read 431 = 0x0
qemu-system-x86-14634 [003] 30389.843510: kvm_msr: msr_read 435 = 0x0
qemu-system-x86-14634 [003] 30389.843513: kvm_msr: msr_read 439 = 0x0
qemu-system-x86-14634 [003] 30389.843516: kvm_msr: msr_read 43d = 0x0
qemu-system-x86-14634 [003] 30389.843519: kvm_msr: msr_read 441 = 0x0
qemu-system-x86-14634 [003] 30389.843522: kvm_msr: msr_read 445 = 0x0
qemu-system-x86-14634 [003] 30389.843525: kvm_msr: msr_read 449 = 0x0
qemu-system-x86-14634 [003] 30389.843528: kvm_msr: msr_read 44d = 0x0
qemu-system-x86-14634 [003] 30389.843531: kvm_msr: msr_read 451 = 0x0
qemu-system-x86-14634 [003] 30389.843534: kvm_msr: msr_read 455 = 0x0
qemu-system-x86-14634 [003] 30389.843537: kvm_msr: msr_read 459 = 0x0
qemu-system-x86-14634 [003] 30389.843540: kvm_msr: msr_read 45d = 0x0
qemu-system-x86-14634 [003] 30389.843543: kvm_msr: msr_read 461 = 0x0
qemu-system-x86-14634 [003] 30389.843546: kvm_msr: msr_read 465 = 0x0
qemu-system-x86-14634 [003] 30389.843548: kvm_msr: msr_read 469 = 0x0
qemu-system-x86-14634 [003] 30389.843551: kvm_msr: msr_read 46d = 0x0
qemu-system-x86-14634 [003] 30389.843555: kvm_msr: msr_read 471 = 0x0
qemu-system-x86-14634 [003] 30389.843558: kvm_msr: msr_read 475 = 0x0
qemu-system-x86-14634 [003] 30389.843561: kvm_msr: msr_read 479 = 0x0
qemu-system-x86-14634 [003] 30389.843564: kvm_msr: msr_read 47d = 0x0
qemu-system-x86-14634 [002] 30436.593753: kvm_msr: msr_read 401 = 0x0
qemu-system-x86-14634 [002] 30436.593758: kvm_msr: msr_read 405 = 0x0
qemu-system-x86-14634 [002] 30436.593761: kvm_msr: msr_read 409 = 0x0
qemu-system-x86-14634 [002] 30436.593763: kvm_msr: msr_read 40d = 0x0
qemu-system-x86-14634 [002] 30436.593765: kvm_msr: msr_read 411 = 0x0
qemu-system-x86-14634 [002] 30436.593767: kvm_msr: msr_read 415 = 0x0
qemu-system-x86-14634 [002] 30436.593770: kvm_msr: msr_read 419 = 0x0
qemu-system-x86-14634 [002] 30436.593772: kvm_msr: msr_read 41d = 0x0
qemu-system-x86-14634 [002] 30436.593774: kvm_msr: msr_read 421 = 0x0
qemu-system-x86-14634 [002] 30436.593776: kvm_msr: msr_read 425 = 0x0
qemu-system-x86-14634 [002] 30436.593779: kvm_msr: msr_read 429 = 0x0
qemu-system-x86-14634 [002] 30436.593781: kvm_msr: msr_read 42d = 0x0
qemu-system-x86-14634 [002] 30436.593783: kvm_msr: msr_read 431 = 0x0
qemu-system-x86-14634 [002] 30436.593785: kvm_msr: msr_read 435 = 0x0
qemu-system-x86-14634 [002] 30436.593787: kvm_msr: msr_read 439 = 0x0
qemu-system-x86-14634 [002] 30436.593790: kvm_msr: msr_read 43d = 0x0
qemu-system-x86-14634 [002] 30436.593792: kvm_msr: msr_read 441 = 0x0
qemu-system-x86-14634 [002] 30436.593794: kvm_msr: msr_read 445 = 0x0
qemu-system-x86-14634 [002] 30436.593797: kvm_msr: msr_read 449 = 0x0
qemu-system-x86-14634 [002] 30436.593800: kvm_msr: msr_read 44d = 0x0
qemu-system-x86-14634 [002] 30436.593803: kvm_msr: msr_read 451 = 0x0
qemu-system-x86-14634 [002] 30436.593806: kvm_msr: msr_read 455 = 0x0
qemu-system-x86-14634 [002] 30436.593809: kvm_msr: msr_read 459 = 0x0
qemu-system-x86-14634 [002] 30436.593812: kvm_msr: msr_read 45d = 0x0
qemu-system-x86-14634 [002] 30436.593816: kvm_msr: msr_read 461 = 0x0
qemu-system-x86-14634 [002] 30436.593818: kvm_msr: msr_read 465 = 0x0
qemu-system-x86-14634 [002] 30436.593821: kvm_msr: msr_read 469 = 0x0
qemu-system-x86-14634 [002] 30436.593824: kvm_msr: msr_read 46d = 0x0
qemu-system-x86-14634 [002] 30436.593827: kvm_msr: msr_read 471 = 0x0
qemu-system-x86-14634 [002] 30436.593830: kvm_msr: msr_read 475 = 0x0
qemu-system-x86-14634 [002] 30436.593834: kvm_msr: msr_read 479 = 0x0
qemu-system-x86-14634 [002] 30436.593837: kvm_msr: msr_read 47d = 0x0
qemu-system-x86-14634 [002] 30483.401291: kvm_msr: msr_read 401 = 0x0
qemu-system-x86-14634 [002] 30483.401296: kvm_msr: msr_read 405 = 0x0
qemu-system-x86-14634 [002] 30483.401298: kvm_msr: msr_read 409 = 0x0
qemu-system-x86-14634 [002] 30483.401301: kvm_msr: msr_read 40d = 0x0
qemu-system-x86-14634 [002] 30483.401303: kvm_msr: msr_read 411 = 0x0
qemu-system-x86-14634 [002] 30483.401305: kvm_msr: msr_read 415 = 0x0
qemu-system-x86-14634 [002] 30483.401308: kvm_msr: msr_read 419 = 0x0
qemu-system-x86-14634 [002] 30483.401310: kvm_msr: msr_read 41d = 0x0
qemu-system-x86-14634 [002] 30483.401312: kvm_msr: msr_read 421 = 0x0
qemu-system-x86-14634 [002] 30483.401314: kvm_msr: msr_read 425 = 0x0
qemu-system-x86-14634 [002] 30483.401316: kvm_msr: msr_read 429 = 0x0
qemu-system-x86-14634 [002] 30483.401319: kvm_msr: msr_read 42d = 0x0
qemu-system-x86-14634 [002] 30483.401321: kvm_msr: msr_read 431 = 0x0
qemu-system-x86-14634 [002] 30483.401323: kvm_msr: msr_read 435 = 0x0
qemu-system-x86-14634 [002] 30483.401325: kvm_msr: msr_read 439 = 0x0
qemu-system-x86-14634 [002] 30483.401328: kvm_msr: msr_read 43d = 0x0
qemu-system-x86-14634 [002] 30483.401330: kvm_msr: msr_read 441 = 0x0
qemu-system-x86-14634 [002] 30483.401332: kvm_msr: msr_read 445 = 0x0
qemu-system-x86-14634 [002] 30483.401334: kvm_msr: msr_read 449 = 0x0
qemu-system-x86-14634 [002] 30483.401336: kvm_msr: msr_read 44d = 0x0
qemu-system-x86-14634 [002] 30483.401339: kvm_msr: msr_read 451 = 0x0
qemu-system-x86-14634 [002] 30483.401341: kvm_msr: msr_read 455 = 0x0
qemu-system-x86-14634 [002] 30483.401343: kvm_msr: msr_read 459 = 0x0
qemu-system-x86-14634 [002] 30483.401345: kvm_msr: msr_read 45d = 0x0
qemu-system-x86-14634 [002] 30483.401347: kvm_msr: msr_read 461 = 0x0
qemu-system-x86-14634 [002] 30483.401350: kvm_msr: msr_read 465 = 0x0
qemu-system-x86-14634 [002] 30483.401352: kvm_msr: msr_read 469 = 0x0
qemu-system-x86-14634 [002] 30483.401354: kvm_msr: msr_read 46d = 0x0
qemu-system-x86-14634 [002] 30483.401356: kvm_msr: msr_read 471 = 0x0
qemu-system-x86-14634 [002] 30483.401359: kvm_msr: msr_read 475 = 0x0
qemu-system-x86-14634 [002] 30483.401361: kvm_msr: msr_read 479 = 0x0
qemu-system-x86-14634 [002] 30483.401363: kvm_msr: msr_read 47d = 0x0
BSOD
next prev parent reply other threads:[~2011-11-17 11:49 UTC|newest]
Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-11-04 10:48 [Qemu-devel] Windows 7 shutdown causes BSOD Stefan Hajnoczi
2011-11-04 11:25 ` Stefan Hajnoczi
2011-11-15 13:17 ` Stefan Hajnoczi
2011-11-16 10:14 ` hkran
2011-11-16 10:48 ` Stefan Hajnoczi
2011-11-16 10:51 ` Gleb Natapov
[not found] ` <4EC4A9DB.2060406@linux.vnet.ibm.com>
2011-11-17 6:37 ` Gleb Natapov
2011-11-17 10:55 ` hkran [this message]
2011-11-17 11:08 ` Gleb Natapov
2011-11-18 9:11 ` hkran
2011-11-22 8:45 ` Gleb Natapov
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4EC4E812.2010403@linux.vnet.ibm.com \
--to=hkran@linux.vnet.ibm.com \
--cc=caobingbu@gmail.com \
--cc=gleb@redhat.com \
--cc=mjt@tls.msk.ru \
--cc=qemu-devel@nongnu.org \
--cc=stefanha@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).