From: stan hoeppner <stan@hardwarefreak.com>
To: Dave Chinner <david@fromorbit.com>
Cc: xfs@oss.sgi.com
Subject: Re: storage, libaio, or XFS problem? 3.4.26
Date: Sun, 07 Sep 2014 00:23:03 -0500 [thread overview]
Message-ID: <540BEBB7.7020306@hardwarefreak.com> (raw)
In-Reply-To: <20140902221915.GK20518@dastard>
On 09/02/2014 05:19 PM, Dave Chinner wrote:
> On Tue, Sep 02, 2014 at 12:15:05PM -0500, stan hoeppner wrote:
>> On 09/01/2014 06:45 PM, Dave Chinner wrote:
>>> On Sun, Aug 31, 2014 at 10:36:25PM -0500, stan hoeppner wrote:
>>>> On 08/31/2014 06:57 PM, Dave Chinner wrote:
>>>>> On Fri, Aug 29, 2014 at 09:55:53PM -0500, Stan Hoeppner wrote:
>>>>>> Have you played with bcache yet?
>>>>>
>>>>> Enough to scare me. So many ways for things to go wrong, no easy way
>>>>> to recover when things go wrong. And that's before I even get to
>>>>> performance warts, like having systems stall completely because
>>>>> there's tens or hundreds of GB of 4k random writes that have to be
>>>>> flushed to slow SATA RAID6 in the cache....
>>>>
>>>> Yikes. I hadn't yet heard such opinions expressed. By go wrong I
>>>> assume you mean the btrees or cached sector data getting broken, corrupted?
>>>
>>> bcache is a complex filesystem hidden inside a block device. If
>>> bcache goes AWOL, so does the all the data on your block device.
>>> Need I say more?
>>
>> So it's no different in that regard than the black box implementations
>> such as LSI's CacheCade and various SAN vendor SSD caching
>> implementations. Or are you saying the bcache code complexity is so
>> much greater that failure is more likely that the vendor implementations?
>
> No, not the code complexity in particular. It's more that compared
> to vendor SSD caching implementations there's an awful lot less
> testing and validation, and people tend to use random, unreliable
> hardware for cache devices. It's great when it works, but the
> configuration and validation of correct behaviour in error
> conditions falls to the user...
Understood. I'm seeing the potential need for a future contract with
Kent if we decide to go forward with bcache. He could advise on a
testing and validation regimen, optimizing for the workload, and
providing code fixes or features to overcome problems. Attempting to
use something so new as bcache in a 24x7 commercial workload likely
needs author support.
>>> screen is your friend when it comes to keeping remote shells
>>> active as the network comes and goes. VPN drops out, just bring it
>>> back up when you need it and reconnect to the remote screen instance
>>> and it's like you never left....
>>
>> Thanks for this tip. I'd heard of screen before but never used it. I
>> will say the man page is a bit intimidating for such an apparently
>> simple tool...
>
> Yeah, I use about 0.0001% of what screen can do. It could lose most
> of it's functionality and I wouldn't notice or care. tmux is another
> option for this functionality, but I've never used it because I
> found out about screen first...
I'd guess there are many utils out there used in the same way.
I have some more information regarding the AIO issue. I fired up the
test harness and it ran for 30 hours at 706 MB/s avg write rate, 303
MB/s per LUN, nearly flawlessly, less than 0.01% buffer loss, and avg IO
times were less than 0.5 seconds. Then the app crashed and I found the
following in dmesg. I had to "hard reset" the box due to the shrapnel.
There are no IO errors of any kind leading up to the forced shutdown.
I assume the inode update and streamRT-sa hung task traces are a result
of the forced shutdown, not a cause of it. In lieu of an xfs_repair
with a version newer than I'm able to install, any ideas what caused the
forced shutdown after 30 hours, given there are no errors preceding it?
Sep 6 06:33:33 Anguish-ssu-1 kernel: [288087.334863] XFS (dm-5):
xfs_do_force_shutdown(0x8) called from line 3732 of file
fs/xfs/xfs_bmap.c. Return address = 0xffffffffa02009a6
Sep 6 06:33:42 Anguish-ssu-1 kernel: [288096.220920] XFS (dm-5): failed
to update timestamps for inode 0x2ffc9caae
Sep 6 06:33:48 Anguish-ssu-1 kernel: [288102.492641] XFS (dm-5): failed
to update timestamps for inode 0x97b7566dd
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599412] INFO: task
streamRT-sa:14706 blocked for more than 120 seconds.
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599414] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599416] streamRT-sa D
ffff883f3c018408 0 14706 14051 0x00000004
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599420] ffff883e6fc09b28
0000000000000086 0000000000000000 ffff8840666f5180
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599425] 0000000000000000
0000000000000000 00000000000122c0 00000000000122c0
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599428] ffff883e6fc09fd8
ffff883e6fc08000 00000000000122c0 ffff883e6fc08000
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599432] Call Trace:
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599441]
[<ffffffff814f5fd7>] schedule+0x64/0x66
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599443]
[<ffffffff814f66ec>] rwsem_down_failed_common+0xdb/0x10d
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599446]
[<ffffffff814f6731>] rwsem_down_write_failed+0x13/0x15
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599451]
[<ffffffff81261913>] call_rwsem_down_write_failed+0x13/0x20
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599454]
[<ffffffff814f5458>] ? down_write+0x25/0x27
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599466]
[<ffffffffa01e75e4>] xfs_ilock+0x4f/0xb4 [xfs]
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599472]
[<ffffffffa01e40e5>] xfs_rw_ilock+0x2c/0x33 [xfs]
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599476]
[<ffffffff814f6ac6>] ? _raw_spin_unlock_irq+0x27/0x32
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599481]
[<ffffffffa01e4519>] xfs_file_aio_write_checks+0x41/0xfe [xfs]
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599487]
[<ffffffffa01e46ff>] xfs_file_dio_aio_write+0x103/0x1fc [xfs]
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599493]
[<ffffffffa01e4ac3>] xfs_file_aio_write+0x152/0x1b5 [xfs]
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599499]
[<ffffffffa01e4971>] ? xfs_file_buffered_aio_write+0x179/0x179 [xfs]
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599503]
[<ffffffff81133694>] aio_rw_vect_retry+0x85/0x18a
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599505]
[<ffffffff8113360f>] ? aio_fsync+0x29/0x29
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599508]
[<ffffffff81134c10>] aio_run_iocb+0x7b/0x149
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599510]
[<ffffffff81134fe9>] io_submit_one+0x199/0x1f3
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599513]
[<ffffffff8113513d>] do_io_submit+0xfa/0x271
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599516]
[<ffffffff811352c4>] sys_io_submit+0x10/0x12
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599519]
[<ffffffff814fc912>] system_call_fastpath+0x16/0x1b
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599521] INFO: task
streamRT-sa:14713 blocked for more than 120 seconds.
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599523] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599524] streamRT-sa D
ffff883b4f52ea48 0 14713 14051 0x00000004
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599527] ffff883e74af9b28
0000000000000086 0000000000000000 ffff884066622140
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599530] 0000000000000000
0000000000000000 00000000000122c0 00000000000122c0
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599534] ffff883e74af9fd8
ffff883e74af8000 00000000000122c0 ffff883e74af8000
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599537] Call Trace:
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599540]
[<ffffffff814f5fd7>] schedule+0x64/0x66
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599542]
[<ffffffff814f66ec>] rwsem_down_failed_common+0xdb/0x10d
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599544]
[<ffffffff814f6731>] rwsem_down_write_failed+0x13/0x15
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599547]
[<ffffffff81261913>] call_rwsem_down_write_failed+0x13/0x20
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599549]
[<ffffffff814f5458>] ? down_write+0x25/0x27
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599555]
[<ffffffffa01e75e4>] xfs_ilock+0x4f/0xb4 [xfs]
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599561]
[<ffffffffa01e40e5>] xfs_rw_ilock+0x2c/0x33 [xfs]
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599563]
[<ffffffff814f6ac6>] ? _raw_spin_unlock_irq+0x27/0x32
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599569]
[<ffffffffa01e4519>] xfs_file_aio_write_checks+0x41/0xfe [xfs]
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599575]
[<ffffffffa01e46ff>] xfs_file_dio_aio_write+0x103/0x1fc [xfs]
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599580]
[<ffffffffa01e4ac3>] xfs_file_aio_write+0x152/0x1b5 [xfs]
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599586]
[<ffffffffa01e4971>] ? xfs_file_buffered_aio_write+0x179/0x179 [xfs]
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599589]
[<ffffffff81133694>] aio_rw_vect_retry+0x85/0x18a
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599591]
[<ffffffff8113360f>] ? aio_fsync+0x29/0x29
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599593]
[<ffffffff81134c10>] aio_run_iocb+0x7b/0x149
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599596]
[<ffffffff81134fe9>] io_submit_one+0x199/0x1f3
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599598]
[<ffffffff8113513d>] do_io_submit+0xfa/0x271
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599601]
[<ffffffff811352c4>] sys_io_submit+0x10/0x12
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599603]
[<ffffffff814fc912>] system_call_fastpath+0x16/0x1b
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599605] INFO: task
streamRT-sa:14723 blocked for more than 120 seconds.
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599607] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599608] streamRT-sa D
ffff883e754b2b88 0 14723 14051 0x00000004
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599610] ffff883e6fca3b28
0000000000000086 0000000000000000 ffff8840662521c0
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599614] 0000000000000000
0000000000000000 00000000000122c0 00000000000122c0
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599617] ffff883e6fca3fd8
ffff883e6fca2000 00000000000122c0 ffff883e6fca2000
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599620] Call Trace:
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599623]
[<ffffffff814f5fd7>] schedule+0x64/0x66
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599625]
[<ffffffff814f66ec>] rwsem_down_failed_common+0xdb/0x10d
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599628]
[<ffffffff814f6731>] rwsem_down_write_failed+0x13/0x15
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599630]
[<ffffffff81261913>] call_rwsem_down_write_failed+0x13/0x20
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599632]
[<ffffffff814f5458>] ? down_write+0x25/0x27
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599638]
[<ffffffffa01e75e4>] xfs_ilock+0x4f/0xb4 [xfs]
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599644]
[<ffffffffa01e40e5>] xfs_rw_ilock+0x2c/0x33 [xfs]
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599646]
[<ffffffff814f6ac6>] ? _raw_spin_unlock_irq+0x27/0x32
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599652]
[<ffffffffa01e4519>] xfs_file_aio_write_checks+0x41/0xfe [xfs]
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599657]
[<ffffffffa01e46ff>] xfs_file_dio_aio_write+0x103/0x1fc [xfs]
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599663]
[<ffffffffa01e4ac3>] xfs_file_aio_write+0x152/0x1b5 [xfs]
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599669]
[<ffffffffa01e4971>] ? xfs_file_buffered_aio_write+0x179/0x179 [xfs]
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599671]
[<ffffffff81133694>] aio_rw_vect_retry+0x85/0x18a
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599674]
[<ffffffff8113360f>] ? aio_fsync+0x29/0x29
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599676]
[<ffffffff81134c10>] aio_run_iocb+0x7b/0x149
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599678]
[<ffffffff81134fe9>] io_submit_one+0x199/0x1f3
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599681]
[<ffffffff8113513d>] do_io_submit+0xfa/0x271
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599684]
[<ffffffff811352c4>] sys_io_submit+0x10/0x12
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599686]
[<ffffffff814fc912>] system_call_fastpath+0x16/0x1b
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599688] INFO: task
streamRT-sa:14730 blocked for more than 120 seconds.
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599689] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599691] streamRT-sa D
ffff883dc2360388 0 14730 14051 0x00000004
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599693] ffff883e6fde1b28
0000000000000086 0000000000000000 ffff884066043080
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599696] 0000000000000000
0000000000000000 00000000000122c0 00000000000122c0
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599700] ffff883e6fde1fd8
ffff883e6fde0000 00000000000122c0 ffff883e6fde0000
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599703] Call Trace:
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599705]
[<ffffffff814f5fd7>] schedule+0x64/0x66
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599708]
[<ffffffff814f66ec>] rwsem_down_failed_common+0xdb/0x10d
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599710]
[<ffffffff814f6731>] rwsem_down_write_failed+0x13/0x15
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599712]
[<ffffffff81261913>] call_rwsem_down_write_failed+0x13/0x20
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599715]
[<ffffffff814f5458>] ? down_write+0x25/0x27
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599720]
[<ffffffffa01e75e4>] xfs_ilock+0x4f/0xb4 [xfs]
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599726]
[<ffffffffa01e40e5>] xfs_rw_ilock+0x2c/0x33 [xfs]
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599728]
[<ffffffff814f6ac6>] ? _raw_spin_unlock_irq+0x27/0x32
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599734]
[<ffffffffa01e4519>] xfs_file_aio_write_checks+0x41/0xfe [xfs]
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599740]
[<ffffffffa01e46ff>] xfs_file_dio_aio_write+0x103/0x1fc [xfs]
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599745]
[<ffffffffa01e4ac3>] xfs_file_aio_write+0x152/0x1b5 [xfs]
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599751]
[<ffffffffa01e4971>] ? xfs_file_buffered_aio_write+0x179/0x179 [xfs]
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599754]
[<ffffffff81133694>] aio_rw_vect_retry+0x85/0x18a
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599756]
[<ffffffff8113360f>] ? aio_fsync+0x29/0x29
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599758]
[<ffffffff81134c10>] aio_run_iocb+0x7b/0x149
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599761]
[<ffffffff81134fe9>] io_submit_one+0x199/0x1f3
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599763]
[<ffffffff8113513d>] do_io_submit+0xfa/0x271
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599766]
[<ffffffff811352c4>] sys_io_submit+0x10/0x12
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599768]
[<ffffffff814fc912>] system_call_fastpath+0x16/0x1b
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599770] INFO: task
streamRT-sa:14733 blocked for more than 120 seconds.
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599771] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599773] streamRT-sa D
ffff883e7555cb08 0 14733 14051 0x00000004
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599775] ffff883e7389db28
0000000000000086 0000000000000000 ffff88406663a040
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599778] 0000000000000000
0000000000000000 00000000000122c0 00000000000122c0
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599782] ffff883e7389dfd8
ffff883e7389c000 00000000000122c0 ffff883e7389c000
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599785] Call Trace:
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599787]
[<ffffffff814f5fd7>] schedule+0x64/0x66
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599790]
[<ffffffff814f66ec>] rwsem_down_failed_common+0xdb/0x10d
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599792]
[<ffffffff814f6731>] rwsem_down_write_failed+0x13/0x15
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599794]
[<ffffffff81261913>] call_rwsem_down_write_failed+0x13/0x20
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599797]
[<ffffffff814f5458>] ? down_write+0x25/0x27
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599802]
[<ffffffffa01e75e4>] xfs_ilock+0x4f/0xb4 [xfs]
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599808]
[<ffffffffa01e40e5>] xfs_rw_ilock+0x2c/0x33 [xfs]
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599811]
[<ffffffff814f6ac6>] ? _raw_spin_unlock_irq+0x27/0x32
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599816]
[<ffffffffa01e4519>] xfs_file_aio_write_checks+0x41/0xfe [xfs]
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599822]
[<ffffffffa01e46ff>] xfs_file_dio_aio_write+0x103/0x1fc [xfs]
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599827]
[<ffffffffa01e4ac3>] xfs_file_aio_write+0x152/0x1b5 [xfs]
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599833]
[<ffffffffa01e4971>] ? xfs_file_buffered_aio_write+0x179/0x179 [xfs]
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599836]
[<ffffffff81133694>] aio_rw_vect_retry+0x85/0x18a
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599838]
[<ffffffff8113360f>] ? aio_fsync+0x29/0x29
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599840]
[<ffffffff81134c10>] aio_run_iocb+0x7b/0x149
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599843]
[<ffffffff81134fe9>] io_submit_one+0x199/0x1f3
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599845]
[<ffffffff8113513d>] do_io_submit+0xfa/0x271
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599848]
[<ffffffff811352c4>] sys_io_submit+0x10/0x12
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599850]
[<ffffffff814fc912>] system_call_fastpath+0x16/0x1b
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599852] INFO: task
streamRT-sa:14736 blocked for more than 120 seconds.
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599853] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599855] streamRT-sa D
ffff883e73915448 0 14736 14051 0x00000004
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599857] ffff883e73bb5b28
0000000000000086 0000000000000000 ffff884066709080
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599860] 000000025600a331
0000000000000000 00000000000122c0 00000000000122c0
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599864] ffff883e73bb5fd8
ffff883e73bb4000 00000000000122c0 ffff883e73bb4000
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599867] Call Trace:
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599870]
[<ffffffff814f5fd7>] schedule+0x64/0x66
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599872]
[<ffffffff814f66ec>] rwsem_down_failed_common+0xdb/0x10d
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599874]
[<ffffffff814f6731>] rwsem_down_write_failed+0x13/0x15
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599877]
[<ffffffff81261913>] call_rwsem_down_write_failed+0x13/0x20
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599879]
[<ffffffff814f5458>] ? down_write+0x25/0x27
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599885]
[<ffffffffa01e75e4>] xfs_ilock+0x4f/0xb4 [xfs]
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599890]
[<ffffffffa01e40e5>] xfs_rw_ilock+0x2c/0x33 [xfs]
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599892]
[<ffffffff814f6ac6>] ? _raw_spin_unlock_irq+0x27/0x32
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599898]
[<ffffffffa01e4519>] xfs_file_aio_write_checks+0x41/0xfe [xfs]
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599904]
[<ffffffffa01e46ff>] xfs_file_dio_aio_write+0x103/0x1fc [xfs]
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599909]
[<ffffffffa01e4ac3>] xfs_file_aio_write+0x152/0x1b5 [xfs]
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599915]
[<ffffffffa01e4971>] ? xfs_file_buffered_aio_write+0x179/0x179 [xfs]
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599918]
[<ffffffff81133694>] aio_rw_vect_retry+0x85/0x18a
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599920]
[<ffffffff8113360f>] ? aio_fsync+0x29/0x29
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599922]
[<ffffffff81134c10>] aio_run_iocb+0x7b/0x149
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599925]
[<ffffffff81134fe9>] io_submit_one+0x199/0x1f3
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599927]
[<ffffffff8113513d>] do_io_submit+0xfa/0x271
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599930]
[<ffffffff811352c4>] sys_io_submit+0x10/0x12
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599932]
[<ffffffff814fc912>] system_call_fastpath+0x16/0x1b
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599934] INFO: task
streamRT-sa:14738 blocked for more than 120 seconds.
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599936] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599937] streamRT-sa D
ffff883f7c605488 0 14738 14051 0x00000004
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599939] ffff883c4cda7b28
0000000000000086 0000000000000000 ffff8840667bd1c0
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599943] 0000000000000000
0000000000000000 00000000000122c0 00000000000122c0
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599946] ffff883c4cda7fd8
ffff883c4cda6000 00000000000122c0 ffff883c4cda6000
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599949] Call Trace:
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599952]
[<ffffffff814f5fd7>] schedule+0x64/0x66
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599954]
[<ffffffff814f66ec>] rwsem_down_failed_common+0xdb/0x10d
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599956]
[<ffffffff814f6731>] rwsem_down_write_failed+0x13/0x15
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599959]
[<ffffffff81261913>] call_rwsem_down_write_failed+0x13/0x20
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599961]
[<ffffffff814f5458>] ? down_write+0x25/0x27
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599967]
[<ffffffffa01e75e4>] xfs_ilock+0x4f/0xb4 [xfs]
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599972]
[<ffffffffa01e40e5>] xfs_rw_ilock+0x2c/0x33 [xfs]
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599975]
[<ffffffff814f6ac6>] ? _raw_spin_unlock_irq+0x27/0x32
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599980]
[<ffffffffa01e4519>] xfs_file_aio_write_checks+0x41/0xfe [xfs]
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599986]
[<ffffffffa01e46ff>] xfs_file_dio_aio_write+0x103/0x1fc [xfs]
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599991]
[<ffffffffa01e4ac3>] xfs_file_aio_write+0x152/0x1b5 [xfs]
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.599997]
[<ffffffffa01e4971>] ? xfs_file_buffered_aio_write+0x179/0x179 [xfs]
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600000]
[<ffffffff81133694>] aio_rw_vect_retry+0x85/0x18a
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600002]
[<ffffffff8113360f>] ? aio_fsync+0x29/0x29
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600004]
[<ffffffff81134c10>] aio_run_iocb+0x7b/0x149
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600007]
[<ffffffff81134fe9>] io_submit_one+0x199/0x1f3
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600009]
[<ffffffff8113513d>] do_io_submit+0xfa/0x271
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600012]
[<ffffffff811352c4>] sys_io_submit+0x10/0x12
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600014]
[<ffffffff814fc912>] system_call_fastpath+0x16/0x1b
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600016] INFO: task
streamRT-sa:14739 blocked for more than 120 seconds.
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600018] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600019] streamRT-sa D
ffff883e75536a08 0 14739 14051 0x00000004
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600021] ffff883b4f411b28
0000000000000086 0000000000000000 ffff884066739140
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600025] 0000000000000000
0000000000000000 00000000000122c0 00000000000122c0
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600028] ffff883b4f411fd8
ffff883b4f410000 00000000000122c0 ffff883b4f410000
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600031] Call Trace:
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600034]
[<ffffffff814f5fd7>] schedule+0x64/0x66
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600036]
[<ffffffff814f66ec>] rwsem_down_failed_common+0xdb/0x10d
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600038]
[<ffffffff814f6731>] rwsem_down_write_failed+0x13/0x15
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600041]
[<ffffffff81261913>] call_rwsem_down_write_failed+0x13/0x20
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600043]
[<ffffffff814f5458>] ? down_write+0x25/0x27
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600048]
[<ffffffffa01e75e4>] xfs_ilock+0x4f/0xb4 [xfs]
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600054]
[<ffffffffa01e40e5>] xfs_rw_ilock+0x2c/0x33 [xfs]
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600056]
[<ffffffff814f6ac6>] ? _raw_spin_unlock_irq+0x27/0x32
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600062]
[<ffffffffa01e4519>] xfs_file_aio_write_checks+0x41/0xfe [xfs]
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600068]
[<ffffffffa01e46ff>] xfs_file_dio_aio_write+0x103/0x1fc [xfs]
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600073]
[<ffffffffa01e4ac3>] xfs_file_aio_write+0x152/0x1b5 [xfs]
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600079]
[<ffffffffa01e4971>] ? xfs_file_buffered_aio_write+0x179/0x179 [xfs]
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600082]
[<ffffffff81133694>] aio_rw_vect_retry+0x85/0x18a
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600084]
[<ffffffff8113360f>] ? aio_fsync+0x29/0x29
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600086]
[<ffffffff81134c10>] aio_run_iocb+0x7b/0x149
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600089]
[<ffffffff81134fe9>] io_submit_one+0x199/0x1f3
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600091]
[<ffffffff8113513d>] do_io_submit+0xfa/0x271
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600094]
[<ffffffff811352c4>] sys_io_submit+0x10/0x12
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600096]
[<ffffffff814fc912>] system_call_fastpath+0x16/0x1b
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600099] INFO: task
streamRT-sa:14768 blocked for more than 120 seconds.
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600100] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600101] streamRT-sa D
ffff883b5f120308 0 14768 14051 0x00000004
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600104] ffff883cca73bb28
0000000000000086 0000000000000000 ffffffff81813020
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600107] 0000000000000000
0000000000000000 00000000000122c0 00000000000122c0
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600110] ffff883cca73bfd8
ffff883cca73a000 00000000000122c0 ffff883cca73a000
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600113] Call Trace:
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600116]
[<ffffffff814f5fd7>] schedule+0x64/0x66
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600118]
[<ffffffff814f66ec>] rwsem_down_failed_common+0xdb/0x10d
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600120]
[<ffffffff814f6731>] rwsem_down_write_failed+0x13/0x15
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600123]
[<ffffffff81261913>] call_rwsem_down_write_failed+0x13/0x20
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600125]
[<ffffffff814f5458>] ? down_write+0x25/0x27
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600131]
[<ffffffffa01e75e4>] xfs_ilock+0x4f/0xb4 [xfs]
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600136]
[<ffffffffa01e40e5>] xfs_rw_ilock+0x2c/0x33 [xfs]
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600139]
[<ffffffff814f6ac6>] ? _raw_spin_unlock_irq+0x27/0x32
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600144]
[<ffffffffa01e4519>] xfs_file_aio_write_checks+0x41/0xfe [xfs]
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600150]
[<ffffffffa01e46ff>] xfs_file_dio_aio_write+0x103/0x1fc [xfs]
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600156]
[<ffffffffa01e4ac3>] xfs_file_aio_write+0x152/0x1b5 [xfs]
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600161]
[<ffffffffa01e4971>] ? xfs_file_buffered_aio_write+0x179/0x179 [xfs]
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600164]
[<ffffffff81133694>] aio_rw_vect_retry+0x85/0x18a
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600166]
[<ffffffff8113360f>] ? aio_fsync+0x29/0x29
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600168]
[<ffffffff81134c10>] aio_run_iocb+0x7b/0x149
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600171]
[<ffffffff81134fe9>] io_submit_one+0x199/0x1f3
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600173]
[<ffffffff8113513d>] do_io_submit+0xfa/0x271
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600176]
[<ffffffff811352c4>] sys_io_submit+0x10/0x12
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600178]
[<ffffffff814fc912>] system_call_fastpath+0x16/0x1b
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600180] INFO: task
streamRT-sa:14789 blocked for more than 120 seconds.
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600181] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600183] streamRT-sa D
ffff883cca430b08 0 14789 14051 0x00000004
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600185] ffff883f3d9c3b28
0000000000000086 0000000000000000 ffff884066739140
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600188] 0000000000000000
0000000000000000 00000000000122c0 00000000000122c0
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600192] ffff883f3d9c3fd8
ffff883f3d9c2000 00000000000122c0 ffff883f3d9c2000
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600195] Call Trace:
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600197]
[<ffffffff814f5fd7>] schedule+0x64/0x66
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600200]
[<ffffffff814f66ec>] rwsem_down_failed_common+0xdb/0x10d
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600202]
[<ffffffff814f6731>] rwsem_down_write_failed+0x13/0x15
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600204]
[<ffffffff81261913>] call_rwsem_down_write_failed+0x13/0x20
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600207]
[<ffffffff814f5458>] ? down_write+0x25/0x27
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600212]
[<ffffffffa01e75e4>] xfs_ilock+0x4f/0xb4 [xfs]
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600218]
[<ffffffffa01e40e5>] xfs_rw_ilock+0x2c/0x33 [xfs]
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600220]
[<ffffffff814f6ac6>] ? _raw_spin_unlock_irq+0x27/0x32
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600226]
[<ffffffffa01e4519>] xfs_file_aio_write_checks+0x41/0xfe [xfs]
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600231]
[<ffffffffa01e46ff>] xfs_file_dio_aio_write+0x103/0x1fc [xfs]
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600237]
[<ffffffffa01e4ac3>] xfs_file_aio_write+0x152/0x1b5 [xfs]
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600243]
[<ffffffffa01e4971>] ? xfs_file_buffered_aio_write+0x179/0x179 [xfs]
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600245]
[<ffffffff81133694>] aio_rw_vect_retry+0x85/0x18a
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600248]
[<ffffffff8113360f>] ? aio_fsync+0x29/0x29
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600250]
[<ffffffff81134c10>] aio_run_iocb+0x7b/0x149
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600252]
[<ffffffff81134fe9>] io_submit_one+0x199/0x1f3
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600255]
[<ffffffff8113513d>] do_io_submit+0xfa/0x271
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600258]
[<ffffffff811352c4>] sys_io_submit+0x10/0x12
Sep 6 06:35:41 Anguish-ssu-1 kernel: [288215.600260]
[<ffffffff814fc912>] system_call_fastpath+0x16/0x1b
Sep 6 15:42:02 Anguish-ssu-1 kernel: [320925.045195] SysRq : Resetting
Thanks,
Stan
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
next prev parent reply other threads:[~2014-09-07 5:23 UTC|newest]
Thread overview: 19+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-08-26 6:18 storage, libaio, or XFS problem? 3.4.26 Stan Hoeppner
2014-08-26 6:25 ` Stan Hoeppner
2014-08-26 7:53 ` Dave Chinner
2014-08-26 17:19 ` Stan Hoeppner
2014-08-28 0:32 ` Dave Chinner
2014-08-28 22:31 ` Stan Hoeppner
2014-08-28 23:08 ` Dave Chinner
2014-08-29 16:38 ` Stan Hoeppner
2014-08-29 23:55 ` Dave Chinner
2014-08-30 2:55 ` Stan Hoeppner
2014-08-31 23:57 ` Dave Chinner
2014-09-01 3:36 ` stan hoeppner
2014-09-01 23:45 ` Dave Chinner
2014-09-02 17:15 ` stan hoeppner
2014-09-02 22:19 ` Dave Chinner
2014-09-07 5:23 ` stan hoeppner [this message]
2014-09-07 23:39 ` Dave Chinner
2014-09-08 15:13 ` stan hoeppner
2014-09-20 19:47 ` stan hoeppner
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=540BEBB7.7020306@hardwarefreak.com \
--to=stan@hardwarefreak.com \
--cc=david@fromorbit.com \
--cc=xfs@oss.sgi.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.