All of lore.kernel.org
 help / color / mirror / Atom feed
From: Ian Campbell <ijc-KcIKpvwj1kUDXYZnReoRVg@public.gmane.org>
To: "J. Bruce Fields" <bfields@fieldses.org>
Cc: Tom Tucker <tom@opengridcomputing.com>,
	Trond Myklebust <trond.myklebust@fys.uio.no>,
	Grant Coady <gcoady.lk@gmail.com>,
	linux-kernel@vger.kernel.org, neilb@suse.de,
	linux-nfs@vger.kernel.org
Subject: Re: NFS regression? Odd delays and lockups accessing an NFS export.
Date: Sat, 27 Sep 2008 11:16:26 +0100	[thread overview]
Message-ID: <1222510586.3949.45.camel@localhost.localdomain> (raw)
In-Reply-To: <20080927035415.GC12765@fieldses.org>

[-- Attachment #1: Type: text/plain, Size: 6946 bytes --]

On Fri, 2008-09-26 at 23:54 -0400, J. Bruce Fields wrote:
> OK, so apologies, but this has been a long thread, and maybe we could
> use a summary of the symptoms and the results so far.  I think you said
> 2.6.24 or .25 was the last you're *positive* was good?

2.6.24.x was good 2.6.25.y was not. The Debian packages don't include
the stable release numbers in their version so I'm unsure of x and y,
Judging from the changelog entries I think 2.6.24.3 and 2.6.25.10. 

Since I have been building my own kernels I have seen repros with:

a551b98d5f6fce5897d497abd8bfb262efb33d2a Merge branch 'for-linus' of
git://git.kernel.dk/linux-2.6-block (was tip of git at the time I tested
it a while ago)

c02a79dedc7f3c3d4fdbb5eb2000cacea5df4cde v2.6.26.5

daedfbe2a67628a40076a6c75fb945c60f608a2e NFS: Ensure we zap only the
access and acl caches when setting new acls (backport of
f41f741838480aeaa3a189cff6e210503cf9c42d to 2.6.26.y stream, identified
by bisect, possibly/probably incorrectly)

f41f741838480aeaa3a189cff6e210503cf9c42d NFS: Ensure we zap only the
access and acl caches when setting new acls (original trunk commit of
above)

2e96d2867245668dbdb973729288cf69b9fafa66 NFS: Fix a warning in
nfs4_async_handle_error (commit before
f41f741838480aeaa3a189cff6e210503cf9c42d)

I did not see the problem with bce7f793daec3e65ec5c5705d2457b81fe7b5725
(v2.6.26) in 4 days 19 hours. Due to the apparent mis-bisection leading
to daedfbe2a67628a40076a6c75fb945c60f608a2e this result is in doubt, I'm
currently retesting this kernel.

The bisect was restricted to paths under fs/nfs* and net/sunrpc. I've
gone over the range v2.6.26..daedfbe2a67628a40076a6c75fb945c60f608a2e
and nothing jumps out at me. If my retest of 2.6.26 shows it to be OK
again I'll bisect over the non-NFS changesets in that range.

The symptoms are a hang of any access to an NFS mounted path, although
the rest of the system continues to work. 

Other potentially interesting info cribbed from earlier postings:

While the hang is occurring rpc_debug on the server gives nothing, on
the client gives:
[144741.637997] -pid- proc flgs status -client- -prog- --rqstp- -timeout -rpcwait -action- ---ops-- 
[144741.637997]  3439 0004 0080    -11 f3f48200 100003 f7770000    0 xprt_sending fa0ae88e fa0bddf4
[144741.637997]  3438 0001 00a0      0 f77f2a00 100003 f77700d0 5000 xprt_pending fa0ae88e fa0bddf4

The mount points are
        hopkins:/storage/music /storage/music nfs rw,nosuid,vers=3,rsize=32768,wsize=32768,namlen=255,hard,intr,proto=tcp,timeo=600,retrans=2,sec=sys,mountproto=udp,addr=192.168.1.6 0 0
        hopkins:/storage/mythtv/recordings /var/lib/mythtv/recordings nfs rw,nosuid,vers=3,rsize=32768,wsize=32768,namlen=255,acregmin=0,acregmax=0,acdirmin=0,acdirmax=0,hard,intr,proto=tcp,timeo=600,retrans=2,sec=sys,mountproto=udp,addr=192.168.1.6 0 0
        hopkins:/var/lib/mythvideo /var/lib/mythvideo nfs rw,nosuid,vers=3,rsize=32768,wsize=32768,namlen=255,acregmin=0,acregmax=0,acdirmin=0,acdirmax=0,hard,intr,proto=tcp,timeo=600,retrans=2,sec=sys,mountproto=udp,addr=192.168.1.6 0 0
        hopkins:/storage/home/ijc /home/ijc nfs rw,vers=3,rsize=131072,wsize=131072,namlen=255,hard,nointr,proto=tcp,timeo=600,retrans=2,sec=sys,mountproto=udp,addr=192.168.1.6 0 0
all of them seem to be effected simultaneously (but I'll check to make double sure next time it occurs).

I've seen hung task backtraces from effected processes, e.g. 
        Aug  4 06:27:28 iranon kernel: [137969.382277] INFO: task mythbackend:3161 blocked for more than 120 seconds.
        Aug  4 06:27:28 iranon kernel: [137969.382287] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
        Aug  4 06:27:28 iranon kernel: [137969.382291] mythbackend   D 00005dfc     0  3161   1
        Aug  4 06:27:28 iranon kernel: [137969.382295]        f2006c70 00000082 4b05cc7e 00005dfc f2006df0 c1c0e920 00000000 f7273390 
        Aug  4 06:27:28 iranon kernel: [137969.382301]        0000c8b4 00000000 00000001 f7273398 00000282 00000202 f71b0ab0 f200ddf0 
        Aug  4 06:27:28 iranon kernel: [137969.382306]        c1c012bc fa18f0a6 fa18f0bf c02b45a7 f71b0ab0 00000000 f200de0c fa18f0a6 
        Aug  4 06:27:28 iranon kernel: [137969.382311] Call Trace:
        Aug  4 06:27:28 iranon kernel: [137969.382347]  [<fa18f0a6>] nfs_wait_schedule+0x0/0x1e [nfs]
        Aug  4 06:27:28 iranon kernel: [137969.382384]  [<fa18f0bf>] nfs_wait_schedule+0x19/0x1e [nfs]
        Aug  4 06:27:28 iranon kernel: [137969.382399]  [<c02b45a7>] __wait_on_bit_lock+0x2a/0x52
        Aug  4 06:27:28 iranon kernel: [137969.382407]  [<fa18f0a6>] nfs_wait_schedule+0x0/0x1e [nfs]
        Aug  4 06:27:28 iranon kernel: [137969.382421]  [<c02b462e>] out_of_line_wait_on_bit_lock+0x5f/0x67
        Aug  4 06:27:28 iranon kernel: [137969.382429]  [<c0133a1d>] wake_bit_function+0x0/0x3c
        Aug  4 06:27:28 iranon kernel: [137969.382441]  [<fa18f16e>] __nfs_revalidate_inode+0xaa/0x211 [nfs]
        Aug  4 06:27:28 iranon kernel: [137969.382458]  [<c017bac0>] do_lookup+0x53/0x145
        Aug  4 06:27:28 iranon kernel: [137969.382466]  [<c0187609>] mntput_no_expire+0x11/0x64
        Aug  4 06:27:28 iranon kernel: [137969.382472]  [<c017d971>] __link_path_walk+0xa71/0xb65
        Aug  4 06:27:28 iranon kernel: [137969.382477]  [<c017bac0>] do_lookup+0x53/0x145
        Aug  4 06:27:28 iranon kernel: [137969.382483]  [<c0187609>] mntput_no_expire+0x11/0x64
        Aug  4 06:27:28 iranon kernel: [137969.382492]  [<c0187609>] mntput_no_expire+0x11/0x64
        Aug  4 06:27:28 iranon kernel: [137969.382496]  [<c017daf5>] path_walk+0x90/0x98
        Aug  4 06:27:28 iranon kernel: [137969.382502]  [<fa18f935>] nfs_getattr+0x8f/0xbe [nfs]
        Aug  4 06:27:28 iranon kernel: [137969.382520]  [<fa18f8a6>] nfs_getattr+0x0/0xbe [nfs]
        Aug  4 06:27:28 iranon kernel: [137969.382536]  [<c01785ac>] vfs_getattr+0x36/0x4d
        Aug  4 06:27:28 iranon kernel: [137969.382545]  [<c0178684>] vfs_lstat_fd+0x27/0x39
        Aug  4 06:27:28 iranon kernel: [137969.382550]  [<fa18bca6>] nfs_permission+0x0/0x129 [nfs]
        Aug  4 06:27:28 iranon kernel: [137969.382567]  [<c0187609>] mntput_no_expire+0x11/0x64
        Aug  4 06:27:28 iranon kernel: [137969.382572]  [<c0174914>] sys_faccessat+0x11e/0x15d
        Aug  4 06:27:28 iranon kernel: [137969.382582]  [<c0178703>] sys_lstat64+0xf/0x23
        Aug  4 06:27:28 iranon kernel: [137969.382588]  [<c017603c>] vfs_read+0xe3/0x11e
        Aug  4 06:27:28 iranon kernel: [137969.382598]  [<c0174962>] sys_access+0xf/0x13
        Aug  4 06:27:28 iranon kernel: [137969.382603]  [<c01077e8>] sysenter_past_esp+0x6d/0xa5
        Aug  4 06:27:28 iranon kernel: [137969.382617]  =======================

Ian.
-- 
Ian Campbell

Contemptuous lights flashed across the computer's console.
		-- Hitchhiker's Guide to the Galaxy

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

WARNING: multiple messages have this Message-ID (diff)
From: Ian Campbell <ijc@hellion.org.uk>
To: "J. Bruce Fields" <bfields@fieldses.org>
Cc: Tom Tucker <tom@opengridcomputing.com>,
	Trond Myklebust <trond.myklebust@fys.uio.no>,
	Grant Coady <gcoady.lk@gmail.com>,
	linux-kernel@vger.kernel.org, neilb@suse.de,
	linux-nfs@vger.kernel.org
Subject: Re: NFS regression? Odd delays and lockups accessing an NFS export.
Date: Sat, 27 Sep 2008 11:16:26 +0100	[thread overview]
Message-ID: <1222510586.3949.45.camel@localhost.localdomain> (raw)
In-Reply-To: <20080927035415.GC12765@fieldses.org>

[-- Attachment #1: Type: text/plain, Size: 6946 bytes --]

On Fri, 2008-09-26 at 23:54 -0400, J. Bruce Fields wrote:
> OK, so apologies, but this has been a long thread, and maybe we could
> use a summary of the symptoms and the results so far.  I think you said
> 2.6.24 or .25 was the last you're *positive* was good?

2.6.24.x was good 2.6.25.y was not. The Debian packages don't include
the stable release numbers in their version so I'm unsure of x and y,
Judging from the changelog entries I think 2.6.24.3 and 2.6.25.10. 

Since I have been building my own kernels I have seen repros with:

a551b98d5f6fce5897d497abd8bfb262efb33d2a Merge branch 'for-linus' of
git://git.kernel.dk/linux-2.6-block (was tip of git at the time I tested
it a while ago)

c02a79dedc7f3c3d4fdbb5eb2000cacea5df4cde v2.6.26.5

daedfbe2a67628a40076a6c75fb945c60f608a2e NFS: Ensure we zap only the
access and acl caches when setting new acls (backport of
f41f741838480aeaa3a189cff6e210503cf9c42d to 2.6.26.y stream, identified
by bisect, possibly/probably incorrectly)

f41f741838480aeaa3a189cff6e210503cf9c42d NFS: Ensure we zap only the
access and acl caches when setting new acls (original trunk commit of
above)

2e96d2867245668dbdb973729288cf69b9fafa66 NFS: Fix a warning in
nfs4_async_handle_error (commit before
f41f741838480aeaa3a189cff6e210503cf9c42d)

I did not see the problem with bce7f793daec3e65ec5c5705d2457b81fe7b5725
(v2.6.26) in 4 days 19 hours. Due to the apparent mis-bisection leading
to daedfbe2a67628a40076a6c75fb945c60f608a2e this result is in doubt, I'm
currently retesting this kernel.

The bisect was restricted to paths under fs/nfs* and net/sunrpc. I've
gone over the range v2.6.26..daedfbe2a67628a40076a6c75fb945c60f608a2e
and nothing jumps out at me. If my retest of 2.6.26 shows it to be OK
again I'll bisect over the non-NFS changesets in that range.

The symptoms are a hang of any access to an NFS mounted path, although
the rest of the system continues to work. 

Other potentially interesting info cribbed from earlier postings:

While the hang is occurring rpc_debug on the server gives nothing, on
the client gives:
[144741.637997] -pid- proc flgs status -client- -prog- --rqstp- -timeout -rpcwait -action- ---ops-- 
[144741.637997]  3439 0004 0080    -11 f3f48200 100003 f7770000    0 xprt_sending fa0ae88e fa0bddf4
[144741.637997]  3438 0001 00a0      0 f77f2a00 100003 f77700d0 5000 xprt_pending fa0ae88e fa0bddf4

The mount points are
        hopkins:/storage/music /storage/music nfs rw,nosuid,vers=3,rsize=32768,wsize=32768,namlen=255,hard,intr,proto=tcp,timeo=600,retrans=2,sec=sys,mountproto=udp,addr=192.168.1.6 0 0
        hopkins:/storage/mythtv/recordings /var/lib/mythtv/recordings nfs rw,nosuid,vers=3,rsize=32768,wsize=32768,namlen=255,acregmin=0,acregmax=0,acdirmin=0,acdirmax=0,hard,intr,proto=tcp,timeo=600,retrans=2,sec=sys,mountproto=udp,addr=192.168.1.6 0 0
        hopkins:/var/lib/mythvideo /var/lib/mythvideo nfs rw,nosuid,vers=3,rsize=32768,wsize=32768,namlen=255,acregmin=0,acregmax=0,acdirmin=0,acdirmax=0,hard,intr,proto=tcp,timeo=600,retrans=2,sec=sys,mountproto=udp,addr=192.168.1.6 0 0
        hopkins:/storage/home/ijc /home/ijc nfs rw,vers=3,rsize=131072,wsize=131072,namlen=255,hard,nointr,proto=tcp,timeo=600,retrans=2,sec=sys,mountproto=udp,addr=192.168.1.6 0 0
all of them seem to be effected simultaneously (but I'll check to make double sure next time it occurs).

I've seen hung task backtraces from effected processes, e.g. 
        Aug  4 06:27:28 iranon kernel: [137969.382277] INFO: task mythbackend:3161 blocked for more than 120 seconds.
        Aug  4 06:27:28 iranon kernel: [137969.382287] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
        Aug  4 06:27:28 iranon kernel: [137969.382291] mythbackend   D 00005dfc     0  3161   1
        Aug  4 06:27:28 iranon kernel: [137969.382295]        f2006c70 00000082 4b05cc7e 00005dfc f2006df0 c1c0e920 00000000 f7273390 
        Aug  4 06:27:28 iranon kernel: [137969.382301]        0000c8b4 00000000 00000001 f7273398 00000282 00000202 f71b0ab0 f200ddf0 
        Aug  4 06:27:28 iranon kernel: [137969.382306]        c1c012bc fa18f0a6 fa18f0bf c02b45a7 f71b0ab0 00000000 f200de0c fa18f0a6 
        Aug  4 06:27:28 iranon kernel: [137969.382311] Call Trace:
        Aug  4 06:27:28 iranon kernel: [137969.382347]  [<fa18f0a6>] nfs_wait_schedule+0x0/0x1e [nfs]
        Aug  4 06:27:28 iranon kernel: [137969.382384]  [<fa18f0bf>] nfs_wait_schedule+0x19/0x1e [nfs]
        Aug  4 06:27:28 iranon kernel: [137969.382399]  [<c02b45a7>] __wait_on_bit_lock+0x2a/0x52
        Aug  4 06:27:28 iranon kernel: [137969.382407]  [<fa18f0a6>] nfs_wait_schedule+0x0/0x1e [nfs]
        Aug  4 06:27:28 iranon kernel: [137969.382421]  [<c02b462e>] out_of_line_wait_on_bit_lock+0x5f/0x67
        Aug  4 06:27:28 iranon kernel: [137969.382429]  [<c0133a1d>] wake_bit_function+0x0/0x3c
        Aug  4 06:27:28 iranon kernel: [137969.382441]  [<fa18f16e>] __nfs_revalidate_inode+0xaa/0x211 [nfs]
        Aug  4 06:27:28 iranon kernel: [137969.382458]  [<c017bac0>] do_lookup+0x53/0x145
        Aug  4 06:27:28 iranon kernel: [137969.382466]  [<c0187609>] mntput_no_expire+0x11/0x64
        Aug  4 06:27:28 iranon kernel: [137969.382472]  [<c017d971>] __link_path_walk+0xa71/0xb65
        Aug  4 06:27:28 iranon kernel: [137969.382477]  [<c017bac0>] do_lookup+0x53/0x145
        Aug  4 06:27:28 iranon kernel: [137969.382483]  [<c0187609>] mntput_no_expire+0x11/0x64
        Aug  4 06:27:28 iranon kernel: [137969.382492]  [<c0187609>] mntput_no_expire+0x11/0x64
        Aug  4 06:27:28 iranon kernel: [137969.382496]  [<c017daf5>] path_walk+0x90/0x98
        Aug  4 06:27:28 iranon kernel: [137969.382502]  [<fa18f935>] nfs_getattr+0x8f/0xbe [nfs]
        Aug  4 06:27:28 iranon kernel: [137969.382520]  [<fa18f8a6>] nfs_getattr+0x0/0xbe [nfs]
        Aug  4 06:27:28 iranon kernel: [137969.382536]  [<c01785ac>] vfs_getattr+0x36/0x4d
        Aug  4 06:27:28 iranon kernel: [137969.382545]  [<c0178684>] vfs_lstat_fd+0x27/0x39
        Aug  4 06:27:28 iranon kernel: [137969.382550]  [<fa18bca6>] nfs_permission+0x0/0x129 [nfs]
        Aug  4 06:27:28 iranon kernel: [137969.382567]  [<c0187609>] mntput_no_expire+0x11/0x64
        Aug  4 06:27:28 iranon kernel: [137969.382572]  [<c0174914>] sys_faccessat+0x11e/0x15d
        Aug  4 06:27:28 iranon kernel: [137969.382582]  [<c0178703>] sys_lstat64+0xf/0x23
        Aug  4 06:27:28 iranon kernel: [137969.382588]  [<c017603c>] vfs_read+0xe3/0x11e
        Aug  4 06:27:28 iranon kernel: [137969.382598]  [<c0174962>] sys_access+0xf/0x13
        Aug  4 06:27:28 iranon kernel: [137969.382603]  [<c01077e8>] sysenter_past_esp+0x6d/0xa5
        Aug  4 06:27:28 iranon kernel: [137969.382617]  =======================

Ian.
-- 
Ian Campbell

Contemptuous lights flashed across the computer's console.
		-- Hitchhiker's Guide to the Galaxy

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

  reply	other threads:[~2008-09-27 10:16 UTC|newest]

Thread overview: 117+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2008-08-18  2:02 NFS regression? Odd delays and lockups accessing an NFS export Grant Coady
2008-08-18  2:02 ` Grant Coady
2008-08-18 18:50 ` Athanasius
2008-08-18 19:19   ` Trond Myklebust
     [not found]   ` <20080818185048.GO20684-QVNlIYdkypLYtjvyW6yDsg@public.gmane.org>
2008-08-18 19:37     ` J. K. Cliburn
2008-08-18 19:37       ` J. K. Cliburn
     [not found]       ` <48A9CF89.9090304-Bdlq13kUjeyLZ21kGMrzwg@public.gmane.org>
2008-08-18 23:13         ` Athanasius
2008-08-18 23:13           ` Athanasius
2009-05-12 20:27       ` Frank Filz
2009-05-13  0:05         ` Jay Cliburn
     [not found] ` <p6lha4te54h04qdbnos1mcemdmn1cfca6s-e09XROE/p8c@public.gmane.org>
2008-08-18 19:20   ` Trond Myklebust
2008-08-18 19:20     ` Trond Myklebust
2008-08-20  1:10     ` Grant Coady
2008-08-20  1:10       ` Grant Coady
2008-08-20 23:17     ` Grant Coady
2008-08-20 23:17       ` Grant Coady
2008-08-22 10:23     ` Ian Campbell
2008-08-22 18:08       ` Trond Myklebust
2008-08-22 18:13         ` Ian Campbell
2008-08-22 18:13           ` Ian Campbell
2008-08-22 19:33           ` John Ronciak
2008-08-22 19:33             ` John Ronciak
     [not found]             ` <56a8daef0808221233h68853587n6015ca7d809b17e1-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2008-08-22 20:00               ` Ian Campbell
2008-08-22 20:00                 ` Ian Campbell
     [not found]                 ` <1219435207.27921.51.camel-bi+AKbBUZKY6gyzm1THtWbp2dZbC/Bob@public.gmane.org>
2008-08-22 21:15                   ` John Ronciak
2008-08-22 21:15                     ` John Ronciak
2008-08-22 21:23                   ` Trond Myklebust
2008-08-22 21:23                     ` Trond Myklebust
2008-08-22 21:37                     ` Ian Campbell
2008-08-22 21:37                       ` Ian Campbell
     [not found]                       ` <1219441041.27921.57.camel-bi+AKbBUZKY6gyzm1THtWbp2dZbC/Bob@public.gmane.org>
2008-08-22 21:56                         ` Trond Myklebust
2008-08-22 21:56                           ` Trond Myklebust
2008-08-22 22:41                           ` Ian Campbell
2008-08-22 22:41                             ` Ian Campbell
2008-08-24 18:53                           ` Ian Campbell
     [not found]                             ` <1219603981.27921.145.camel-bi+AKbBUZKY6gyzm1THtWbp2dZbC/Bob@public.gmane.org>
2008-08-24 19:17                               ` Trond Myklebust
2008-08-24 19:17                                 ` Trond Myklebust
2008-08-24 19:19                                 ` Trond Myklebust
2008-08-24 19:19                                   ` Trond Myklebust
2008-08-24 22:09                                   ` Ian Campbell
2008-08-24 22:09                                     ` Ian Campbell
     [not found]                                     ` <1219615789.27921.152.camel-bi+AKbBUZKY6gyzm1THtWbp2dZbC/Bob@public.gmane.org>
2008-08-24 22:15                                       ` Trond Myklebust
2008-08-24 22:15                                         ` Trond Myklebust
2008-08-25  9:59                                         ` Ian Campbell
2008-08-25  9:59                                           ` Ian Campbell
2008-08-25 16:04                                         ` Tom Tucker
2008-08-25 16:54                                           ` Trond Myklebust
2008-08-25 20:15                                             ` Tom Tucker
2008-08-25 20:15                                               ` Tom Tucker
2008-08-26 19:27                                           ` J. Bruce Fields
2008-08-26 19:27                                             ` J. Bruce Fields
2008-08-27 14:43                                             ` Tom Tucker
2008-08-27 14:43                                               ` Tom Tucker
2008-08-30 15:47                                               ` Ian Campbell
2008-08-30 15:47                                                 ` Ian Campbell
     [not found]                                                 ` <1220111261.31172.14.camel-bi+AKbBUZKY6gyzm1THtWbp2dZbC/Bob@public.gmane.org>
2008-08-31 19:30                                                   ` J. Bruce Fields
2008-08-31 19:30                                                     ` J. Bruce Fields
2008-08-31 19:44                                                     ` Ian Campbell
2008-08-31 19:44                                                       ` Ian Campbell
     [not found]                                                       ` <1220211842.31172.18.camel-bi+AKbBUZKY6gyzm1THtWbp2dZbC/Bob@public.gmane.org>
2008-08-31 19:46                                                         ` J. Bruce Fields
2008-08-31 19:46                                                           ` J. Bruce Fields
2008-08-31 19:49                                                           ` Ian Campbell
2008-08-31 19:49                                                             ` Ian Campbell
2008-08-31 19:51                                                             ` Tom Tucker
2008-08-31 19:51                                                               ` Tom Tucker
2008-08-31 19:51                                                         ` Tom Tucker
2008-08-31 19:51                                                           ` Tom Tucker
2008-08-31 21:18                                                           ` Ian Campbell
2008-08-31 21:18                                                             ` Ian Campbell
     [not found]                                                             ` <1220217505.31172.26.camel-bi+AKbBUZKY6gyzm1THtWbp2dZbC/Bob@public.gmane.org>
2008-09-01 17:20                                                               ` Tom Tucker
2008-09-01 17:20                                                                 ` Tom Tucker
2008-09-01 17:46                                                                 ` Ian Campbell
2008-09-01 17:46                                                                   ` Ian Campbell
2008-09-10  8:40                                                                 ` Ian Campbell
2008-09-10  8:40                                                                   ` Ian Campbell
     [not found]                                                                   ` <1221036015.24993.27.camel-o4Be2W7LfRlXesXXhkcM7miJhflN2719@public.gmane.org>
2008-09-12 22:43                                                                     ` J. Bruce Fields
2008-09-12 22:43                                                                       ` J. Bruce Fields
2008-09-12 23:15                                                                       ` Tom Tucker
2008-09-12 23:15                                                                         ` Tom Tucker
2008-09-13  8:57                                                                         ` Ian Campbell
2008-09-13  8:57                                                                           ` Ian Campbell
     [not found]                                                                           ` <1221296243.2534.7.camel-bi+AKbBUZKY6gyzm1THtWbp2dZbC/Bob@public.gmane.org>
2008-09-16  5:48                                                                             ` Ian Campbell
2008-09-16  5:48                                                                               ` Ian Campbell
     [not found]                                                                               ` <1221544139.2534.18.camel-bi+AKbBUZKY6gyzm1THtWbp2dZbC/Bob@public.gmane.org>
2008-09-16 11:38                                                                                 ` Tom Tucker
2008-09-16 11:38                                                                                   ` Tom Tucker
2008-09-16 15:03                                                                                   ` Ian Campbell
     [not found]                                                                                     ` <1221577412.28572.60.camel-o4Be2W7LfRlXesXXhkcM7miJhflN2719@public.gmane.org>
2008-09-16 15:58                                                                                       ` Tom Tucker
2008-09-16 15:58                                                                                         ` Tom Tucker
2008-09-16 16:24                                                                                         ` Ian Campbell
2008-09-16 16:24                                                                                           ` Ian Campbell
     [not found]                                                                                           ` <1221582285.28572.67.camel-o4Be2W7LfRlXesXXhkcM7miJhflN2719@public.gmane.org>
2008-09-23  7:59                                                                                             ` Ian Campbell
2008-09-23  7:59                                                                                               ` Ian Campbell
     [not found]                                                                                               ` <1222156770.6869.13.camel-bi+AKbBUZKY6gyzm1THtWbp2dZbC/Bob@public.gmane.org>
2008-09-23 11:33                                                                                                 ` Ian Campbell
2008-09-23 11:33                                                                                                   ` Ian Campbell
     [not found]                                                                                                   ` <1222169589.6869.20.camel-bi+AKbBUZKY6gyzm1THtWbp2dZbC/Bob@public.gmane.org>
2008-09-23 17:03                                                                                                     ` J. Bruce Fields
2008-09-23 17:03                                                                                                       ` J. Bruce Fields
2008-09-26 15:37                                                                                                       ` Ian Campbell
2008-09-26 15:37                                                                                                         ` Ian Campbell
     [not found]                                                                                                         ` <1222443426.3949.18.camel-bi+AKbBUZKY6gyzm1THtWbp2dZbC/Bob@public.gmane.org>
2008-09-26 18:17                                                                                                           ` Ian Campbell
2008-09-26 18:17                                                                                                             ` Ian Campbell
     [not found]                                                                                                             ` <1222453053.3949.21.camel-bi+AKbBUZKY6gyzm1THtWbp2dZbC/Bob@public.gmane.org>
2008-09-27  3:54                                                                                                               ` J. Bruce Fields
2008-09-27  3:54                                                                                                                 ` J. Bruce Fields
2008-09-27 10:16                                                                                                                 ` Ian Campbell [this message]
2008-09-27 10:16                                                                                                                   ` Ian Campbell
2008-08-25 21:39                                     ` Roger Heflin
2008-08-25 21:39                                       ` Roger Heflin
2008-08-25 20:23                           ` Grant Coady
2008-08-25 20:23                             ` Grant Coady
     [not found]                             ` <4156b4tgrrsflla1svmu4jl6j5sme4a715-e09XROE/p8c@public.gmane.org>
2008-08-25 22:11                               ` Trond Myklebust
2008-08-25 22:11                                 ` Trond Myklebust
2008-08-26  0:29                                 ` Grant Coady
2008-08-26  0:29                                   ` Grant Coady
2008-08-26  0:59                                   ` Muntz, Daniel
2008-08-26  0:59                                     ` Muntz, Daniel
2008-08-26  1:06                                 ` Grant Coady
2008-08-26  1:06                                   ` Grant Coady
  -- strict thread matches above, loose matches on Subject: below --
2008-09-10  2:51 Benoit Plessis

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1222510586.3949.45.camel@localhost.localdomain \
    --to=ijc-kcikpvwj1kudxyznreorvg@public.gmane.org \
    --cc=bfields@fieldses.org \
    --cc=gcoady.lk@gmail.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-nfs@vger.kernel.org \
    --cc=neilb@suse.de \
    --cc=tom@opengridcomputing.com \
    --cc=trond.myklebust@fys.uio.no \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.