* autofs stops working after a while with USE_MISC_DEVICE
@ 2009-07-02 18:55 Thomas Chan
2009-07-03 4:40 ` Ian Kent
0 siblings, 1 reply; 9+ messages in thread
From: Thomas Chan @ 2009-07-02 18:55 UTC (permalink / raw)
To: autofs
I am using autofs 5.0.4 with the latest patches up to 12-Jun-2009 with
the 2.6.29.1 kernel.
I noticed that after a while (overnight) new automounts would no longer
work.
An strace shows:
rt_sigtimedwait(~[CHLD CONT RTMIN RT_1],
[pid 26289] 11:22:57 gettimeofday({1246558977, 61322}, NULL) = 0
[pid 26289] 11:22:57 futex(0x7f6f5db43160, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 26289] 11:22:57 clock_gettime(CLOCK_REALTIME, {1246558977, 61421426}) = 0
[pid 26289] 11:22:57 futex(0x7f6f5db43124, FUTEX_WAIT_PRIVATE, 150501, {0, 999900574}) = -1 ETIMEDOUT (Connection timed out)
[pid 26289] 11:22:58 gettimeofday({1246558978, 61986}, NULL) = 0
[pid 26289] 11:22:58 futex(0x7f6f5db43160, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 26289] 11:22:58 clock_gettime(CLOCK_REALTIME, {1246558978, 62086850}) = 0
[pid 26289] 11:22:58 futex(0x7f6f5db43124, FUTEX_WAIT_PRIVATE, 150503, {0, 999899150}) = -1 ETIMEDOUT (Connection timed out)
lsof shows that automount has 10288 files open, of which most of these
are like:
automount 15539 root *217u sock 0,4
595374 can't identify protocol
automount 15539 root *218u sock 0,4
595376 can't identify protocol
automount 15539 root *219u sock 0,4
595378 can't identify protocol
automount 15539 root *220u sock 0,4
595380 can't identify protocol
Upon restarting autofs, these open files are cleared out and automount
works, but the open files gradually build up again.
After setting USE_MISC_DEVICE=no, I no longer get the open file issue, but
we are trying to remedy the autofs lazy unmount / cwd issue.
Any ideas would be appreciated.
Thomas
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: autofs stops working after a while with USE_MISC_DEVICE 2009-07-02 18:55 autofs stops working after a while with USE_MISC_DEVICE Thomas Chan @ 2009-07-03 4:40 ` Ian Kent 2009-07-03 5:35 ` Thomas Chan 0 siblings, 1 reply; 9+ messages in thread From: Ian Kent @ 2009-07-03 4:40 UTC (permalink / raw) To: Thomas Chan; +Cc: autofs Thomas Chan wrote: > > I am using autofs 5.0.4 with the latest patches up to 12-Jun-2009 with > the 2.6.29.1 kernel. > > I noticed that after a while (overnight) new automounts would no longer > work. > > An strace shows: > rt_sigtimedwait(~[CHLD CONT RTMIN RT_1], > > [pid 26289] 11:22:57 gettimeofday({1246558977, 61322}, NULL) = 0 > [pid 26289] 11:22:57 futex(0x7f6f5db43160, FUTEX_WAKE_PRIVATE, 1) = 0 > [pid 26289] 11:22:57 clock_gettime(CLOCK_REALTIME, {1246558977, > 61421426}) = 0 > [pid 26289] 11:22:57 futex(0x7f6f5db43124, FUTEX_WAIT_PRIVATE, 150501, > {0, 999900574}) = -1 ETIMEDOUT (Connection timed out) > [pid 26289] 11:22:58 gettimeofday({1246558978, 61986}, NULL) = 0 > [pid 26289] 11:22:58 futex(0x7f6f5db43160, FUTEX_WAKE_PRIVATE, 1) = 0 > [pid 26289] 11:22:58 clock_gettime(CLOCK_REALTIME, {1246558978, > 62086850}) = 0 > [pid 26289] 11:22:58 futex(0x7f6f5db43124, FUTEX_WAIT_PRIVATE, 150503, > {0, 999899150}) = -1 ETIMEDOUT (Connection timed out) > > lsof shows that automount has 10288 files open, of which most of these > are like: > > automount 15539 root *217u sock 0,4 595374 can't > identify protocol > automount 15539 root *218u sock 0,4 595376 can't > identify protocol > automount 15539 root *219u sock 0,4 595378 can't > identify protocol > automount 15539 root *220u sock 0,4 595380 can't > identify protocol > > Upon restarting autofs, these open files are cleared out and automount > works, but the open files gradually build up again. > > After setting USE_MISC_DEVICE=no, I no longer get the open file issue, > but we are trying to remedy the autofs lazy unmount / cwd issue. Mmm .. that's not good. I'll have a look and see what I can see. Ian ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: autofs stops working after a while with USE_MISC_DEVICE 2009-07-03 4:40 ` Ian Kent @ 2009-07-03 5:35 ` Thomas Chan 2009-07-03 8:41 ` Ian Kent 2009-07-04 3:49 ` Ian Kent 0 siblings, 2 replies; 9+ messages in thread From: Thomas Chan @ 2009-07-03 5:35 UTC (permalink / raw) To: Ian Kent; +Cc: autofs >> lsof shows that automount has 10288 files open, of which most of these >> are like: >> >> automount 15539 root *217u sock 0,4 595374 can't >> identify protocol >> automount 15539 root *218u sock 0,4 595376 can't >> identify protocol >> automount 15539 root *219u sock 0,4 595378 can't >> identify protocol >> automount 15539 root *220u sock 0,4 595380 can't >> identify protocol >> >> Upon restarting autofs, these open files are cleared out and automount >> works, but the open files gradually build up again. >> >> After setting USE_MISC_DEVICE=no, I no longer get the open file issue, >> but we are trying to remedy the autofs lazy unmount / cwd issue. > > Mmm .. that's not good. > I'll have a look and see what I can see. > > Ian Sorry about the initial red herring, but it looks like I'm getting the increasing open files even with USE_MISC_DEVICE=no. I must have previously done a grep for autofs instead of automount. In any case, after a restart of autofs: [root@talon ~]# /etc/init.d/autofs restart Stopping automount: [ OK ] Starting automount: [ OK ] [root@talon ~]# lsof |grep automount | grep sock automount 309 root 52u sock 0,4 3230868911 can't identify protocol automount 309 root 54u sock 0,4 3230868918 can't identify protocol automount 309 root 55u sock 0,4 3230868920 can't identify protocol automount 309 root 56u sock 0,4 3230868922 can't identify protocol automount 309 root 57u sock 0,4 3230868924 can't identify protocol automount 309 root 59u sock 0,4 3230868930 can't identify protocol automount 309 root 61u sock 0,4 3230868937 can't identify protocol automount 309 root 62u sock 0,4 3230868939 can't identify protocol automount 309 root 63u sock 0,4 3230868941 can't identify protocol automount 309 root 64u sock 0,4 3230868943 can't identify protocol automount 309 root 66u sock 0,4 3230868949 can't identify protocol automount 309 root 68u sock 0,4 3230868956 can't identify protocol automount 309 root 69u sock 0,4 3230868958 can't identify protocol automount 309 root 70u sock 0,4 3230868960 can't identify protocol automount 309 root 71u sock 0,4 3230868962 can't identify protocol automount 309 root 73u sock 0,4 3230868968 can't identify protocol automount 309 root 75u sock 0,4 3230868975 can't identify protocol automount 309 root 76u sock 0,4 3230868977 can't identify protocol automount 309 root 77u sock 0,4 3230868979 can't identify protocol automount 309 root 78u sock 0,4 3230868981 can't identify protocol [root@talon ~]# This gradually increases with every new mount until automount has 10288 files open. Thomas ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: autofs stops working after a while with USE_MISC_DEVICE 2009-07-03 5:35 ` Thomas Chan @ 2009-07-03 8:41 ` Ian Kent 2009-07-03 17:15 ` Thomas Chan 2009-07-04 3:49 ` Ian Kent 1 sibling, 1 reply; 9+ messages in thread From: Ian Kent @ 2009-07-03 8:41 UTC (permalink / raw) To: Thomas Chan; +Cc: autofs Thomas Chan wrote: > >>> lsof shows that automount has 10288 files open, of which most of these >>> are like: >>> >>> automount 15539 root *217u sock 0,4 595374 can't >>> identify protocol >>> automount 15539 root *218u sock 0,4 595376 can't >>> identify protocol >>> automount 15539 root *219u sock 0,4 595378 can't >>> identify protocol >>> automount 15539 root *220u sock 0,4 595380 can't >>> identify protocol >>> >>> Upon restarting autofs, these open files are cleared out and automount >>> works, but the open files gradually build up again. >>> >>> After setting USE_MISC_DEVICE=no, I no longer get the open file issue, >>> but we are trying to remedy the autofs lazy unmount / cwd issue. >> >> Mmm .. that's not good. >> I'll have a look and see what I can see. >> >> Ian > > Sorry about the initial red herring, but it looks like I'm getting the > increasing open files even with USE_MISC_DEVICE=no. I must have > previously done a grep for autofs instead of automount. > > In any case, after a restart of autofs: > [root@talon ~]# /etc/init.d/autofs restart > Stopping automount: [ OK ] > Starting automount: [ OK ] > [root@talon ~]# lsof |grep automount | grep sock > automount 309 root 52u sock 0,4 3230868911 > can't identify protocol > automount 309 root 54u sock 0,4 3230868918 > can't identify protocol > automount 309 root 55u sock 0,4 3230868920 > can't identify protocol > automount 309 root 56u sock 0,4 3230868922 > can't identify protocol > automount 309 root 57u sock 0,4 3230868924 > can't identify protocol > automount 309 root 59u sock 0,4 3230868930 > can't identify protocol > automount 309 root 61u sock 0,4 3230868937 > can't identify protocol > automount 309 root 62u sock 0,4 3230868939 > can't identify protocol > automount 309 root 63u sock 0,4 3230868941 > can't identify protocol > automount 309 root 64u sock 0,4 3230868943 > can't identify protocol > automount 309 root 66u sock 0,4 3230868949 > can't identify protocol > automount 309 root 68u sock 0,4 3230868956 > can't identify protocol > automount 309 root 69u sock 0,4 3230868958 > can't identify protocol > automount 309 root 70u sock 0,4 3230868960 > can't identify protocol > automount 309 root 71u sock 0,4 3230868962 > can't identify protocol > automount 309 root 73u sock 0,4 3230868968 > can't identify protocol > automount 309 root 75u sock 0,4 3230868975 > can't identify protocol > automount 309 root 76u sock 0,4 3230868977 > can't identify protocol > automount 309 root 77u sock 0,4 3230868979 > can't identify protocol > automount 309 root 78u sock 0,4 3230868981 > can't identify protocol > [root@talon ~]# > > This gradually increases with every new mount until automount has 10288 > files open. What distribution, and more specifically, what libtirpc, if any? Ian ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: autofs stops working after a while with USE_MISC_DEVICE 2009-07-03 8:41 ` Ian Kent @ 2009-07-03 17:15 ` Thomas Chan 0 siblings, 0 replies; 9+ messages in thread From: Thomas Chan @ 2009-07-03 17:15 UTC (permalink / raw) To: Ian Kent; +Cc: autofs > > What distribution, and more specifically, what libtirpc, if any? > > Ian > CentOS 5.3, no libtirpc. Thomas ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: autofs stops working after a while with USE_MISC_DEVICE 2009-07-03 5:35 ` Thomas Chan 2009-07-03 8:41 ` Ian Kent @ 2009-07-04 3:49 ` Ian Kent 2009-07-04 4:32 ` Thomas Chan 1 sibling, 1 reply; 9+ messages in thread From: Ian Kent @ 2009-07-04 3:49 UTC (permalink / raw) To: Thomas Chan; +Cc: autofs mailing list, Wolfram Wagner On Thu, 2 Jul 2009, Thomas Chan wrote: > > > > lsof shows that automount has 10288 files open, of which most of these > > > are like: > > > > > > automount 15539 root *217u sock 0,4 595374 can't > > > identify protocol > > > automount 15539 root *218u sock 0,4 595376 can't > > > identify protocol > > > automount 15539 root *219u sock 0,4 595378 can't > > > identify protocol > > > automount 15539 root *220u sock 0,4 595380 can't > > > identify protocol > > > > > > Upon restarting autofs, these open files are cleared out and automount > > > works, but the open files gradually build up again. > > > > > > After setting USE_MISC_DEVICE=no, I no longer get the open file issue, > > > but we are trying to remedy the autofs lazy unmount / cwd issue. > > > > Mmm .. that's not good. > > I'll have a look and see what I can see. > > > > Ian > > Sorry about the initial red herring, but it looks like I'm getting the > increasing open files even with USE_MISC_DEVICE=no. I must have previously > done a grep for autofs instead of automount. > > In any case, after a restart of autofs: > [root@talon ~]# /etc/init.d/autofs restart > Stopping automount: [ OK ] > Starting automount: [ OK ] > [root@talon ~]# lsof |grep automount | grep sock > automount 309 root 52u sock 0,4 3230868911 can't > identify protocol > automount 309 root 54u sock 0,4 3230868918 can't > identify protocol > automount 309 root 55u sock 0,4 3230868920 can't > identify protocol > automount 309 root 56u sock 0,4 3230868922 can't > identify protocol > automount 309 root 57u sock 0,4 3230868924 can't > identify protocol > automount 309 root 59u sock 0,4 3230868930 can't > identify protocol > automount 309 root 61u sock 0,4 3230868937 can't > identify protocol > automount 309 root 62u sock 0,4 3230868939 can't > identify protocol > automount 309 root 63u sock 0,4 3230868941 can't > identify protocol > automount 309 root 64u sock 0,4 3230868943 can't > identify protocol > automount 309 root 66u sock 0,4 3230868949 can't > identify protocol > automount 309 root 68u sock 0,4 3230868956 can't > identify protocol > automount 309 root 69u sock 0,4 3230868958 can't > identify protocol > automount 309 root 70u sock 0,4 3230868960 can't > identify protocol > automount 309 root 71u sock 0,4 3230868962 can't > identify protocol > automount 309 root 73u sock 0,4 3230868968 can't > identify protocol > automount 309 root 75u sock 0,4 3230868975 can't > identify protocol > automount 309 root 76u sock 0,4 3230868977 can't > identify protocol > automount 309 root 77u sock 0,4 3230868979 can't > identify protocol > automount 309 root 78u sock 0,4 3230868981 can't > identify protocol > [root@talon ~]# > > This gradually increases with every new mount until automount has 10288 files > open. I may have spotted this problem. Could you try this patch please. Ian autofs-5.0.4 - fix rpc fd leak From: Ian Kent <raven@themaw.net> Recent changes which introduced the ability to handle to cope with IPv6 addresses causes a file descriptor leak in the RPC library. --- lib/rpc_subs.c | 11 ----------- 1 files changed, 0 insertions(+), 11 deletions(-) diff --git a/lib/rpc_subs.c b/lib/rpc_subs.c index d034b29..cafc775 100644 --- a/lib/rpc_subs.c +++ b/lib/rpc_subs.c @@ -253,17 +253,6 @@ static CLIENT *rpc_do_create_client(struct sockaddr *addr, struct conn_info *inf return NULL; } - if (!info->client) { - *fd = open_sock(addr->sa_family, type, proto); - if (*fd < 0) - return NULL; - - if (bind(*fd, laddr, slen) < 0) { - close(*fd); - return NULL; - } - } - switch (info->proto->p_proto) { case IPPROTO_UDP: if (!info->client) { ^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: autofs stops working after a while with USE_MISC_DEVICE 2009-07-04 3:49 ` Ian Kent @ 2009-07-04 4:32 ` Thomas Chan 2009-07-07 18:20 ` Thomas Chan 0 siblings, 1 reply; 9+ messages in thread From: Thomas Chan @ 2009-07-04 4:32 UTC (permalink / raw) To: Ian Kent; +Cc: autofs mailing list, Wolfram Wagner On Sat, 4 Jul 2009, Ian Kent wrote: > On Thu, 2 Jul 2009, Thomas Chan wrote: > >> >>>> lsof shows that automount has 10288 files open, of which most of these >>>> are like: >>>> >>>> automount 15539 root *217u sock 0,4 595374 can't >>>> identify protocol >>>> automount 15539 root *218u sock 0,4 595376 can't >>>> identify protocol >>>> automount 15539 root *219u sock 0,4 595378 can't >>>> identify protocol >>>> automount 15539 root *220u sock 0,4 595380 can't >>>> identify protocol >>>> >>>> Upon restarting autofs, these open files are cleared out and automount >>>> works, but the open files gradually build up again. >>>> >>>> After setting USE_MISC_DEVICE=no, I no longer get the open file issue, >>>> but we are trying to remedy the autofs lazy unmount / cwd issue. >>> >>> Mmm .. that's not good. >>> I'll have a look and see what I can see. >>> >>> Ian >> >> Sorry about the initial red herring, but it looks like I'm getting the >> increasing open files even with USE_MISC_DEVICE=no. I must have previously >> done a grep for autofs instead of automount. >> >> In any case, after a restart of autofs: >> [root@talon ~]# /etc/init.d/autofs restart >> Stopping automount: [ OK ] >> Starting automount: [ OK ] >> [root@talon ~]# lsof |grep automount | grep sock >> automount 309 root 52u sock 0,4 3230868911 can't >> identify protocol >> automount 309 root 54u sock 0,4 3230868918 can't >> identify protocol >> automount 309 root 55u sock 0,4 3230868920 can't >> identify protocol >> automount 309 root 56u sock 0,4 3230868922 can't >> identify protocol >> automount 309 root 57u sock 0,4 3230868924 can't >> identify protocol >> automount 309 root 59u sock 0,4 3230868930 can't >> identify protocol >> automount 309 root 61u sock 0,4 3230868937 can't >> identify protocol >> automount 309 root 62u sock 0,4 3230868939 can't >> identify protocol >> automount 309 root 63u sock 0,4 3230868941 can't >> identify protocol >> automount 309 root 64u sock 0,4 3230868943 can't >> identify protocol >> automount 309 root 66u sock 0,4 3230868949 can't >> identify protocol >> automount 309 root 68u sock 0,4 3230868956 can't >> identify protocol >> automount 309 root 69u sock 0,4 3230868958 can't >> identify protocol >> automount 309 root 70u sock 0,4 3230868960 can't >> identify protocol >> automount 309 root 71u sock 0,4 3230868962 can't >> identify protocol >> automount 309 root 73u sock 0,4 3230868968 can't >> identify protocol >> automount 309 root 75u sock 0,4 3230868975 can't >> identify protocol >> automount 309 root 76u sock 0,4 3230868977 can't >> identify protocol >> automount 309 root 77u sock 0,4 3230868979 can't >> identify protocol >> automount 309 root 78u sock 0,4 3230868981 can't >> identify protocol >> [root@talon ~]# >> >> This gradually increases with every new mount until automount has 10288 files >> open. > > I may have spotted this problem. > Could you try this patch please. > > Ian > > autofs-5.0.4 - fix rpc fd leak > > From: Ian Kent <raven@themaw.net> > > Recent changes which introduced the ability to handle to cope with > IPv6 addresses causes a file descriptor leak in the RPC library. > --- > > lib/rpc_subs.c | 11 ----------- > 1 files changed, 0 insertions(+), 11 deletions(-) > > > diff --git a/lib/rpc_subs.c b/lib/rpc_subs.c > index d034b29..cafc775 100644 > --- a/lib/rpc_subs.c > +++ b/lib/rpc_subs.c > @@ -253,17 +253,6 @@ static CLIENT *rpc_do_create_client(struct sockaddr *addr, struct conn_info *inf > return NULL; > } > > - if (!info->client) { > - *fd = open_sock(addr->sa_family, type, proto); > - if (*fd < 0) > - return NULL; > - > - if (bind(*fd, laddr, slen) < 0) { > - close(*fd); > - return NULL; > - } > - } > - > switch (info->proto->p_proto) { > case IPPROTO_UDP: > if (!info->client) { > This looks good so far. After restarting autofs, I don't see the increasing open files any more. Thomas ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: autofs stops working after a while with USE_MISC_DEVICE 2009-07-04 4:32 ` Thomas Chan @ 2009-07-07 18:20 ` Thomas Chan 2009-07-08 1:04 ` Ian Kent 0 siblings, 1 reply; 9+ messages in thread From: Thomas Chan @ 2009-07-07 18:20 UTC (permalink / raw) To: Ian Kent; +Cc: autofs mailing list, Wolfram Wagner >> I may have spotted this problem. >> Could you try this patch please. >> >> Ian >> >> autofs-5.0.4 - fix rpc fd leak >> >> From: Ian Kent <raven@themaw.net> >> >> Recent changes which introduced the ability to handle to cope with >> IPv6 addresses causes a file descriptor leak in the RPC library. >> --- >> >> lib/rpc_subs.c | 11 ----------- >> 1 files changed, 0 insertions(+), 11 deletions(-) >> >> >> diff --git a/lib/rpc_subs.c b/lib/rpc_subs.c >> index d034b29..cafc775 100644 >> --- a/lib/rpc_subs.c >> +++ b/lib/rpc_subs.c >> @@ -253,17 +253,6 @@ static CLIENT *rpc_do_create_client(struct sockaddr >> *addr, struct conn_info *inf >> return NULL; >> } >> >> - if (!info->client) { >> - *fd = open_sock(addr->sa_family, type, proto); >> - if (*fd < 0) >> - return NULL; >> - >> - if (bind(*fd, laddr, slen) < 0) { >> - close(*fd); >> - return NULL; >> - } >> - } >> - >> switch (info->proto->p_proto) { >> case IPPROTO_UDP: >> if (!info->client) { >> > > This looks good so far. After restarting autofs, I don't see the increasing > open files any more. > > Thomas > After a few days of running with this patch, everything looks to still be functioning as expected. Thanks for the speedy fix! Thomas ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: autofs stops working after a while with USE_MISC_DEVICE 2009-07-07 18:20 ` Thomas Chan @ 2009-07-08 1:04 ` Ian Kent 0 siblings, 0 replies; 9+ messages in thread From: Ian Kent @ 2009-07-08 1:04 UTC (permalink / raw) To: Thomas Chan; +Cc: autofs mailing list, Wolfram Wagner Thomas Chan wrote: > >>> I may have spotted this problem. >>> Could you try this patch please. >>> >>> Ian >>> >>> autofs-5.0.4 - fix rpc fd leak >>> >>> From: Ian Kent <raven@themaw.net> >>> >>> Recent changes which introduced the ability to handle to cope with >>> IPv6 addresses causes a file descriptor leak in the RPC library. >>> --- >>> >>> lib/rpc_subs.c | 11 ----------- >>> 1 files changed, 0 insertions(+), 11 deletions(-) >>> >>> >>> diff --git a/lib/rpc_subs.c b/lib/rpc_subs.c >>> index d034b29..cafc775 100644 >>> --- a/lib/rpc_subs.c >>> +++ b/lib/rpc_subs.c >>> @@ -253,17 +253,6 @@ static CLIENT *rpc_do_create_client(struct >>> sockaddr *addr, struct conn_info *inf >>> return NULL; >>> } >>> >>> - if (!info->client) { >>> - *fd = open_sock(addr->sa_family, type, proto); >>> - if (*fd < 0) >>> - return NULL; >>> - >>> - if (bind(*fd, laddr, slen) < 0) { >>> - close(*fd); >>> - return NULL; >>> - } >>> - } >>> - >>> switch (info->proto->p_proto) { >>> case IPPROTO_UDP: >>> if (!info->client) { >>> >> >> This looks good so far. After restarting autofs, I don't see the >> increasing open files any more. >> >> Thomas >> > > After a few days of running with this patch, everything looks to still > be functioning as expected. Thanks for the speedy fix! Thanks, I get onto committing the fix. Ian ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2009-07-08 1:04 UTC | newest] Thread overview: 9+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2009-07-02 18:55 autofs stops working after a while with USE_MISC_DEVICE Thomas Chan 2009-07-03 4:40 ` Ian Kent 2009-07-03 5:35 ` Thomas Chan 2009-07-03 8:41 ` Ian Kent 2009-07-03 17:15 ` Thomas Chan 2009-07-04 3:49 ` Ian Kent 2009-07-04 4:32 ` Thomas Chan 2009-07-07 18:20 ` Thomas Chan 2009-07-08 1:04 ` Ian Kent
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.