* XFS: I/O Error Detected / 2.6.27.39 @ 2010-11-16 13:10 Piotr Kandziora 2010-11-16 20:44 ` Emmanuel Florac 2010-11-16 21:54 ` Eric Sandeen 0 siblings, 2 replies; 6+ messages in thread From: Piotr Kandziora @ 2010-11-16 13:10 UTC (permalink / raw) To: xfs; +Cc: Artur Piechocki, Janusz Bak, lukasz.wittig Hi, Our environment is following: - we have 24GB RAM, - we are using 3ware controller (and it does not report any errors), - we have one big logical volume (20TB) exported via NFS with large amount of small files (about 150k), - we are doing periodically backup of this logical volume using rsync to another server. - we have kernel 2.6.27.39, Unfortunately our system is freezing unexpectedly without reason. We started investigating this problem and noticed that cache memory is slowly increasing. We tried to dump this cache memory using: /bin/echo "3" > /proc/sys/vm/drop_caches In a result, cache was dumped, but in logs we noticed a lot of errors with XFS: [kern.warning] kernel: xfs_iunlink_remove: xfs_inotobp() returned an error 22 on dm-16. Returning error. [kern.notice] kernel: xfs_inactive:\011xfs_ifree() returned an error = 22 on dm-16 [kern.notice] kernel: xfs_force_shutdown(dm-16,0x1) called from line 1406 of file fs/xfs/xfs_vnodeops.c. Return address = 0x [kern.alert] kernel: Filesystem \"dm-16\": I/O Error Detected. Shutting down filesystem: dm-16 [kern.alert] kernel: Please umount the filesystem, and rectify the problem(s) [kern.warning] kernel: xfs_imap_to_bp: xfs_trans_read_buf()returned an error 5 on dm-16. Returning error. [kern.warning] kernel: xfs_imap_to_bp: xfs_trans_read_buf()returned an error 5 on dm-16. Returning error. [kern.warning] kernel: xfs_imap_to_bp: xfs_trans_read_buf()returned an error 5 on dm-16. Returning error. We are wondering if this is problem connected to hardware or rather this is XFS problem (if yes, was it fixed?). Best regards Piotr K _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: XFS: I/O Error Detected / 2.6.27.39 2010-11-16 13:10 XFS: I/O Error Detected / 2.6.27.39 Piotr Kandziora @ 2010-11-16 20:44 ` Emmanuel Florac 2010-11-17 10:43 ` Piotr Kandziora 2010-11-16 21:54 ` Eric Sandeen 1 sibling, 1 reply; 6+ messages in thread From: Emmanuel Florac @ 2010-11-16 20:44 UTC (permalink / raw) To: Piotr Kandziora; +Cc: Janusz, Artur Piechocki, lukasz.wittig, Bak, xfs Le Tue, 16 Nov 2010 14:10:51 +0100 vous écriviez: > Hi, > > Our environment is following: > - we have 24GB RAM, > - we are using 3ware controller (and it does not report any errors), What model? 9550SX? 9650SE? 9690SA? 9750 ? > - we have one big logical volume (20TB) exported via NFS with large > amount of small files (about 150k), > - we are doing periodically backup of this logical volume using rsync > to another server. > - we have kernel 2.6.27.39, What distribution, architecture? What is the version of xfs tools (try xfs_info -V for instance)? What are the xfs mount options? > Unfortunately our system is freezing unexpectedly without reason. What are the symptoms ? does the whole system freeze up? Or does it crash with kernel panic, or otherwise "Oops" messages? Older 3Ware cards (9550, early 9650) are prone to overheating and may fail. > We > started investigating this problem and noticed that cache memory is > slowly increasing. This is completely normal and expected. Linux uses up all available memory as a disk cache. > We tried to dump this cache memory using: > /bin/echo "3" > /proc/sys/vm/drop_caches > > In a result, cache was dumped, but in logs we noticed a lot of errors > with XFS: > > [kern.warning] kernel: xfs_iunlink_remove: xfs_inotobp() returned an > error 22 on dm-16. Returning error. > [kern.notice] kernel: xfs_inactive:\011xfs_ifree() returned an error > = 22 on dm-16 > [kern.notice] kernel: xfs_force_shutdown(dm-16,0x1) called from line > 1406 of file fs/xfs/xfs_vnodeops.c. Return address = 0x > [kern.alert] kernel: Filesystem \"dm-16\": I/O Error Detected. > Shutting down filesystem: dm-16 > [kern.alert] kernel: Please umount the filesystem, and rectify the > problem(s) > [kern.warning] kernel: xfs_imap_to_bp: xfs_trans_read_buf()returned > an error 5 on dm-16. Returning error. > [kern.warning] kernel: xfs_imap_to_bp: xfs_trans_read_buf()returned > an error 5 on dm-16. Returning error. > [kern.warning] kernel: xfs_imap_to_bp: xfs_trans_read_buf()returned > an error 5 on dm-16. Returning error. > > We are wondering if this is problem connected to hardware or rather > this is XFS problem (if yes, was it fixed?). This may be an xfs bug but more details would be necessary. -- ------------------------------------------------------------------------ Emmanuel Florac | Direction technique | Intellique | <eflorac@intellique.com> | +33 1 78 94 84 02 ------------------------------------------------------------------------ _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: XFS: I/O Error Detected / 2.6.27.39 2010-11-16 20:44 ` Emmanuel Florac @ 2010-11-17 10:43 ` Piotr Kandziora 2010-11-17 11:35 ` Emmanuel Florac 2010-11-17 21:17 ` Michael Monnerie 0 siblings, 2 replies; 6+ messages in thread From: Piotr Kandziora @ 2010-11-17 10:43 UTC (permalink / raw) To: Emmanuel Florac; +Cc: Artur Piechocki, lukasz.wittig, Janusz Bak, xfs Emmanuel, Below answers for your questions. > Le Tue, 16 Nov 2010 14:10:51 +0100 vous écriviez: > > >> Hi, >> >> Our environment is following: >> - we have 24GB RAM, >> - we are using 3ware controller (and it does not report any errors), >> > What model? 9550SX? 9650SE? 9690SA? 9750 ? > > 3ware 9650SE SATA-2 RAID PCIe supported by 3w_9xxx kernel module (version 2.26.08.006-2.6.28) - we have one big logical volume (20TB) exported via NFS with large >> amount of small files (about 150k), >> - we are doing periodically backup of this logical volume using rsync >> to another server. >> - we have kernel 2.6.27.39, >> > What distribution, architecture? What is the version of xfs tools > (try xfs_info -V for instance)? What are the xfs mount options? > > - debian based distribution with a lot of modification, - architecture x86_64, - xfs tools version 2.10.1 - mount options for this LV: rw,noatime,nodiratime,attr2,nobarrier,usrquota,prjquota,grpquota - NFS share is exported with following options: rw,no_root_squash,insecure,insecure_locks,async,anonuid=65534,anongid=65534,subtree_check >> Unfortunately our system is freezing unexpectedly without reason. >> > What are the symptoms ? does the whole system freeze up? Or does it > crash with kernel panic, or otherwise "Oops" messages? > Symptoms are different. One time we've got a few oom-killers: [kern.warning] kernel: load_average invoked oom-killer: gfp_mask=0xd0, order=0, oomkilladj=0 [kern.emerg] kernel: Pid: 19927, comm: load_average Not tainted 2.6.27.39-oe64-00000-g17059a5 #30 [kern.emerg] kernel: [kern.emerg] kernel: Call Trace: [kern.emerg] kernel: [<ffffffff80273418>] oom_kill_process+0x118/0x210 [kern.emerg] kernel: [<ffffffff802730b3>] badness+0x163/0x1e0 [kern.emerg] kernel: [<ffffffff80273926>] out_of_memory+0x1b6/0x230 [kern.emerg] kernel: [<ffffffff8027560d>] __alloc_pages_internal+0x3cd/0x430 [kern.emerg] kernel: [<ffffffff802934cd>] cache_alloc_refill+0x2bd/0x580 [kern.emerg] kernel: [<ffffffff802b59e0>] single_release+0x0/0x40 [kern.emerg] kernel: [<ffffffff80293960>] __kmalloc+0xf0/0x110 [kern.emerg] kernel: [<ffffffff802e7b4a>] stat_open+0x5a/0xc0 [kern.emerg] kernel: [<ffffffff802e066d>] proc_reg_open+0x8d/0x140 [kern.emerg] kernel: [<ffffffff802e05e0>] proc_reg_open+0x0/0x140 [kern.emerg] kernel: [<ffffffff80296038>] __dentry_open+0xb8/0x2e0 [kern.emerg] kernel: [<ffffffff80296306>] nameidata_to_filp+0x26/0x40 [kern.emerg] kernel: [<ffffffff802a1666>] do_filp_open+0x246/0x7b0 [kern.emerg] kernel: [<ffffffff802dff80>] proc_delete_inode+0x0/0x70 [kern.emerg] kernel: [<ffffffff8024a928>] wake_up_bit+0x18/0x40 [kern.emerg] kernel: [<ffffffff802afce1>] mntput_no_expire+0x21/0x120 [kern.emerg] kernel: [<ffffffff802aeccc>] alloc_fd+0x7c/0x130 [kern.emerg] kernel: [<ffffffff8029650c>] do_sys_open+0x5c/0xf0 [kern.emerg] kernel: [<ffffffff802cd4a4>] compat_sys_open+0x64/0xf0 [kern.emerg] kernel: [<ffffffff802cdff3>] compat_sys_select+0x133/0x180 [kern.emerg] kernel: [<ffffffff80228452>] ia32_sysret+0x0/0x5 [kern.warning] kernel: 3dm2 invoked oom-killer: gfp_mask=0xd0, order=0, oomkilladj=0 [kern.emerg] kernel: Pid: 18662, comm: 3dm2 Not tainted 2.6.27.39-oe64-00000-g17059a5 #30 [kern.emerg] kernel: [kern.emerg] kernel: Call Trace: [kern.emerg] kernel: [<ffffffff80273418>] oom_kill_process+0x118/0x210 [kern.emerg] kernel: [<ffffffff802730b3>] badness+0x163/0x1e0 [kern.emerg] kernel: [<ffffffff80273926>] out_of_memory+0x1b6/0x230 [kern.emerg] kernel: [<ffffffff8027560d>] __alloc_pages_internal+0x3cd/0x430 [auth.info] CRON[25434]: (pam_unix) session opened for user root by (uid=0) [kern.emerg] kernel: [<ffffffff802110dd>] dma_alloc_pages+0x1d/0x30 [kern.emerg] kernel: [<ffffffff802111f4>] dma_alloc_coherent+0x104/0x360 [kern.emerg] kernel: [<ffffffffa0091a7d>] twa_chrdev_ioctl+0x11d/0x7a0 [3w_9xxx] [kern.emerg] kernel: [<ffffffff802afce1>] mntput_no_expire+0x21/0x120 [kern.emerg] kernel: [<ffffffff8022b23c>] __dequeue_entity+0x6c/0xa0 [kern.emerg] kernel: [<ffffffff8022b4b7>] set_next_entity+0x47/0x50 [kern.emerg] kernel: [<ffffffff802a4a2d>] vfs_ioctl+0x7d/0xc0 [kern.emerg] kernel: [<ffffffff8024dd70>] hrtimer_wakeup+0x0/0x30 [kern.emerg] kernel: [<ffffffff802a4afb>] do_vfs_ioctl+0x8b/0x2e0 [kern.emerg] kernel: [<ffffffff802a4de1>] sys_ioctl+0x91/0xb0 [auth.info] CRON[25132]: (pam_unix) session closed for user root [kern.emerg] kernel: [<ffffffff8020c27b>] system_call_fastpath+0x16/0x1b another time call-trace: 2010/11/11 10:56:21|Pid: 4324, comm: nfsd Not tainted 2.6.27.39-oe64-00000-gc758227 #39 2010/11/11 10:56:21| 2010/11/11 10:56:21|Call Trace: 2010/11/11 10:56:21|[<ffffffff803d849b>] xfs_rename+0x28b/0x610 2010/11/11 10:56:21|[<ffffffff803da086>] xfs_trans_cancel+0x126/0x150 2010/11/11 10:56:21|[<ffffffff803d849b>] xfs_rename+0x28b/0x610 2010/11/11 10:56:21|[<ffffffff803eab2d>] xfs_vn_rename+0x7d/0xb0 2010/11/11 10:56:21|[<ffffffff8029f50b>] vfs_rename+0x41b/0x4c0 2010/11/11 10:56:21|[<ffffffff80350f94>] nfsd_rename+0x354/0x3a0 2010/11/11 10:56:21|[<ffffffff803584f3>] nfsd3_proc_rename+0xd3/0x1a0 2010/11/11 10:56:21|[<ffffffff8034a3d1>] nfsd_dispatch+0xb1/0x230 2010/11/11 10:56:21|[<ffffffff8066470a>] svc_process+0x47a/0x780 2010/11/11 10:56:21|[<ffffffff806968d2>] __down_read+0x12/0xa7 2010/11/11 10:56:21|[<ffffffff8034ab1a>] nfsd+0x17a/0x2a0 2010/11/11 10:56:21|[<ffffffff8034a9a0>] nfsd+0x0/0x2a0 2010/11/11 10:56:21|[<ffffffff802499ab>] kthread+0x4b/0x80 2010/11/11 10:56:21|[<ffffffff8020d149>] child_rip+0xa/0x11 2010/11/11 10:56:21|[<ffffffff80249960>] kthread+0x0/0x80 2010/11/11 10:56:21|[<ffffffff8020d13f>] child_rip+0x0/0x11 and after this call-trace series of messages: Filesystem "dm-37": xfs_log_force: error 5 returned. Filesystem "dm-37": xfs_log_force: error 5 returned. Filesystem "dm-37": xfs_log_force: error 5 returned. Filesystem "dm-37": xfs_log_force: error 5 returned. Filesystem "dm-37": xfs_log_force: error 5 returned. Filesystem "dm-37": xfs_log_force: error 5 returned. Filesystem "dm-37": xfs_log_force: error 5 returned. Filesystem "dm-37": xfs_log_force: error 5 returned. Filesystem "dm-37": xfs_log_force: error 5 returned. Filesystem "dm-37": xfs_log_force: error 5 returned. Filesystem "dm-37": xfs_log_force: error 5 returned. xfs_force_shutdown(dm-37,0x1) called from line 420 of file fs/xfs/xfs_rw.c. Return address = 0xffffffff803e2c39 Filesystem "dm-37": xfs_log_force: error 5 returned. Filesystem "dm-37": xfs_log_force: error 5 returned. xfs_force_shutdown(dm-37,0x1) called from line 420 of file fs/xfs/xfs_rw.c. Return address = 0xffffffff803e2c39 Filesystem "dm-37": xfs_log_force: error 5 returned. Filesystem "dm-37": xfs_log_force: error 5 returned. Filesystem "dm-37": xfs_log_force: error 5 returned. xfs_force_shutdown(dm-37,0x1) called from line 420 of file fs/xfs/xfs_rw.c. Return address = 0xffffffff803e2c39 > Older 3Ware cards (9550, early 9650) are prone to overheating and may > fail. > We've checked temperature on each disk using LSI/3ware CLI (tw_cli) and average is 35C. >> We >> started investigating this problem and noticed that cache memory is >> slowly increasing. >> > This is completely normal and expected. Linux uses up all available > memory as a disk cache. > > >> We tried to dump this cache memory using: >> /bin/echo "3"> /proc/sys/vm/drop_caches >> >> In a result, cache was dumped, but in logs we noticed a lot of errors >> with XFS: >> >> [kern.warning] kernel: xfs_iunlink_remove: xfs_inotobp() returned an >> error 22 on dm-16. Returning error. >> [kern.notice] kernel: xfs_inactive:\011xfs_ifree() returned an error >> = 22 on dm-16 >> [kern.notice] kernel: xfs_force_shutdown(dm-16,0x1) called from line >> 1406 of file fs/xfs/xfs_vnodeops.c. Return address = 0x >> [kern.alert] kernel: Filesystem \"dm-16\": I/O Error Detected. >> Shutting down filesystem: dm-16 >> [kern.alert] kernel: Please umount the filesystem, and rectify the >> problem(s) >> [kern.warning] kernel: xfs_imap_to_bp: xfs_trans_read_buf()returned >> an error 5 on dm-16. Returning error. >> [kern.warning] kernel: xfs_imap_to_bp: xfs_trans_read_buf()returned >> an error 5 on dm-16. Returning error. >> [kern.warning] kernel: xfs_imap_to_bp: xfs_trans_read_buf()returned >> an error 5 on dm-16. Returning error. >> >> We are wondering if this is problem connected to hardware or rather >> this is XFS problem (if yes, was it fixed?). >> > This may be an xfs bug but more details would be necessary. > > This problem occurred two times in the past, we repaired fs using xfs_repair (and it showed errors). We simulated it using dumping cache yesterday ... Best regards Piotr K _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: XFS: I/O Error Detected / 2.6.27.39 2010-11-17 10:43 ` Piotr Kandziora @ 2010-11-17 11:35 ` Emmanuel Florac 2010-11-17 21:17 ` Michael Monnerie 1 sibling, 0 replies; 6+ messages in thread From: Emmanuel Florac @ 2010-11-17 11:35 UTC (permalink / raw) To: Piotr Kandziora; +Cc: Janusz, Artur Piechocki, lukasz.wittig, Bak, xfs Le Wed, 17 Nov 2010 11:43:12 +0100 Piotr Kandziora <piotr.kandziora@open-e.com> écrivait: > - debian based distribution with a lot of modification, > - architecture x86_64, So this isn't a case of 32 bits rsync running out of memory (case seen...) > - xfs tools version 2.10.1 > - mount options for this LV: > rw,noatime,nodiratime,attr2,nobarrier,usrquota,prjquota,grpquota > - NFS share is exported with following options: > rw,no_root_squash,insecure,insecure_locks,async,anonuid=65534,anongid=65534,subtree_check > > >> Unfortunately our system is freezing unexpectedly without reason. > >> > > What are the symptoms ? does the whole system freeze up? Or does it > > crash with kernel panic, or otherwise "Oops" messages? > > > > Symptoms are different. One time we've got a few oom-killers: This is abnormal, but kernel disk cache shouldn't interfere anyway. > [kern.warning] kernel: load_average invoked oom-killer: > [kern.warning] kernel: 3dm2 invoked oom-killer: gfp_mask=0xd0, > 2010/11/11 10:56:21|Pid: 4324, comm: nfsd Not tainted > Filesystem "dm-37": xfs_log_force: error 5 returned. This looks like an IO error, as Eric Sandeen said. > We've checked temperature on each disk using LSI/3ware CLI (tw_cli) > and average is 35C. It's the card itself that's overheating and fails, not the disks. I had to replace quite a number of these on field after a year of use or two; they work well for a while, but crash when heavily sollicited. However this should cause a complete IO freeze. Did you try to run something like "tw_cli alarms" right after the failure? If tw_cli can't communicate with the controller anymore, than that's a case of a fried card. > This problem occurred two times in the past, we repaired fs using > xfs_repair (and it showed errors). We simulated it using dumping > cache yesterday ... I don't understand how or why dumping cache would cause the system to fail... except that it may augment actual disk IOs. -- ------------------------------------------------------------------------ Emmanuel Florac | Direction technique | Intellique | <eflorac@intellique.com> | +33 1 78 94 84 02 ------------------------------------------------------------------------ _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: XFS: I/O Error Detected / 2.6.27.39 2010-11-17 10:43 ` Piotr Kandziora 2010-11-17 11:35 ` Emmanuel Florac @ 2010-11-17 21:17 ` Michael Monnerie 1 sibling, 0 replies; 6+ messages in thread From: Michael Monnerie @ 2010-11-17 21:17 UTC (permalink / raw) To: xfs; +Cc: Piotr Kandziora, Artur Piechocki, Janusz Bak, lukasz.wittig [-- Attachment #1.1: Type: Text/Plain, Size: 564 bytes --] On Mittwoch, 17. November 2010 Piotr Kandziora wrote: > - xfs tools version 2.10.1 You should really use 3.1.x, but maybe "xfs tools" has a different numbering than "xfsprogs"? xfs_repair -V should show 3.1.x at least. -- mit freundlichen Grüssen, Michael Monnerie, Ing. BSc it-management Internet Services: Protéger http://proteger.at [gesprochen: Prot-e-schee] Tel: +43 660 / 415 6531 // ****** Radiointerview zum Thema Spam ****** // http://www.it-podcast.at/archiv.html#podcast-100716 // // Haus zu verkaufen: http://zmi.at/langegg/ [-- Attachment #1.2: This is a digitally signed message part. --] [-- Type: application/pgp-signature, Size: 198 bytes --] [-- Attachment #2: Type: text/plain, Size: 121 bytes --] _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: XFS: I/O Error Detected / 2.6.27.39 2010-11-16 13:10 XFS: I/O Error Detected / 2.6.27.39 Piotr Kandziora 2010-11-16 20:44 ` Emmanuel Florac @ 2010-11-16 21:54 ` Eric Sandeen 1 sibling, 0 replies; 6+ messages in thread From: Eric Sandeen @ 2010-11-16 21:54 UTC (permalink / raw) To: Piotr Kandziora; +Cc: Artur Piechocki, lukasz.wittig, Janusz Bak, xfs On 11/16/10 7:10 AM, Piotr Kandziora wrote: > Hi, > > Our environment is following: > - we have 24GB RAM, > - we are using 3ware controller (and it does not report any errors), > - we have one big logical volume (20TB) exported via NFS with large amount of small files (about 150k), > - we are doing periodically backup of this logical volume using rsync to another server. > - we have kernel 2.6.27.39, > > Unfortunately our system is freezing unexpectedly without reason. We started investigating this problem and noticed that cache memory is slowly increasing. > > We tried to dump this cache memory using: > /bin/echo "3" > /proc/sys/vm/drop_caches > > In a result, cache was dumped, but in logs we noticed a lot of errors with XFS: > > [kern.warning] kernel: xfs_iunlink_remove: xfs_inotobp() returned an error 22 on dm-16. Returning error. so this got EINVAL: #define EINVAL 22 /* Invalid argument */ > [kern.notice] kernel: xfs_inactive:\011xfs_ifree() returned an error = 22 on dm-16 > [kern.notice] kernel: xfs_force_shutdown(dm-16,0x1) called from line 1406 of file fs/xfs/xfs_vnodeops.c. Return address = 0x > [kern.alert] kernel: Filesystem \"dm-16\": I/O Error Detected. Shutting down filesystem: dm-16 which shut down the filesystem: > [kern.alert] kernel: Please umount the filesystem, and rectify the problem(s) leading to IO errors after that: #define EIO 5 /* I/O error */ > [kern.warning] kernel: xfs_imap_to_bp: xfs_trans_read_buf()returned an error 5 on dm-16. Returning error. > [kern.warning] kernel: xfs_imap_to_bp: xfs_trans_read_buf()returned an error 5 on dm-16. Returning error. > [kern.warning] kernel: xfs_imap_to_bp: xfs_trans_read_buf()returned an error 5 on dm-16. Returning error. > > We are wondering if this is problem connected to hardware or rather this is XFS problem (if yes, was it fixed?). I'd probably start with xfs_repair (with -n if you want a dry run) and see if it finds any metadata corruption, first. -Eric > Best regards > Piotr K > > _______________________________________________ > xfs mailing list > xfs@oss.sgi.com > http://oss.sgi.com/mailman/listinfo/xfs > _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2010-11-17 21:16 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2010-11-16 13:10 XFS: I/O Error Detected / 2.6.27.39 Piotr Kandziora 2010-11-16 20:44 ` Emmanuel Florac 2010-11-17 10:43 ` Piotr Kandziora 2010-11-17 11:35 ` Emmanuel Florac 2010-11-17 21:17 ` Michael Monnerie 2010-11-16 21:54 ` Eric Sandeen
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox