* Re: 2.6.32-rc1: various BUGs on resume from hibernation [not found] <4AC8B4A8.3050601@tuffmail.co.uk> @ 2009-10-04 17:03 ` Alan Jenkins 2009-10-04 20:47 ` Rafael J. Wysocki 0 siblings, 1 reply; 10+ messages in thread From: Alan Jenkins @ 2009-10-04 17:03 UTC (permalink / raw) To: pm list; +Cc: Kernel Testers List, Linux Kernel Mailing List Alan Jenkins wrote: > I'm seeing a variety of BUGs on my EeePC 701 after hibernation. > Sometimes they cause a hang during resume; sometimes they happen just > after resume. It doesn't happen all the time either - I've just > hibernated three times in a row with no problems. It's most perplexing. > > One resume hang showed a series of SCSI backtraces and errors. > Unfortunately I wasn't able to capture it at the time. They were most > probably related to the root device, an SSD controlled by ata_piix. > > Today on latest -git I captured this "bad swap file entry" problem. I > use a swap file for hibernation (I don't have a swap partition). > Previously, on a slightly older kernel (still after 32-rc1), I > captured this BUG in fget_light(): And later suspend to ram failed, and then showed a hung task warning. I've attached everything at <http://bugzilla.kernel.org/show_bug.cgi?id=14320>. Thanks Alan ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: 2.6.32-rc1: various BUGs on resume from hibernation 2009-10-04 17:03 ` 2.6.32-rc1: various BUGs on resume from hibernation Alan Jenkins @ 2009-10-04 20:47 ` Rafael J. Wysocki 2009-10-05 7:53 ` Stefan Seyfried 0 siblings, 1 reply; 10+ messages in thread From: Rafael J. Wysocki @ 2009-10-04 20:47 UTC (permalink / raw) To: Alan Jenkins; +Cc: pm list, Kernel Testers List, Linux Kernel Mailing List On Sunday 04 October 2009, Alan Jenkins wrote: > Alan Jenkins wrote: > > I'm seeing a variety of BUGs on my EeePC 701 after hibernation. > > Sometimes they cause a hang during resume; sometimes they happen just > > after resume. It doesn't happen all the time either - I've just > > hibernated three times in a row with no problems. It's most perplexing. > > > > One resume hang showed a series of SCSI backtraces and errors. > > Unfortunately I wasn't able to capture it at the time. They were most > > probably related to the root device, an SSD controlled by ata_piix. > > > > Today on latest -git I captured this "bad swap file entry" problem. I > > use a swap file for hibernation (I don't have a swap partition). > > > Previously, on a slightly older kernel (still after 32-rc1), I > > captured this BUG in fget_light(): > > And later suspend to ram failed, and then showed a hung task warning. As I already asked in the Bugzilla, can you verify that this is reproducible without KMS? Best, Rafael ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: 2.6.32-rc1: various BUGs on resume from hibernation 2009-10-04 20:47 ` Rafael J. Wysocki @ 2009-10-05 7:53 ` Stefan Seyfried 2009-10-05 8:08 ` Alan Jenkins 2009-10-05 20:02 ` Alan Jenkins 0 siblings, 2 replies; 10+ messages in thread From: Stefan Seyfried @ 2009-10-05 7:53 UTC (permalink / raw) To: Rafael J. Wysocki Cc: pm list, Kernel Testers List, Alan Jenkins, Linux Kernel Mailing List On Sun, 4 Oct 2009 22:47:43 +0200 "Rafael J. Wysocki" <rjw@sisk.pl> wrote: > On Sunday 04 October 2009, Alan Jenkins wrote: > > Alan Jenkins wrote: > > > I'm seeing a variety of BUGs on my EeePC 701 after hibernation. > > > Sometimes they cause a hang during resume; sometimes they happen > > > just after resume. It doesn't happen all the time either - I've > > > just hibernated three times in a row with no problems. It's most > > > perplexing. > As I already asked in the Bugzilla, can you verify that this is > reproducible without KMS? This is important, there *are* hibernation issues with intel/KMS, e.g. http://bugs.freedesktop.org/show_bug.cgi?id=23701 -- Stefan Seyfried "Any ideas, John?" "Well, surrounding them's out." ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: 2.6.32-rc1: various BUGs on resume from hibernation 2009-10-05 7:53 ` Stefan Seyfried @ 2009-10-05 8:08 ` Alan Jenkins 2009-10-05 8:10 ` Alan Jenkins 2009-10-05 20:02 ` Alan Jenkins 1 sibling, 1 reply; 10+ messages in thread From: Alan Jenkins @ 2009-10-05 8:08 UTC (permalink / raw) To: Stefan Seyfried; +Cc: pm list, Kernel Testers List, Linux Kernel Mailing List Stefan Seyfried wrote: > On Sun, 4 Oct 2009 22:47:43 +0200 > "Rafael J. Wysocki" <rjw@sisk.pl> wrote: > >> On Sunday 04 October 2009, Alan Jenkins wrote: >> >>> Alan Jenkins wrote: >>> >>>> I'm seeing a variety of BUGs on my EeePC 701 after hibernation. >>>> Sometimes they cause a hang during resume; sometimes they happen >>>> just after resume. It doesn't happen all the time either - I've >>>> just hibernated three times in a row with no problems. It's most >>>> perplexing. >>>> > > >> As I already asked in the Bugzilla, can you verify that this is >> reproducible without KMS? >> > > This is important, there *are* hibernation issues with intel/KMS, > e.g. http://bugs.freedesktop.org/show_bug.cgi?id=23701 > Sure, it's definitely on my suspect list. I'm now booting with i915.modeset=1, which gets me back to slow and flickery VT switches again :-). Thanks Alan ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: 2.6.32-rc1: various BUGs on resume from hibernation 2009-10-05 8:08 ` Alan Jenkins @ 2009-10-05 8:10 ` Alan Jenkins 0 siblings, 0 replies; 10+ messages in thread From: Alan Jenkins @ 2009-10-05 8:10 UTC (permalink / raw) To: Stefan Seyfried; +Cc: pm list, Kernel Testers List, Linux Kernel Mailing List On 10/5/09, Alan Jenkins <alan-jenkins@tuffmail.co.uk> wrote: > Stefan Seyfried wrote: >> On Sun, 4 Oct 2009 22:47:43 +0200 >> "Rafael J. Wysocki" <rjw@sisk.pl> wrote: >> >>> On Sunday 04 October 2009, Alan Jenkins wrote: >>> >>>> Alan Jenkins wrote: >>>> >>>>> I'm seeing a variety of BUGs on my EeePC 701 after hibernation. >>>>> Sometimes they cause a hang during resume; sometimes they happen >>>>> just after resume. It doesn't happen all the time either - I've >>>>> just hibernated three times in a row with no problems. It's most >>>>> perplexing. >>>>> >> >> >>> As I already asked in the Bugzilla, can you verify that this is >>> reproducible without KMS? >>> >> >> This is important, there *are* hibernation issues with intel/KMS, >> e.g. http://bugs.freedesktop.org/show_bug.cgi?id=23701 >> > > Sure, it's definitely on my suspect list. I'm now booting with > i915.modeset=1 heh, I mean =0 , which gets me back to slow and flickery VT switches > again :-). > > Thanks > Alan > -- > To unsubscribe from this list: send the line "unsubscribe kernel-testers" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: 2.6.32-rc1: various BUGs on resume from hibernation 2009-10-05 7:53 ` Stefan Seyfried 2009-10-05 8:08 ` Alan Jenkins @ 2009-10-05 20:02 ` Alan Jenkins 2009-10-05 20:59 ` Alan Jenkins 1 sibling, 1 reply; 10+ messages in thread From: Alan Jenkins @ 2009-10-05 20:02 UTC (permalink / raw) To: Stefan Seyfried; +Cc: pm list, Kernel Testers List, Linux Kernel Mailing List Stefan Seyfried wrote: > On Sun, 4 Oct 2009 22:47:43 +0200 > "Rafael J. Wysocki" <rjw@sisk.pl> wrote: > >> On Sunday 04 October 2009, Alan Jenkins wrote: >> >>> Alan Jenkins wrote: >>> >>>> I'm seeing a variety of BUGs on my EeePC 701 after hibernation. >>>> Sometimes they cause a hang during resume; sometimes they happen >>>> just after resume. It doesn't happen all the time either - I've >>>> just hibernated three times in a row with no problems. It's most >>>> perplexing. >>>> > > >> As I already asked in the Bugzilla, can you verify that this is >> reproducible without KMS? >> No. I couldn't reproduce it without KMS. Then I re-enabled KMS, hibernated a couple of times, and had another resume hang (different again - this time in the freezer). ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: 2.6.32-rc1: various BUGs on resume from hibernation 2009-10-05 20:02 ` Alan Jenkins @ 2009-10-05 20:59 ` Alan Jenkins 2009-10-05 22:48 ` Rafael J. Wysocki 0 siblings, 1 reply; 10+ messages in thread From: Alan Jenkins @ 2009-10-05 20:59 UTC (permalink / raw) To: Stefan Seyfried; +Cc: pm list, Kernel Testers List, Linux Kernel Mailing List Alan Jenkins wrote: > Stefan Seyfried wrote: >> On Sun, 4 Oct 2009 22:47:43 +0200 >> "Rafael J. Wysocki" <rjw@sisk.pl> wrote: >>> On Sunday 04 October 2009, Alan Jenkins wrote: >>>> Alan Jenkins wrote: >>>>> I'm seeing a variety of BUGs on my EeePC 701 after hibernation. >>>>> Sometimes they cause a hang during resume; sometimes they happen >>>>> just after resume. It doesn't happen all the time either - I've >>>>> just hibernated three times in a row with no problems. It's most >>>>> perplexing. >> >>> As I already asked in the Bugzilla, can you verify that this is >>> reproducible without KMS? > > No. I couldn't reproduce it without KMS. Then I re-enabled KMS, > hibernated a couple of times, and had another resume hang (different > again - this time in the freezer). > Perhaps this GEM backtrace is more relevant than the others, or perhaps not. I closed the lid during resume from hibernation; the screen stayed black when I re-opened it, but syslogd recorded the following. (Again, I've done the exact thing before and often nothing unusual happens) - 207.793317] ACPI: Waking up from system sleep state S4 208.480594] [drm] LVDS-8: set mode 800x480 1c 209.022770] HDA Intel 0000:00:1b.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16 ... 209.653483] usb 1-8: reset high speed USB device using ehci_hcd and address 3 210.134607] Restarting tasks ... done. 210.743796] ADDRCONF(NETDEV_UP): eth0: link is not ready 210.860322] ADDRCONF(NETDEV_UP): wlan0: link is not ready 211.352971] *pdpt = 00000000175e7001 *pde = 0000000000000000 211.353006] Modules linked in: af_packet fbcon tileblit font bitblit softcursor i915 drm_kms_helper drm i2c_algo_bit cfbcopyarea cfbimgblt cfbfillrect ipv6 loop joydev snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep snd_pcm_oss ath5k snd_mixer_oss mac80211 ath snd_pcm uvcvideo cfg80211 psmouse i2c_i801 snd_timer videodev v4l1_compat eeepc_laptop snd serio_raw pcspkr i2c_core rfkill battery ac pci_hotplug snd_page_alloc evdev processor intel_agp video backlight output agpgart button thermal fan [last unloaded: scsi_wait_scan] 211.353126] 211.353137] Pid: 1933, comm: Xorg Not tainted (2.6.32-rc2eeepc-test #505) 701 211.353147] EIP: 0060:[<e02930fa>] EFLAGS: 00213246 CPU: 0 211.353187] EIP is at i915_gem_object_get_pages+0x96/0xf4 [i915] (i915_gem_object_get_pages (drivers/gpu/drm/i915/i915_gem.c:2238) 00009064 <i915_gem_object_get_pages>: ... (0x9064 + 0x96 = 0x90fa) ... 90de: 85 d2 test %edx,%edx 90e0: 89 53 24 mov %edx,0x24(%ebx) 90e3: 75 0a jne 90ef <i915_gem_object_get_pages+0x8b> 90e5: ff 4b 28 decl 0x28(%ebx) 90e8: be f4 ff ff ff mov $0xfffffff4,%esi 90ed: eb 5f jmp 914e <i915_gem_object_get_pages+0xea> 90ef: 8b 55 e8 mov -0x18(%ebp),%edx 90f2: 31 ff xor %edi,%edi 90f4: 8b 42 0c mov 0xc(%edx),%eax 90f7: 8b 40 0c mov 0xc(%eax),%eax 90fa: 8b 40 0c mov 0xc(%eax),%eax 90fd: 8b 80 98 00 00 00 mov 0x98(%eax),%eax 9103: 89 45 f0 mov %eax,-0x10(%ebp) 9106: eb 31 jmp 9139 <i915_gem_object_get_pages+0xd5> 9108: 8b 55 f0 mov -0x10(%ebp),%edx 910b: 8b 42 30 mov 0x30(%edx),%eax 910e: 89 fa mov %edi,%edx 9110: 8b 48 04 mov 0x4(%eax),%ecx 9113: 8b 45 f0 mov -0x10(%ebp),%eax 9116: 6a 00 push $0x0 9118: e8 fc ff ff ff call 9119 <i915_gem_object_get_pages+0xb5> ... 211.353197] EAX: 00000000 EBX: dbeafb80 ECX: 00000008 EDX: dbeb68a0 211.353206] ESI: dbeb68a0 EDI: 00000000 EBP: d74cfd8c ESP: d74cfd74 211.353216] DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 0068 211.353239] dbeb68a0 00000001 dbeafb80 dbeafb80 dbeb68a0 dbd53c00 d74cfdb4 e029435c 211.353258] <0> dc341d28 00001000 dc341000 00296e52 fffffff0 dbe22800 dbd53c00 dbeafb80 211.353277] <0> d74cfdcc e029447f dbeb68a0 dbe22800 dbeafb80 de2baa20 d74cfe80 e0297090 211.353323] [<e029435c>] ? i915_gem_object_bind_to_gtt+0x10a/0x210 [i915] (drivers/gpu/drm/i915/i915_gem.c:2615) 211.353323] [<e029447f>] ? i915_gem_object_pin+0x1d/0x117 [i915] (drivers/gpu/drm/i915/i915_gem.c:3857) 211.353323] [<e0297090>] ? i915_gem_execbuffer+0x4af/0x1031 [i915] (drivers/gpu/drm/i915/i915_gem.c:3178) 211.353323] [<c02ed818>] ? __kfree_skb+0x66/0x69 211.353323] [<e029c95b>] ? intel_mark_busy+0x3a/0xb3 [i915] (drivers/gpu/drm/i915/intel_display.c:2263) 211.353323] [<e01c155c>] ? drm_ioctl+0x1e8/0x259 [drm] (drivers/gpu/drm/drm_drv.c:476) 211.353323] [<e0296be1>] ? i915_gem_execbuffer+0x0/0x1031 [i915] 211.353323] [<c010aae7>] ? sched_clock+0xb/0x1c 211.353323] [<c0142b23>] ? sched_clock_local+0x11/0x134 211.353323] [<e01c1374>] ? drm_ioctl+0x0/0x259 [drm] 211.353323] [<c01a47f8>] ? vfs_ioctl+0x3c/0x4f 211.353323] [<c01a507c>] ? do_vfs_ioctl+0x474/0x4ac 211.353323] [<c0123fad>] ? set_next_entity+0x1d/0x83 211.353323] [<c0124097>] ? pick_next_task_fair+0x5b/0x6e 211.353323] [<c0343fa4>] ? schedule+0x175/0x3e6 211.353323] [<c01a50e0>] ? sys_ioctl+0x2c/0x45 211.353323] [<c0107154>] ? sysenter_do_call+0x12/0x28 211.353960] ---[ end trace 99be0f1305a0c4a1 ]--- Oct 5 21:10:37 alan-eeepc kernel: [ 219.229840] ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready Oct 5 21:11:13 alan-eeepc kernel: [ 255.136410] SysRq : Emergency Sync ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: 2.6.32-rc1: various BUGs on resume from hibernation 2009-10-05 20:59 ` Alan Jenkins @ 2009-10-05 22:48 ` Rafael J. Wysocki 2009-10-06 0:06 ` Rafael J. Wysocki 2009-10-06 8:58 ` Alan Jenkins 0 siblings, 2 replies; 10+ messages in thread From: Rafael J. Wysocki @ 2009-10-05 22:48 UTC (permalink / raw) To: Alan Jenkins Cc: pm list, Kernel Testers List, Linux Kernel Mailing List, Jesse Barnes, Stefan Seyfried On Monday 05 October 2009, Alan Jenkins wrote: > Alan Jenkins wrote: > > Stefan Seyfried wrote: > >> On Sun, 4 Oct 2009 22:47:43 +0200 > >> "Rafael J. Wysocki" <rjw@sisk.pl> wrote: > >>> On Sunday 04 October 2009, Alan Jenkins wrote: > >>>> Alan Jenkins wrote: > >>>>> I'm seeing a variety of BUGs on my EeePC 701 after hibernation. > >>>>> Sometimes they cause a hang during resume; sometimes they happen > >>>>> just after resume. It doesn't happen all the time either - I've > >>>>> just hibernated three times in a row with no problems. It's most > >>>>> perplexing. > >> > >>> As I already asked in the Bugzilla, can you verify that this is > >>> reproducible without KMS? > > > > No. I couldn't reproduce it without KMS. Then I re-enabled KMS, > > hibernated a couple of times, and had another resume hang (different > > again - this time in the freezer). > > > > Perhaps this GEM backtrace is more relevant than the others, or perhaps > not. I closed the lid during resume from hibernation; the screen stayed > black when I re-opened it, but syslogd recorded the following. (Again, > I've done the exact thing before and often nothing unusual happens) - OK, let's assume it's related to KMS, then. 1. Do we already have a Bugzilla entry for it? 2. Did it work with 2.6.31 (and with KMS)? 3. Are you aware of http://marc.info/?l=linux-kernel&m=125452324520623&w=4 ? Thanks, Rafael ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: 2.6.32-rc1: various BUGs on resume from hibernation 2009-10-05 22:48 ` Rafael J. Wysocki @ 2009-10-06 0:06 ` Rafael J. Wysocki 2009-10-06 8:58 ` Alan Jenkins 1 sibling, 0 replies; 10+ messages in thread From: Rafael J. Wysocki @ 2009-10-06 0:06 UTC (permalink / raw) To: linux-pm Cc: Kernel Testers List, Alan Jenkins, Linux Kernel Mailing List, Jesse Barnes, Stefan Seyfried On Tuesday 06 October 2009, Rafael J. Wysocki wrote: > On Monday 05 October 2009, Alan Jenkins wrote: > > Alan Jenkins wrote: > > > Stefan Seyfried wrote: > > >> On Sun, 4 Oct 2009 22:47:43 +0200 > > >> "Rafael J. Wysocki" <rjw@sisk.pl> wrote: > > >>> On Sunday 04 October 2009, Alan Jenkins wrote: > > >>>> Alan Jenkins wrote: > > >>>>> I'm seeing a variety of BUGs on my EeePC 701 after hibernation. > > >>>>> Sometimes they cause a hang during resume; sometimes they happen > > >>>>> just after resume. It doesn't happen all the time either - I've > > >>>>> just hibernated three times in a row with no problems. It's most > > >>>>> perplexing. > > >> > > >>> As I already asked in the Bugzilla, can you verify that this is > > >>> reproducible without KMS? > > > > > > No. I couldn't reproduce it without KMS. Then I re-enabled KMS, > > > hibernated a couple of times, and had another resume hang (different > > > again - this time in the freezer). > > > > > > > Perhaps this GEM backtrace is more relevant than the others, or perhaps > > not. I closed the lid during resume from hibernation; the screen stayed > > black when I re-opened it, but syslogd recorded the following. (Again, > > I've done the exact thing before and often nothing unusual happens) - > > OK, let's assume it's related to KMS, then. > > 1. Do we already have a Bugzilla entry for it? Ah, yes, we do, http://bugzilla.kernel.org/show_bug.cgi?id=14320 (I tend to forget about these things). > 2. Did it work with 2.6.31 (and with KMS)? > 3. Are you aware of http://marc.info/?l=linux-kernel&m=125452324520623&w=4 ? Thanks, Rafael ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: 2.6.32-rc1: various BUGs on resume from hibernation 2009-10-05 22:48 ` Rafael J. Wysocki 2009-10-06 0:06 ` Rafael J. Wysocki @ 2009-10-06 8:58 ` Alan Jenkins 1 sibling, 0 replies; 10+ messages in thread From: Alan Jenkins @ 2009-10-06 8:58 UTC (permalink / raw) To: Rafael J. Wysocki Cc: pm list, Kernel Testers List, Linux Kernel Mailing List, Jesse Barnes, Stefan Seyfried Rafael J. Wysocki wrote: > On Monday 05 October 2009, Alan Jenkins wrote: > >> Alan Jenkins wrote: >> >>> Stefan Seyfried wrote: >>> >>>> On Sun, 4 Oct 2009 22:47:43 +0200 >>>> "Rafael J. Wysocki" <rjw@sisk.pl> wrote: >>>> >>>>> On Sunday 04 October 2009, Alan Jenkins wrote: >>>>> >>>>>> Alan Jenkins wrote: >>>>>> >>>>>>> I'm seeing a variety of BUGs on my EeePC 701 after hibernation. >>>>>>> Sometimes they cause a hang during resume; sometimes they happen >>>>>>> just after resume. It doesn't happen all the time either - I've >>>>>>> just hibernated three times in a row with no problems. It's most >>>>>>> perplexing. >>>>>>> >>>>> As I already asked in the Bugzilla, can you verify that this is >>>>> reproducible without KMS? >>>>> >>> No. I couldn't reproduce it without KMS. Then I re-enabled KMS, >>> hibernated a couple of times, and had another resume hang (different >>> again - this time in the freezer). >>> >>> >> Perhaps this GEM backtrace is more relevant than the others, or perhaps >> not. I closed the lid during resume from hibernation; the screen stayed >> black when I re-opened it, but syslogd recorded the following. (Again, >> I've done the exact thing before and often nothing unusual happens) - >> > > OK, let's assume it's related to KMS, then. > > 1. Do we already have a Bugzilla entry for it? > 2. Did it work with 2.6.31 (and with KMS)? > I thought so, but my recollection is hazy. I didn't test it for very long if I did. I've tried it now and 2.6.31 behaves pretty similarly. Sorry for ringing the regression bell. Firstly the _hibernation_ process hung (after a couple of suspend-to-ram cycles). No text on the console (despite using s2disk). It echoed keypresses and responds to SysRq keys. No messages from lockdep or the hung task detector (after waiting 5 minutes). SysRq-P said we're in the idle loop; SysRq-T said both events/0 and hald-addon-input were runnable. Then suspend-to-ram hung (following a hibernation cycle). This time it showed the contents of vt1, but didn't appear to respond to anything short of SysRq+B. > 3. Are you aware of http://marc.info/?l=linux-kernel&m=125452324520623&w=4 ? > > Thanks, > Rafael > I guess there's no connection. That issue seems to be about generally flickery screens. I've seen a few rare flickers that might be related. But in general nothing bad happens until I try to suspend, and the badness is in the form of random crashes or hangs. Also that bug is confirmed to be a regression, and my issue isn't :-). Many thanks Alan ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2009-10-06 8:58 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <4AC8B4A8.3050601@tuffmail.co.uk>
2009-10-04 17:03 ` 2.6.32-rc1: various BUGs on resume from hibernation Alan Jenkins
2009-10-04 20:47 ` Rafael J. Wysocki
2009-10-05 7:53 ` Stefan Seyfried
2009-10-05 8:08 ` Alan Jenkins
2009-10-05 8:10 ` Alan Jenkins
2009-10-05 20:02 ` Alan Jenkins
2009-10-05 20:59 ` Alan Jenkins
2009-10-05 22:48 ` Rafael J. Wysocki
2009-10-06 0:06 ` Rafael J. Wysocki
2009-10-06 8:58 ` Alan Jenkins
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).