* INFO: task journal:337 blocked for more than 120 seconds
@ 2009-09-30 21:11 Shirley Ma
2009-10-01 13:20 ` Marcelo Tosatti
` (3 more replies)
0 siblings, 4 replies; 22+ messages in thread
From: Shirley Ma @ 2009-09-30 21:11 UTC (permalink / raw)
To: virtualization, kvm, linux-kernel
Hello all,
Anybody found this problem before? I kept hitting this issue for 2.6.31
guest kernel even with a simple network test.
INFO: task kjournal:337 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_sec" disables this message.
kjournald D 00000041 0 337 2 0x00000000
My test is totally being blocked.
Thanks
Shirley
^ permalink raw reply [flat|nested] 22+ messages in thread* Re: INFO: task journal:337 blocked for more than 120 seconds 2009-09-30 21:11 INFO: task journal:337 blocked for more than 120 seconds Shirley Ma @ 2009-10-01 13:20 ` Marcelo Tosatti 2009-10-01 16:58 ` Shirley Ma 2009-10-01 16:58 ` Shirley Ma 2009-10-01 13:20 ` INFO: task journal:337 " Marcelo Tosatti ` (2 subsequent siblings) 3 siblings, 2 replies; 22+ messages in thread From: Marcelo Tosatti @ 2009-10-01 13:20 UTC (permalink / raw) To: Shirley Ma; +Cc: virtualization, kvm, linux-kernel On Wed, Sep 30, 2009 at 02:11:52PM -0700, Shirley Ma wrote: > Hello all, > > Anybody found this problem before? I kept hitting this issue for 2.6.31 > guest kernel even with a simple network test. > > INFO: task kjournal:337 blocked for more than 120 seconds. > "echo 0 > /proc/sys/kernel/hung_task_timeout_sec" disables this message. > > kjournald D 00000041 0 337 2 0x00000000 > > My test is totally being blocked. I've hit this in the past with ext3, mounting with data=writeback made it disappear. ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: INFO: task journal:337 blocked for more than 120 seconds 2009-10-01 13:20 ` Marcelo Tosatti @ 2009-10-01 16:58 ` Shirley Ma 2009-10-01 16:58 ` Shirley Ma 1 sibling, 0 replies; 22+ messages in thread From: Shirley Ma @ 2009-10-01 16:58 UTC (permalink / raw) To: Marcelo Tosatti; +Cc: mcao, linux-kernel, kvm, virtualization On Thu, 2009-10-01 at 10:20 -0300, Marcelo Tosatti wrote: > I've hit this in the past with ext3, mounting with data=writeback made > it > disappear. Thanks. I will make a try. Someone should fix this. Shirley ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: INFO: task journal:337 blocked for more than 120 seconds 2009-10-01 13:20 ` Marcelo Tosatti 2009-10-01 16:58 ` Shirley Ma @ 2009-10-01 16:58 ` Shirley Ma 2009-10-01 21:00 ` INFO: task kjournal:337 " Shirley Ma 2009-10-01 21:00 ` Shirley Ma 1 sibling, 2 replies; 22+ messages in thread From: Shirley Ma @ 2009-10-01 16:58 UTC (permalink / raw) To: Marcelo Tosatti; +Cc: virtualization, kvm, linux-kernel, mcao On Thu, 2009-10-01 at 10:20 -0300, Marcelo Tosatti wrote: > I've hit this in the past with ext3, mounting with data=writeback made > it > disappear. Thanks. I will make a try. Someone should fix this. Shirley ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: INFO: task kjournal:337 blocked for more than 120 seconds 2009-10-01 16:58 ` Shirley Ma @ 2009-10-01 21:00 ` Shirley Ma 2009-10-01 21:00 ` Shirley Ma 1 sibling, 0 replies; 22+ messages in thread From: Shirley Ma @ 2009-10-01 21:00 UTC (permalink / raw) To: linux-kernel; +Cc: mcao, kvm, virtualization I talked to Mingming, she suggested to use different IO scheduler. The default scheduler is cfg, after I switch to noop, the problem is gone. So there seems a bug in cfg scheduler. It's easily reproduced it when running the guest kernel, so far I haven't hit this problem on the host side. If I need to file a bug for some one to look at, please let me know. Thanks Shirley ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: INFO: task kjournal:337 blocked for more than 120 seconds 2009-10-01 16:58 ` Shirley Ma 2009-10-01 21:00 ` INFO: task kjournal:337 " Shirley Ma @ 2009-10-01 21:00 ` Shirley Ma 2009-10-01 21:03 ` Javier Guerra 2009-10-01 21:03 ` Javier Guerra 1 sibling, 2 replies; 22+ messages in thread From: Shirley Ma @ 2009-10-01 21:00 UTC (permalink / raw) To: linux-kernel; +Cc: Marcelo Tosatti, virtualization, kvm, mcao I talked to Mingming, she suggested to use different IO scheduler. The default scheduler is cfg, after I switch to noop, the problem is gone. So there seems a bug in cfg scheduler. It's easily reproduced it when running the guest kernel, so far I haven't hit this problem on the host side. If I need to file a bug for some one to look at, please let me know. Thanks Shirley ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: INFO: task kjournal:337 blocked for more than 120 seconds 2009-10-01 21:00 ` Shirley Ma @ 2009-10-01 21:03 ` Javier Guerra 2009-10-01 21:17 ` Shirley Ma 2009-10-01 21:17 ` Shirley Ma 2009-10-01 21:03 ` Javier Guerra 1 sibling, 2 replies; 22+ messages in thread From: Javier Guerra @ 2009-10-01 21:03 UTC (permalink / raw) To: Shirley Ma; +Cc: linux-kernel, Marcelo Tosatti, virtualization, kvm, mcao On Thu, Oct 1, 2009 at 4:00 PM, Shirley Ma <mashirle@us.ibm.com> wrote: > I talked to Mingming, she suggested to use different IO scheduler. The > default scheduler is cfg, after I switch to noop, the problem is gone. deadline is the most recommended one for virtualization hosts. some distros set it as default if you select Xen or KVM at installation time. (and noop for the guests) -- Javier ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: INFO: task kjournal:337 blocked for more than 120 seconds 2009-10-01 21:03 ` Javier Guerra @ 2009-10-01 21:17 ` Shirley Ma 2009-10-01 22:09 ` Shirley Ma 2009-10-01 22:09 ` Shirley Ma 2009-10-01 21:17 ` Shirley Ma 1 sibling, 2 replies; 22+ messages in thread From: Shirley Ma @ 2009-10-01 21:17 UTC (permalink / raw) To: Javier Guerra; +Cc: linux-kernel, Marcelo Tosatti, virtualization, kvm, mcao On Thu, 2009-10-01 at 16:03 -0500, Javier Guerra wrote: > deadline is the most recommended one for virtualization hosts. some > distros set it as default if you select Xen or KVM at installation > time. (and noop for the guests) I spoke too earlier, after a while noop scheduler hit the same issue. I am switching to deadline to test it again. Thanks Shirley ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: INFO: task kjournal:337 blocked for more than 120 seconds 2009-10-01 21:17 ` Shirley Ma @ 2009-10-01 22:09 ` Shirley Ma 2009-10-01 22:09 ` Shirley Ma 1 sibling, 0 replies; 22+ messages in thread From: Shirley Ma @ 2009-10-01 22:09 UTC (permalink / raw) To: Javier Guerra; +Cc: linux-kernel, Marcelo Tosatti, virtualization, kvm, mcao Switching to different scheduler doesn't make the problem gone away. Shirley ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: INFO: task kjournal:337 blocked for more than 120 seconds 2009-10-01 21:17 ` Shirley Ma 2009-10-01 22:09 ` Shirley Ma @ 2009-10-01 22:09 ` Shirley Ma 1 sibling, 0 replies; 22+ messages in thread From: Shirley Ma @ 2009-10-01 22:09 UTC (permalink / raw) To: Javier Guerra; +Cc: mcao, linux-kernel, kvm, virtualization Switching to different scheduler doesn't make the problem gone away. Shirley ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: INFO: task kjournal:337 blocked for more than 120 seconds 2009-10-01 21:03 ` Javier Guerra 2009-10-01 21:17 ` Shirley Ma @ 2009-10-01 21:17 ` Shirley Ma 1 sibling, 0 replies; 22+ messages in thread From: Shirley Ma @ 2009-10-01 21:17 UTC (permalink / raw) To: Javier Guerra; +Cc: mcao, linux-kernel, kvm, virtualization On Thu, 2009-10-01 at 16:03 -0500, Javier Guerra wrote: > deadline is the most recommended one for virtualization hosts. some > distros set it as default if you select Xen or KVM at installation > time. (and noop for the guests) I spoke too earlier, after a while noop scheduler hit the same issue. I am switching to deadline to test it again. Thanks Shirley ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: INFO: task kjournal:337 blocked for more than 120 seconds 2009-10-01 21:00 ` Shirley Ma 2009-10-01 21:03 ` Javier Guerra @ 2009-10-01 21:03 ` Javier Guerra 1 sibling, 0 replies; 22+ messages in thread From: Javier Guerra @ 2009-10-01 21:03 UTC (permalink / raw) To: Shirley Ma; +Cc: mcao, linux-kernel, kvm, virtualization On Thu, Oct 1, 2009 at 4:00 PM, Shirley Ma <mashirle@us.ibm.com> wrote: > I talked to Mingming, she suggested to use different IO scheduler. The > default scheduler is cfg, after I switch to noop, the problem is gone. deadline is the most recommended one for virtualization hosts. some distros set it as default if you select Xen or KVM at installation time. (and noop for the guests) -- Javier ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: INFO: task journal:337 blocked for more than 120 seconds 2009-09-30 21:11 INFO: task journal:337 blocked for more than 120 seconds Shirley Ma 2009-10-01 13:20 ` Marcelo Tosatti @ 2009-10-01 13:20 ` Marcelo Tosatti 2009-10-02 18:30 ` Jeremy Fitzhardinge 2009-10-02 18:30 ` Jeremy Fitzhardinge 3 siblings, 0 replies; 22+ messages in thread From: Marcelo Tosatti @ 2009-10-01 13:20 UTC (permalink / raw) To: Shirley Ma; +Cc: linux-kernel, kvm, virtualization On Wed, Sep 30, 2009 at 02:11:52PM -0700, Shirley Ma wrote: > Hello all, > > Anybody found this problem before? I kept hitting this issue for 2.6.31 > guest kernel even with a simple network test. > > INFO: task kjournal:337 blocked for more than 120 seconds. > "echo 0 > /proc/sys/kernel/hung_task_timeout_sec" disables this message. > > kjournald D 00000041 0 337 2 0x00000000 > > My test is totally being blocked. I've hit this in the past with ext3, mounting with data=writeback made it disappear. ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: INFO: task journal:337 blocked for more than 120 seconds 2009-09-30 21:11 INFO: task journal:337 blocked for more than 120 seconds Shirley Ma 2009-10-01 13:20 ` Marcelo Tosatti 2009-10-01 13:20 ` INFO: task journal:337 " Marcelo Tosatti @ 2009-10-02 18:30 ` Jeremy Fitzhardinge 2009-10-02 19:06 ` Shirley Ma ` (3 more replies) 2009-10-02 18:30 ` Jeremy Fitzhardinge 3 siblings, 4 replies; 22+ messages in thread From: Jeremy Fitzhardinge @ 2009-10-02 18:30 UTC (permalink / raw) To: Shirley Ma Cc: virtualization, kvm, linux-kernel, Jens Axboe, Kevin Bowling, Andrew Jones On 09/30/09 14:11, Shirley Ma wrote: > Anybody found this problem before? I kept hitting this issue for 2.6.31 > guest kernel even with a simple network test. > > INFO: task kjournal:337 blocked for more than 120 seconds. > "echo 0 > /proc/sys/kernel/hung_task_timeout_sec" disables this message. > > kjournald D 00000041 0 337 2 0x00000000 > > My test is totally being blocked. I'm assuming from the lists you've posted to that this is under KVM? What disk drivers are you using (virtio or emulated)? Can you get a full stack backtrace of kjournald? Kevin Bowling submitted a RH bug against Xen with apparently the same symptoms (https://bugzilla.redhat.com/show_bug.cgi?id=526627). I'm wondering if there's a core kernel bug here, which is perhaps more easily triggered by the changed timing in a virtual machine. Thanks, J ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: INFO: task journal:337 blocked for more than 120 seconds 2009-10-02 18:30 ` Jeremy Fitzhardinge @ 2009-10-02 19:06 ` Shirley Ma 2009-10-02 19:13 ` Jeremy Fitzhardinge 2009-10-02 19:13 ` Jeremy Fitzhardinge 2009-10-02 19:06 ` Shirley Ma ` (2 subsequent siblings) 3 siblings, 2 replies; 22+ messages in thread From: Shirley Ma @ 2009-10-02 19:06 UTC (permalink / raw) To: Jeremy Fitzhardinge Cc: virtualization, kvm, linux-kernel, Jens Axboe, Kevin Bowling, Andrew Jones [-- Attachment #1: Type: text/plain, Size: 404 bytes --] On Fri, 2009-10-02 at 11:30 -0700, Jeremy Fitzhardinge wrote: > I'm assuming from the lists you've posted to that this is under KVM? > What disk drivers are you using (virtio or emulated)? > > Can you get a full stack backtrace of kjournald? Yes, it's under KVM, disk driver is virtio. Since the io has issue, the stack can't be saved on the disk. I have the image file attached here. Thanks Shirley [-- Attachment #2: $696984E9BF50266.jpg --] [-- Type: image/jpeg, Size: 77669 bytes --] [-- Attachment #3: ext3.jpg --] [-- Type: image/jpeg, Size: 60064 bytes --] ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: INFO: task journal:337 blocked for more than 120 seconds 2009-10-02 19:06 ` Shirley Ma @ 2009-10-02 19:13 ` Jeremy Fitzhardinge 2009-10-02 19:13 ` Jeremy Fitzhardinge 1 sibling, 0 replies; 22+ messages in thread From: Jeremy Fitzhardinge @ 2009-10-02 19:13 UTC (permalink / raw) To: Shirley Ma Cc: virtualization, kvm, linux-kernel, Jens Axboe, Kevin Bowling, Andrew Jones On 10/02/09 12:06, Shirley Ma wrote: > On Fri, 2009-10-02 at 11:30 -0700, Jeremy Fitzhardinge wrote: > >> I'm assuming from the lists you've posted to that this is under KVM? >> What disk drivers are you using (virtio or emulated)? >> >> Can you get a full stack backtrace of kjournald? >> > Yes, it's under KVM, disk driver is virtio. Since the io has issue, the > stack can't be saved on the disk. I have the image file attached here. > Ah, thank you. The backtrace does indeed look very similar. (BTW, you could get a serial console with "qemu-kvm -nographic -append console=ttyS0 ...") J ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: INFO: task journal:337 blocked for more than 120 seconds 2009-10-02 19:06 ` Shirley Ma 2009-10-02 19:13 ` Jeremy Fitzhardinge @ 2009-10-02 19:13 ` Jeremy Fitzhardinge 1 sibling, 0 replies; 22+ messages in thread From: Jeremy Fitzhardinge @ 2009-10-02 19:13 UTC (permalink / raw) To: Shirley Ma Cc: Andrew Jones, kvm, Jens Axboe, linux-kernel, virtualization, Kevin Bowling On 10/02/09 12:06, Shirley Ma wrote: > On Fri, 2009-10-02 at 11:30 -0700, Jeremy Fitzhardinge wrote: > >> I'm assuming from the lists you've posted to that this is under KVM? >> What disk drivers are you using (virtio or emulated)? >> >> Can you get a full stack backtrace of kjournald? >> > Yes, it's under KVM, disk driver is virtio. Since the io has issue, the > stack can't be saved on the disk. I have the image file attached here. > Ah, thank you. The backtrace does indeed look very similar. (BTW, you could get a serial console with "qemu-kvm -nographic -append console=ttyS0 ...") J ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: INFO: task journal:337 blocked for more than 120 seconds 2009-10-02 18:30 ` Jeremy Fitzhardinge 2009-10-02 19:06 ` Shirley Ma @ 2009-10-02 19:06 ` Shirley Ma 2009-10-04 4:12 ` Kevin Bowling 2009-10-04 4:12 ` Kevin Bowling 3 siblings, 0 replies; 22+ messages in thread From: Shirley Ma @ 2009-10-02 19:06 UTC (permalink / raw) To: Jeremy Fitzhardinge Cc: Andrew Jones, kvm, Jens Axboe, linux-kernel, virtualization, Kevin Bowling [-- Attachment #1: Type: text/plain, Size: 404 bytes --] On Fri, 2009-10-02 at 11:30 -0700, Jeremy Fitzhardinge wrote: > I'm assuming from the lists you've posted to that this is under KVM? > What disk drivers are you using (virtio or emulated)? > > Can you get a full stack backtrace of kjournald? Yes, it's under KVM, disk driver is virtio. Since the io has issue, the stack can't be saved on the disk. I have the image file attached here. Thanks Shirley [-- Attachment #2: $696984E9BF50266.jpg --] [-- Type: image/jpeg, Size: 77669 bytes --] [-- Attachment #3: ext3.jpg --] [-- Type: image/jpeg, Size: 60064 bytes --] [-- Attachment #4: Type: text/plain, Size: 184 bytes --] _______________________________________________ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linux-foundation.org/mailman/listinfo/virtualization ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: INFO: task journal:337 blocked for more than 120 seconds 2009-10-02 18:30 ` Jeremy Fitzhardinge 2009-10-02 19:06 ` Shirley Ma 2009-10-02 19:06 ` Shirley Ma @ 2009-10-04 4:12 ` Kevin Bowling 2009-10-04 4:12 ` Kevin Bowling 3 siblings, 0 replies; 22+ messages in thread From: Kevin Bowling @ 2009-10-04 4:12 UTC (permalink / raw) To: Jeremy Fitzhardinge Cc: Andrew Jones, kvm, Shirley Ma, linux-kernel, virtualization, Jens Axboe On 10/2/2009 2:30 PM, Jeremy Fitzhardinge wrote: > On 09/30/09 14:11, Shirley Ma wrote: > >> Anybody found this problem before? I kept hitting this issue for 2.6.31 >> guest kernel even with a simple network test. >> >> INFO: task kjournal:337 blocked for more than 120 seconds. >> "echo 0> /proc/sys/kernel/hung_task_timeout_sec" disables this message. >> >> kjournald D 00000041 0 337 2 0x00000000 >> >> My test is totally being blocked. >> > I'm assuming from the lists you've posted to that this is under KVM? > What disk drivers are you using (virtio or emulated)? > > Can you get a full stack backtrace of kjournald? > > Kevin Bowling submitted a RH bug against Xen with apparently the same > symptoms (https://bugzilla.redhat.com/show_bug.cgi?id=526627). I'm > wondering if there's a core kernel bug here, which is perhaps more > easily triggered by the changed timing in a virtual machine. > > Thanks, > J > I've had a stable system thus far by appending "clocksource=jiffies" to the kernel boot line. The default clocksource is otherwise "xen". The dmesg boot warnings in my bugzilla report still occur. Regards, Kevin Bowling http://www.analograils.com/ ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: INFO: task journal:337 blocked for more than 120 seconds 2009-10-02 18:30 ` Jeremy Fitzhardinge ` (2 preceding siblings ...) 2009-10-04 4:12 ` Kevin Bowling @ 2009-10-04 4:12 ` Kevin Bowling 3 siblings, 0 replies; 22+ messages in thread From: Kevin Bowling @ 2009-10-04 4:12 UTC (permalink / raw) To: Jeremy Fitzhardinge Cc: Shirley Ma, virtualization, kvm, linux-kernel, Jens Axboe, Andrew Jones On 10/2/2009 2:30 PM, Jeremy Fitzhardinge wrote: > On 09/30/09 14:11, Shirley Ma wrote: > >> Anybody found this problem before? I kept hitting this issue for 2.6.31 >> guest kernel even with a simple network test. >> >> INFO: task kjournal:337 blocked for more than 120 seconds. >> "echo 0> /proc/sys/kernel/hung_task_timeout_sec" disables this message. >> >> kjournald D 00000041 0 337 2 0x00000000 >> >> My test is totally being blocked. >> > I'm assuming from the lists you've posted to that this is under KVM? > What disk drivers are you using (virtio or emulated)? > > Can you get a full stack backtrace of kjournald? > > Kevin Bowling submitted a RH bug against Xen with apparently the same > symptoms (https://bugzilla.redhat.com/show_bug.cgi?id=526627). I'm > wondering if there's a core kernel bug here, which is perhaps more > easily triggered by the changed timing in a virtual machine. > > Thanks, > J > I've had a stable system thus far by appending "clocksource=jiffies" to the kernel boot line. The default clocksource is otherwise "xen". The dmesg boot warnings in my bugzilla report still occur. Regards, Kevin Bowling http://www.analograils.com/ ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: INFO: task journal:337 blocked for more than 120 seconds 2009-09-30 21:11 INFO: task journal:337 blocked for more than 120 seconds Shirley Ma ` (2 preceding siblings ...) 2009-10-02 18:30 ` Jeremy Fitzhardinge @ 2009-10-02 18:30 ` Jeremy Fitzhardinge 3 siblings, 0 replies; 22+ messages in thread From: Jeremy Fitzhardinge @ 2009-10-02 18:30 UTC (permalink / raw) To: Shirley Ma Cc: Andrew Jones, kvm, Jens Axboe, linux-kernel, virtualization, Kevin Bowling On 09/30/09 14:11, Shirley Ma wrote: > Anybody found this problem before? I kept hitting this issue for 2.6.31 > guest kernel even with a simple network test. > > INFO: task kjournal:337 blocked for more than 120 seconds. > "echo 0 > /proc/sys/kernel/hung_task_timeout_sec" disables this message. > > kjournald D 00000041 0 337 2 0x00000000 > > My test is totally being blocked. I'm assuming from the lists you've posted to that this is under KVM? What disk drivers are you using (virtio or emulated)? Can you get a full stack backtrace of kjournald? Kevin Bowling submitted a RH bug against Xen with apparently the same symptoms (https://bugzilla.redhat.com/show_bug.cgi?id=526627). I'm wondering if there's a core kernel bug here, which is perhaps more easily triggered by the changed timing in a virtual machine. Thanks, J ^ permalink raw reply [flat|nested] 22+ messages in thread
* INFO: task journal:337 blocked for more than 120 seconds @ 2009-09-30 21:11 Shirley Ma 0 siblings, 0 replies; 22+ messages in thread From: Shirley Ma @ 2009-09-30 21:11 UTC (permalink / raw) To: virtualization, kvm, linux-kernel Hello all, Anybody found this problem before? I kept hitting this issue for 2.6.31 guest kernel even with a simple network test. INFO: task kjournal:337 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_sec" disables this message. kjournald D 00000041 0 337 2 0x00000000 My test is totally being blocked. Thanks Shirley ^ permalink raw reply [flat|nested] 22+ messages in thread
end of thread, other threads:[~2009-10-04 4:24 UTC | newest] Thread overview: 22+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2009-09-30 21:11 INFO: task journal:337 blocked for more than 120 seconds Shirley Ma 2009-10-01 13:20 ` Marcelo Tosatti 2009-10-01 16:58 ` Shirley Ma 2009-10-01 16:58 ` Shirley Ma 2009-10-01 21:00 ` INFO: task kjournal:337 " Shirley Ma 2009-10-01 21:00 ` Shirley Ma 2009-10-01 21:03 ` Javier Guerra 2009-10-01 21:17 ` Shirley Ma 2009-10-01 22:09 ` Shirley Ma 2009-10-01 22:09 ` Shirley Ma 2009-10-01 21:17 ` Shirley Ma 2009-10-01 21:03 ` Javier Guerra 2009-10-01 13:20 ` INFO: task journal:337 " Marcelo Tosatti 2009-10-02 18:30 ` Jeremy Fitzhardinge 2009-10-02 19:06 ` Shirley Ma 2009-10-02 19:13 ` Jeremy Fitzhardinge 2009-10-02 19:13 ` Jeremy Fitzhardinge 2009-10-02 19:06 ` Shirley Ma 2009-10-04 4:12 ` Kevin Bowling 2009-10-04 4:12 ` Kevin Bowling 2009-10-02 18:30 ` Jeremy Fitzhardinge -- strict thread matches above, loose matches on Subject: below -- 2009-09-30 21:11 Shirley Ma
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.
![[-- Attachment #3: ext3.jpg --] [-- Type: image/jpeg, Size: 60064 bytes --]](../../1254510407.7999.3.camel@localhost.localdomain/3-ext3.jpg){kind=link}
![[-- Attachment #3: ext3.jpg --] [-- Type: image/jpeg, Size: 60064 bytes --]](../../1254510407.7999.3.camel__27084.139842964$1254510551$gmane$org@localhost.localdomain/3-ext3.jpg){kind=link}