* Re: [uml-devel] WARNING: at mm/mmap.c:2757 exit_mmap+0x161/0x170() [not found] ` <CAFLxGvxhDu6jVHZ0iOhjSBEfEY5JZRJ9xw0gxcduvuTR6BGOPg@mail.gmail.com> @ 2013-05-15 19:30 ` Toralf Förster 2013-05-15 19:35 ` richard -rw- weinberger 0 siblings, 1 reply; 6+ messages in thread From: Toralf Förster @ 2013-05-15 19:30 UTC (permalink / raw) To: richard -rw- weinberger Cc: user-mode-linux-devel@lists.sourceforge.net, trinity On 05/15/2013 09:11 PM, richard -rw- weinberger wrote: > On Wed, May 15, 2013 at 9:06 PM, Toralf Förster <toralf.foerster@gmx.de> wrote: >> On 05/13/2013 09:12 AM, richard -rw- weinberger wrote: >>> This looks like another issue. >>> Are you testing process_vm_writev() with trinity? >>> Looks like it managed to overwrite the stub page of a process, which >>> is not good. >> nope, it is the mremap syscall. >> >> A command like >> >> $>trinity -c mremap -N 10 >> >> immediately after starting a 32 bit Gentoo linux guest with current kernel 3.10-rc1-... + >> strnlen + stub4 patch works, but later a >> >> $>trinity -c mremap -N 1000 >> >> yields into >> >> 2013-05-15T21:02:04.061+02:00 trinity kernel: Stub registers - >> 2013-05-15T21:02:04.061+02:00 trinity kernel: 0 - 100000 >> 2013-05-15T21:02:04.061+02:00 trinity kernel: 1 - 300000 >> 2013-05-15T21:02:04.061+02:00 trinity kernel: 2 - 0 >> 2013-05-15T21:02:04.061+02:00 trinity kernel: 3 - 0 >> 2013-05-15T21:02:04.061+02:00 trinity kernel: 4 - 0 >> 2013-05-15T21:02:04.061+02:00 trinity kernel: 5 - 0 >> 2013-05-15T21:02:04.061+02:00 trinity kernel: 6 - 0 >> 2013-05-15T21:02:04.061+02:00 trinity kernel: 7 - 7b >> 2013-05-15T21:02:04.061+02:00 trinity kernel: 8 - 7b >> 2013-05-15T21:02:04.065+02:00 trinity kernel: 9 - 0 >> 2013-05-15T21:02:04.065+02:00 trinity kernel: 10 - 33 >> 2013-05-15T21:02:04.065+02:00 trinity kernel: 11 - ffffffff >> 2013-05-15T21:02:04.065+02:00 trinity kernel: 12 - 1000c3 >> 2013-05-15T21:02:04.065+02:00 trinity kernel: 13 - 73 >> 2013-05-15T21:02:04.065+02:00 trinity kernel: 14 - 10206 >> 2013-05-15T21:02:04.065+02:00 trinity kernel: 15 - 101028 >> 2013-05-15T21:02:04.065+02:00 trinity kernel: 16 - 7b >> 2013-05-15T21:02:04.065+02:00 trinity kernel: wait_stub_done : failed to wait for SIGTRAP, pid = 15692, n = 15692, errno = 0, status = 0xb7f >> >> and now that process can't be killed - I had to stop the UML guest. > > Hmm, you've remapped the stub page and therefore the process broke. > I think it would make sense to kill the process in stead of writing > the "wait_stub_done ..." message. > Changing the stub page is as destructive than overwriting the stack. Unfortunately no trinity process can be killed as soon as that happen. Neither pgrep, pkill, nor "ps -efla" do return any result. Killing any of those processes by its pid won't work too. > Maybe we can teach triniy to no change the stub page. > I'm sure trinity has also a mechanism to not destroy the stack. @trinity Mailing list What do you think about that ? -- MfG/Sincerely Toralf Förster pgp finger print: 7B1A 07F4 EC82 0F90 D4C2 8936 872A E508 7DB6 9DA3 ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [uml-devel] WARNING: at mm/mmap.c:2757 exit_mmap+0x161/0x170() 2013-05-15 19:30 ` [uml-devel] WARNING: at mm/mmap.c:2757 exit_mmap+0x161/0x170() Toralf Förster @ 2013-05-15 19:35 ` richard -rw- weinberger 2013-05-17 10:00 ` richard -rw- weinberger 0 siblings, 1 reply; 6+ messages in thread From: richard -rw- weinberger @ 2013-05-15 19:35 UTC (permalink / raw) To: Toralf Förster; +Cc: user-mode-linux-devel@lists.sourceforge.net, trinity On Wed, May 15, 2013 at 9:30 PM, Toralf Förster <toralf.foerster@gmx.de> wrote: > On 05/15/2013 09:11 PM, richard -rw- weinberger wrote: >> On Wed, May 15, 2013 at 9:06 PM, Toralf Förster <toralf.foerster@gmx.de> wrote: >>> On 05/13/2013 09:12 AM, richard -rw- weinberger wrote: >>>> This looks like another issue. >>>> Are you testing process_vm_writev() with trinity? >>>> Looks like it managed to overwrite the stub page of a process, which >>>> is not good. >>> nope, it is the mremap syscall. >>> >>> A command like >>> >>> $>trinity -c mremap -N 10 >>> >>> immediately after starting a 32 bit Gentoo linux guest with current kernel 3.10-rc1-... + >>> strnlen + stub4 patch works, but later a >>> >>> $>trinity -c mremap -N 1000 >>> >>> yields into >>> >>> 2013-05-15T21:02:04.061+02:00 trinity kernel: Stub registers - >>> 2013-05-15T21:02:04.061+02:00 trinity kernel: 0 - 100000 >>> 2013-05-15T21:02:04.061+02:00 trinity kernel: 1 - 300000 >>> 2013-05-15T21:02:04.061+02:00 trinity kernel: 2 - 0 >>> 2013-05-15T21:02:04.061+02:00 trinity kernel: 3 - 0 >>> 2013-05-15T21:02:04.061+02:00 trinity kernel: 4 - 0 >>> 2013-05-15T21:02:04.061+02:00 trinity kernel: 5 - 0 >>> 2013-05-15T21:02:04.061+02:00 trinity kernel: 6 - 0 >>> 2013-05-15T21:02:04.061+02:00 trinity kernel: 7 - 7b >>> 2013-05-15T21:02:04.061+02:00 trinity kernel: 8 - 7b >>> 2013-05-15T21:02:04.065+02:00 trinity kernel: 9 - 0 >>> 2013-05-15T21:02:04.065+02:00 trinity kernel: 10 - 33 >>> 2013-05-15T21:02:04.065+02:00 trinity kernel: 11 - ffffffff >>> 2013-05-15T21:02:04.065+02:00 trinity kernel: 12 - 1000c3 >>> 2013-05-15T21:02:04.065+02:00 trinity kernel: 13 - 73 >>> 2013-05-15T21:02:04.065+02:00 trinity kernel: 14 - 10206 >>> 2013-05-15T21:02:04.065+02:00 trinity kernel: 15 - 101028 >>> 2013-05-15T21:02:04.065+02:00 trinity kernel: 16 - 7b >>> 2013-05-15T21:02:04.065+02:00 trinity kernel: wait_stub_done : failed to wait for SIGTRAP, pid = 15692, n = 15692, errno = 0, status = 0xb7f >>> >>> and now that process can't be killed - I had to stop the UML guest. >> >> Hmm, you've remapped the stub page and therefore the process broke. >> I think it would make sense to kill the process in stead of writing >> the "wait_stub_done ..." message. >> Changing the stub page is as destructive than overwriting the stack. > > Unfortunately no trinity process can be killed as soon as that happen. > Neither pgrep, pkill, nor "ps -efla" do return any result. > Killing any of those processes by its pid won't work too. Hmm, not good. I need to create me a reproducer for that. I'm unsure what exactly is going on. -- Thanks, //richard ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [uml-devel] WARNING: at mm/mmap.c:2757 exit_mmap+0x161/0x170() 2013-05-15 19:35 ` richard -rw- weinberger @ 2013-05-17 10:00 ` richard -rw- weinberger 2013-05-17 12:22 ` richard -rw- weinberger 0 siblings, 1 reply; 6+ messages in thread From: richard -rw- weinberger @ 2013-05-17 10:00 UTC (permalink / raw) To: Toralf Förster; +Cc: user-mode-linux-devel@lists.sourceforge.net, trinity On Wed, May 15, 2013 at 9:35 PM, richard -rw- weinberger <richard.weinberger@gmail.com> wrote: > On Wed, May 15, 2013 at 9:30 PM, Toralf Förster <toralf.foerster@gmx.de> wrote: >> On 05/15/2013 09:11 PM, richard -rw- weinberger wrote: >>> On Wed, May 15, 2013 at 9:06 PM, Toralf Förster <toralf.foerster@gmx.de> wrote: >>>> On 05/13/2013 09:12 AM, richard -rw- weinberger wrote: >>>>> This looks like another issue. >>>>> Are you testing process_vm_writev() with trinity? >>>>> Looks like it managed to overwrite the stub page of a process, which >>>>> is not good. >>>> nope, it is the mremap syscall. >>>> >>>> A command like >>>> >>>> $>trinity -c mremap -N 10 >>>> >>>> immediately after starting a 32 bit Gentoo linux guest with current kernel 3.10-rc1-... + >>>> strnlen + stub4 patch works, but later a >>>> >>>> $>trinity -c mremap -N 1000 >>>> >>>> yields into >>>> >>>> 2013-05-15T21:02:04.061+02:00 trinity kernel: Stub registers - >>>> 2013-05-15T21:02:04.061+02:00 trinity kernel: 0 - 100000 >>>> 2013-05-15T21:02:04.061+02:00 trinity kernel: 1 - 300000 >>>> 2013-05-15T21:02:04.061+02:00 trinity kernel: 2 - 0 >>>> 2013-05-15T21:02:04.061+02:00 trinity kernel: 3 - 0 >>>> 2013-05-15T21:02:04.061+02:00 trinity kernel: 4 - 0 >>>> 2013-05-15T21:02:04.061+02:00 trinity kernel: 5 - 0 >>>> 2013-05-15T21:02:04.061+02:00 trinity kernel: 6 - 0 >>>> 2013-05-15T21:02:04.061+02:00 trinity kernel: 7 - 7b >>>> 2013-05-15T21:02:04.061+02:00 trinity kernel: 8 - 7b >>>> 2013-05-15T21:02:04.065+02:00 trinity kernel: 9 - 0 >>>> 2013-05-15T21:02:04.065+02:00 trinity kernel: 10 - 33 >>>> 2013-05-15T21:02:04.065+02:00 trinity kernel: 11 - ffffffff >>>> 2013-05-15T21:02:04.065+02:00 trinity kernel: 12 - 1000c3 >>>> 2013-05-15T21:02:04.065+02:00 trinity kernel: 13 - 73 >>>> 2013-05-15T21:02:04.065+02:00 trinity kernel: 14 - 10206 >>>> 2013-05-15T21:02:04.065+02:00 trinity kernel: 15 - 101028 >>>> 2013-05-15T21:02:04.065+02:00 trinity kernel: 16 - 7b >>>> 2013-05-15T21:02:04.065+02:00 trinity kernel: wait_stub_done : failed to wait for SIGTRAP, pid = 15692, n = 15692, errno = 0, status = 0xb7f >>>> >>>> and now that process can't be killed - I had to stop the UML guest. >>> >>> Hmm, you've remapped the stub page and therefore the process broke. >>> I think it would make sense to kill the process in stead of writing >>> the "wait_stub_done ..." message. >>> Changing the stub page is as destructive than overwriting the stack. >> >> Unfortunately no trinity process can be killed as soon as that happen. >> Neither pgrep, pkill, nor "ps -efla" do return any result. >> Killing any of those processes by its pid won't work too. > > Hmm, not good. > I need to create me a reproducer for that. > I'm unsure what exactly is going on. Good news, I have a reproducer for the problem and found out what the root cause is. UML is unable to terminate the task with the broken skas page. A fix is on the way... -- Thanks, //richard ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [uml-devel] WARNING: at mm/mmap.c:2757 exit_mmap+0x161/0x170() 2013-05-17 10:00 ` richard -rw- weinberger @ 2013-05-17 12:22 ` richard -rw- weinberger 2013-05-17 14:28 ` Toralf Förster 0 siblings, 1 reply; 6+ messages in thread From: richard -rw- weinberger @ 2013-05-17 12:22 UTC (permalink / raw) To: Toralf Förster; +Cc: user-mode-linux-devel@lists.sourceforge.net, trinity [-- Attachment #1: Type: text/plain, Size: 3321 bytes --] On Fri, May 17, 2013 at 12:00 PM, richard -rw- weinberger <richard.weinberger@gmail.com> wrote: > On Wed, May 15, 2013 at 9:35 PM, richard -rw- weinberger > <richard.weinberger@gmail.com> wrote: >> On Wed, May 15, 2013 at 9:30 PM, Toralf Förster <toralf.foerster@gmx.de> wrote: >>> On 05/15/2013 09:11 PM, richard -rw- weinberger wrote: >>>> On Wed, May 15, 2013 at 9:06 PM, Toralf Förster <toralf.foerster@gmx.de> wrote: >>>>> On 05/13/2013 09:12 AM, richard -rw- weinberger wrote: >>>>>> This looks like another issue. >>>>>> Are you testing process_vm_writev() with trinity? >>>>>> Looks like it managed to overwrite the stub page of a process, which >>>>>> is not good. >>>>> nope, it is the mremap syscall. >>>>> >>>>> A command like >>>>> >>>>> $>trinity -c mremap -N 10 >>>>> >>>>> immediately after starting a 32 bit Gentoo linux guest with current kernel 3.10-rc1-... + >>>>> strnlen + stub4 patch works, but later a >>>>> >>>>> $>trinity -c mremap -N 1000 >>>>> >>>>> yields into >>>>> >>>>> 2013-05-15T21:02:04.061+02:00 trinity kernel: Stub registers - >>>>> 2013-05-15T21:02:04.061+02:00 trinity kernel: 0 - 100000 >>>>> 2013-05-15T21:02:04.061+02:00 trinity kernel: 1 - 300000 >>>>> 2013-05-15T21:02:04.061+02:00 trinity kernel: 2 - 0 >>>>> 2013-05-15T21:02:04.061+02:00 trinity kernel: 3 - 0 >>>>> 2013-05-15T21:02:04.061+02:00 trinity kernel: 4 - 0 >>>>> 2013-05-15T21:02:04.061+02:00 trinity kernel: 5 - 0 >>>>> 2013-05-15T21:02:04.061+02:00 trinity kernel: 6 - 0 >>>>> 2013-05-15T21:02:04.061+02:00 trinity kernel: 7 - 7b >>>>> 2013-05-15T21:02:04.061+02:00 trinity kernel: 8 - 7b >>>>> 2013-05-15T21:02:04.065+02:00 trinity kernel: 9 - 0 >>>>> 2013-05-15T21:02:04.065+02:00 trinity kernel: 10 - 33 >>>>> 2013-05-15T21:02:04.065+02:00 trinity kernel: 11 - ffffffff >>>>> 2013-05-15T21:02:04.065+02:00 trinity kernel: 12 - 1000c3 >>>>> 2013-05-15T21:02:04.065+02:00 trinity kernel: 13 - 73 >>>>> 2013-05-15T21:02:04.065+02:00 trinity kernel: 14 - 10206 >>>>> 2013-05-15T21:02:04.065+02:00 trinity kernel: 15 - 101028 >>>>> 2013-05-15T21:02:04.065+02:00 trinity kernel: 16 - 7b >>>>> 2013-05-15T21:02:04.065+02:00 trinity kernel: wait_stub_done : failed to wait for SIGTRAP, pid = 15692, n = 15692, errno = 0, status = 0xb7f >>>>> >>>>> and now that process can't be killed - I had to stop the UML guest. >>>> >>>> Hmm, you've remapped the stub page and therefore the process broke. >>>> I think it would make sense to kill the process in stead of writing >>>> the "wait_stub_done ..." message. >>>> Changing the stub page is as destructive than overwriting the stack. >>> >>> Unfortunately no trinity process can be killed as soon as that happen. >>> Neither pgrep, pkill, nor "ps -efla" do return any result. >>> Killing any of those processes by its pid won't work too. >> >> Hmm, not good. >> I need to create me a reproducer for that. >> I'm unsure what exactly is going on. > > Good news, I have a reproducer for the problem and found out what the > root cause is. > UML is unable to terminate the task with the broken skas page. > A fix is on the way... Toralf, can you please append the attached patch too? It makes processes killable which corrupted their stub pages. -- Thanks, //richard [-- Attachment #2: wait_stub_done_fix.diff --] [-- Type: application/octet-stream, Size: 922 bytes --] diff --git a/arch/um/os-Linux/skas/process.c b/arch/um/os-Linux/skas/process.c index 4625949..441e4ba 100644 --- a/arch/um/os-Linux/skas/process.c +++ b/arch/um/os-Linux/skas/process.c @@ -54,7 +54,7 @@ static int ptrace_dump_regs(int pid) void wait_stub_done(int pid) { - int n, status, err; + int n, status, err, bad_stop = 0; while (1) { CATCH_EINTR(n = waitpid(pid, &status, WUNTRACED | __WALL)); @@ -74,6 +74,8 @@ void wait_stub_done(int pid) if (((1 << WSTOPSIG(status)) & STUB_DONE_MASK) != 0) return; + else + bad_stop = 1; bad_wait: err = ptrace_dump_regs(pid); @@ -83,7 +85,10 @@ bad_wait: printk(UM_KERN_ERR "wait_stub_done : failed to wait for SIGTRAP, " "pid = %d, n = %d, errno = %d, status = 0x%x\n", pid, n, errno, status); - fatal_sigsegv(); + if (bad_stop) + kill(pid, SIGKILL); + else + fatal_sigsegv(); } extern unsigned long current_stub_stack(void); ^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [uml-devel] WARNING: at mm/mmap.c:2757 exit_mmap+0x161/0x170() 2013-05-17 12:22 ` richard -rw- weinberger @ 2013-05-17 14:28 ` Toralf Förster 2013-05-17 14:50 ` Richard RW. Weinberger 0 siblings, 1 reply; 6+ messages in thread From: Toralf Förster @ 2013-05-17 14:28 UTC (permalink / raw) To: richard -rw- weinberger Cc: user-mode-linux-devel@lists.sourceforge.net, trinity On 05/17/2013 02:22 PM, richard -rw- weinberger wrote: > Toralf, can you please append the attached patch too? > It makes processes killable which corrupted their stub pages. yes - now I can run an trinity infinite with the syscall mremap(). I still get for every x-th the syslog seen below, but processes are now killable (and trinity itself can kill its childs too so far) 2013-05-17T16:27:30.234+02:00 trinity kernel: do_syscall_stub: syscall 91 failed, return value = 0x0, expected return value = 0x0 2013-05-17T16:27:30.234+02:00 trinity kernel: syscall parameters: 0x100000 0x300000 0x0 0x0 0x0 0x0 2013-05-17T16:27:30.441+02:00 trinity kernel: Stub registers - 2013-05-17T16:27:30.441+02:00 trinity kernel: 0 - 100000 2013-05-17T16:27:30.441+02:00 trinity kernel: 1 - 300000 2013-05-17T16:27:30.441+02:00 trinity kernel: 2 - 0 2013-05-17T16:27:30.441+02:00 trinity kernel: 3 - 0 2013-05-17T16:27:30.441+02:00 trinity kernel: 4 - 0 2013-05-17T16:27:30.441+02:00 trinity kernel: 5 - 0 2013-05-17T16:27:30.441+02:00 trinity kernel: 6 - 0 2013-05-17T16:27:30.441+02:00 trinity kernel: 7 - 7b 2013-05-17T16:27:30.441+02:00 trinity kernel: 8 - 7b 2013-05-17T16:27:30.442+02:00 trinity kernel: 9 - 0 2013-05-17T16:27:30.442+02:00 trinity kernel: 10 - 33 2013-05-17T16:27:30.442+02:00 trinity kernel: 11 - ffffffff 2013-05-17T16:27:30.442+02:00 trinity kernel: 12 - 1000c3 2013-05-17T16:27:30.442+02:00 trinity kernel: 13 - 73 2013-05-17T16:27:30.442+02:00 trinity kernel: 14 - 10206 2013-05-17T16:27:30.442+02:00 trinity kernel: 15 - 101028 2013-05-17T16:27:30.442+02:00 trinity kernel: 16 - 7b 2013-05-17T16:27:30.442+02:00 trinity kernel: wait_stub_done : failed to wait for SIGTRAP, pid = 31946, n = 31946, errno = 0, status = 0xb7f 2013-05-17T16:27:30.442+02:00 trinity kernel: do_syscall_stub : ret = 0, offset = 1052680, data = 41c36008 2013-05-17T16:27:30.443+02:00 trinity kernel: do_syscall_stub: syscall 91 failed, return value = 0x0, expected return value = 0x0 2013-05-17T16:27:30.443+02:00 trinity kernel: syscall parameters: 0x100000 0x300000 0x0 0x0 0x0 0x0 -- MfG/Sincerely Toralf Förster pgp finger print: 7B1A 07F4 EC82 0F90 D4C2 8936 872A E508 7DB6 9DA3 ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [uml-devel] WARNING: at mm/mmap.c:2757 exit_mmap+0x161/0x170() 2013-05-17 14:28 ` Toralf Förster @ 2013-05-17 14:50 ` Richard RW. Weinberger 0 siblings, 0 replies; 6+ messages in thread From: Richard RW. Weinberger @ 2013-05-17 14:50 UTC (permalink / raw) To: Toralf Förster Cc: user-mode-linux-devel, trinity, richard -rw- weinberger ----- Ursprüngliche Mail ----- > On 05/17/2013 02:22 PM, richard -rw- weinberger wrote: > > Toralf, can you please append the attached patch too? > > It makes processes killable which corrupted their stub pages. > yes - now I can run an trinity infinite with the syscall mremap(). > > I still get for every x-th the syslog seen below, but processes are > now killable (and trinity itself can kill its childs too so far) That's perfectly fine. Thanks, //richard ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2013-05-17 14:50 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <518FB97A.5070907@gmx.de>
[not found] ` <CAFLxGvzn+Q=D4vjTNuggSTnQKVp0QYLPoHmUwAVBcRbSmD3HWQ@mail.gmail.com>
[not found] ` <518FBE6A.50605@gmx.de>
[not found] ` <CAFLxGvyvRb+WD-eLBFnH5O-sXjUdtUkXbKzG6HFqgzm+x2LUow@mail.gmail.com>
[not found] ` <CAFLxGvwEiE0dfw9Vju_Xg6mXkF_NH_OU_t7puPceYN42dphTLA@mail.gmail.com>
[not found] ` <518FE33B.60701@gmx.de>
[not found] ` <CAFLxGvyiKbbkPAG0h42y1fb=Z6YuboJPdyb+2Xk9Z2YYt7oryg@mail.gmail.com>
[not found] ` <518FF354.7020408@gmx.de>
[not found] ` <CAFLxGvxYjpt6X9MQF3Huh-Sqzm3CBMJAfGb_Qjpqu00iUzj=qg@mail.gmail.com>
[not found] ` <518FFBA3.6000800@gmx.de>
[not found] ` <CAFLxGvx+8wpBeF_TqOk3JKBEoADAS-NeatpJMuSVvaDhZNqE0g@mail.gmail.com>
[not found] ` <CAFLxGvzSEY8iciEONkxNGeO6Lt5v4VcB2sF1XPcqLA=n8PUaAQ@mail.gmail.com>
[not found] ` <51901400.4060302@gmx.de>
[not found] ` <CAFLxGvx-saWcJ+JCubUBAVT70OSMZDYNETqiNijJgXeqCPwwUA@mail.gmail.com>
[not found] ` <5193DCA7.1070708@gmx.de>
[not found] ` <CAFLxGvxhDu6jVHZ0iOhjSBEfEY5JZRJ9xw0gxcduvuTR6BGOPg@mail. gmail.com>
[not found] ` <CAFLxGvxhDu6jVHZ0iOhjSBEfEY5JZRJ9xw0gxcduvuTR6BGOPg@mail.gmail.com>
2013-05-15 19:30 ` [uml-devel] WARNING: at mm/mmap.c:2757 exit_mmap+0x161/0x170() Toralf Förster
2013-05-15 19:35 ` richard -rw- weinberger
2013-05-17 10:00 ` richard -rw- weinberger
2013-05-17 12:22 ` richard -rw- weinberger
2013-05-17 14:28 ` Toralf Förster
2013-05-17 14:50 ` Richard RW. Weinberger
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox