* [PATCH v2 0/1] perf test: Add test for branch stack sampling
@ 2022-07-05 15:05 German Gomez
2022-07-05 15:05 ` [PATCH v2 1/1] " German Gomez
0 siblings, 1 reply; 7+ messages in thread
From: German Gomez @ 2022-07-05 15:05 UTC (permalink / raw)
To: linux-kernel, linux-perf-users, acme, irogers
Cc: German Gomez, Mark Rutland, Alexander Shishkin, Jiri Olsa,
Namhyung Kim
Hi,
While testing Anshuman Khandual's branch stack sampling ABI, and BRBE
driver ([1] and [2] respectively) I've been using this test to check the
consistency between arches (x86 and Arm64).
We will extend it with Arm64-specific tests once the BRBE driver has
been reviewed and merged, but for now I think this is generic enough to
also be useful for x86.
Thanks,
German
[1] https://lore.kernel.org/linux-arm-kernel/20220610035101.424112-1-anshuman.khandual@arm.com/
[2] https://lore.kernel.org/all/20220613100119.684673-1-anshuman.khandual@arm.com/
---
v2
- (Ian Rogers) Include C test program in the shell script.
- Drop references to Anshuman's ABI ext (FAULT_DATA and FAULT_INST).
We will track and update this later if necessary.
German Gomez (1):
perf test: Add test for branch stack sampling
tools/perf/tests/shell/test_brstack.sh | 114 +++++++++++++++++++++++++
1 file changed, 114 insertions(+)
create mode 100755 tools/perf/tests/shell/test_brstack.sh
--
2.25.1
^ permalink raw reply [flat|nested] 7+ messages in thread* [PATCH v2 1/1] perf test: Add test for branch stack sampling 2022-07-05 15:05 [PATCH v2 0/1] perf test: Add test for branch stack sampling German Gomez @ 2022-07-05 15:05 ` German Gomez 2022-07-08 20:31 ` Namhyung Kim 2022-07-11 20:56 ` Jiri Olsa 0 siblings, 2 replies; 7+ messages in thread From: German Gomez @ 2022-07-05 15:05 UTC (permalink / raw) To: linux-kernel, linux-perf-users, acme, irogers Cc: German Gomez, Anshuman Khandual, Mark Rutland, Alexander Shishkin, Jiri Olsa, Namhyung Kim Add a self test for branch stack sampling, to check that we get the expected branch types, and filters behave as expected. Suggested-by: Anshuman Khandual <anshuman.khandual@arm.com> Signed-off-by: German Gomez <german.gomez@arm.com> --- tools/perf/tests/shell/test_brstack.sh | 114 +++++++++++++++++++++++++ 1 file changed, 114 insertions(+) create mode 100755 tools/perf/tests/shell/test_brstack.sh diff --git a/tools/perf/tests/shell/test_brstack.sh b/tools/perf/tests/shell/test_brstack.sh new file mode 100755 index 000000000..5f6f40a77 --- /dev/null +++ b/tools/perf/tests/shell/test_brstack.sh @@ -0,0 +1,114 @@ +#!/bin/sh +# Check branch stack sampling + +# SPDX-License-Identifier: GPL-2.0 +# German Gomez <german.gomez@arm.com>, 2022 + +# we need a C compiler to build the test programs +# so bail if none is found +if ! [ -x "$(command -v cc)" ]; then + echo "failed: no compiler, install gcc" + exit 2 +fi + +# skip the test if the hardware doesn't support branch stack sampling +perf record -b -o- -e dummy -B true > /dev/null 2>&1 || exit 2 + +TMPDIR=$(mktemp -d /tmp/__perf_test.program.XXXXX) + +cleanup() { + rm -rf $TMPDIR +} + +trap cleanup exit term int + +gen_test_program() { + # generate test program + cat << EOF > $1 +#define BENCH_RUNS 999999 +int cnt; +void bar(void) { +} /* return */ +void foo(void) { + bar(); /* call */ +} /* return */ +void bench(void) { + void (*foo_ind)(void) = foo; + if ((cnt++) % 3) /* branch (cond) */ + foo(); /* call */ + bar(); /* call */ + foo_ind(); /* call (ind) */ +} +int main(void) +{ + int cnt = 0; + while (1) { + if ((cnt++) > BENCH_RUNS) + break; + bench(); /* call */ + } /* branch (uncond) */ + return 0; +} +EOF +} + +test_user_branches() { + echo "Testing user branch stack sampling" + + gen_test_program "$TEMPDIR/program.c" + cc -fno-inline -g "$TEMPDIR/program.c" -o $TMPDIR/a.out + + perf record -o $TMPDIR/perf.data --branch-filter any,save_type,u -- $TMPDIR/a.out > /dev/null 2>&1 + perf script -i $TMPDIR/perf.data --fields brstacksym | xargs -n1 > $TMPDIR/perf.script + + # example of branch entries: + # foo+0x14/bar+0x40/P/-/-/0/CALL + + set -x + egrep -m1 "^bench\+[^ ]*/foo\+[^ ]*/IND_CALL$" $TMPDIR/perf.script + egrep -m1 "^foo\+[^ ]*/bar\+[^ ]*/CALL$" $TMPDIR/perf.script + egrep -m1 "^bench\+[^ ]*/foo\+[^ ]*/CALL$" $TMPDIR/perf.script + egrep -m1 "^bench\+[^ ]*/bar\+[^ ]*/CALL$" $TMPDIR/perf.script + egrep -m1 "^bar\+[^ ]*/foo\+[^ ]*/RET$" $TMPDIR/perf.script + egrep -m1 "^foo\+[^ ]*/bench\+[^ ]*/RET$" $TMPDIR/perf.script + egrep -m1 "^bench\+[^ ]*/bench\+[^ ]*/COND$" $TMPDIR/perf.script + egrep -m1 "^main\+[^ ]*/main\+[^ ]*/UNCOND$" $TMPDIR/perf.script + set +x + + # some branch types are still not being tested: + # IND COND_CALL COND_RET SYSCALL SYSRET IRQ SERROR NO_TX +} + +# first argument <arg0> is the argument passed to "--branch-stack <arg0>,save_type,u" +# second argument are the expected branch types for the given filter +test_filter() { + local filter=$1 + local expect=$2 + + echo "Testing branch stack filtering permutation ($filter,$expect)" + + gen_test_program "$TEMPDIR/program.c" + cc -fno-inline -g "$TEMPDIR/program.c" -o $TMPDIR/a.out + + perf record -o $TMPDIR/perf.data --branch-filter $filter,save_type,u -- $TMPDIR/a.out > /dev/null 2>&1 + perf script -i $TMPDIR/perf.data --fields brstack | xargs -n1 > $TMPDIR/perf.script + + # fail if we find any branch type that doesn't match any of the expected ones + # also consider UNKNOWN branch types (-) + if egrep -vm1 "^[^ ]*/($expect|-|( *))$" $TMPDIR/perf.script; then + return 1 + fi +} + +set -e + +test_user_branches + +test_filter "any_call" "CALL|IND_CALL|COND_CALL|SYSCALL|IRQ" +test_filter "call" "CALL|SYSCALL" +test_filter "cond" "COND" +test_filter "any_ret" "RET|COND_RET|SYSRET|ERET" + +test_filter "call,cond" "CALL|SYSCALL|COND" +test_filter "any_call,cond" "CALL|IND_CALL|COND_CALL|IRQ|SYSCALL|COND" +test_filter "cond,any_call,any_ret" "COND|CALL|IND_CALL|COND_CALL|SYSCALL|IRQ|RET|COND_RET|SYSRET|ERET" -- 2.25.1 ^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [PATCH v2 1/1] perf test: Add test for branch stack sampling 2022-07-05 15:05 ` [PATCH v2 1/1] " German Gomez @ 2022-07-08 20:31 ` Namhyung Kim 2022-07-11 12:50 ` German Gomez 2022-07-12 11:56 ` Arnaldo Carvalho de Melo 2022-07-11 20:56 ` Jiri Olsa 1 sibling, 2 replies; 7+ messages in thread From: Namhyung Kim @ 2022-07-08 20:31 UTC (permalink / raw) To: German Gomez Cc: linux-kernel, linux-perf-users, Arnaldo Carvalho de Melo, Ian Rogers, Anshuman Khandual, Mark Rutland, Alexander Shishkin, Jiri Olsa Hello, On Tue, Jul 5, 2022 at 8:05 AM German Gomez <german.gomez@arm.com> wrote: > > Add a self test for branch stack sampling, to check that we get the > expected branch types, and filters behave as expected. > > Suggested-by: Anshuman Khandual <anshuman.khandual@arm.com> > Signed-off-by: German Gomez <german.gomez@arm.com> Acked-by: Namhyung Kim <namhyung@kernel.org> Thanks, Namhyung ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH v2 1/1] perf test: Add test for branch stack sampling 2022-07-08 20:31 ` Namhyung Kim @ 2022-07-11 12:50 ` German Gomez 2022-07-12 11:56 ` Arnaldo Carvalho de Melo 1 sibling, 0 replies; 7+ messages in thread From: German Gomez @ 2022-07-11 12:50 UTC (permalink / raw) To: Namhyung Kim Cc: linux-kernel, linux-perf-users, Arnaldo Carvalho de Melo, Ian Rogers, Anshuman Khandual, Mark Rutland, Alexander Shishkin, Jiri Olsa On 08/07/2022 21:31, Namhyung Kim wrote: > Hello, > > On Tue, Jul 5, 2022 at 8:05 AM German Gomez <german.gomez@arm.com> wrote: >> Add a self test for branch stack sampling, to check that we get the >> expected branch types, and filters behave as expected. >> >> Suggested-by: Anshuman Khandual <anshuman.khandual@arm.com> >> Signed-off-by: German Gomez <german.gomez@arm.com> > Acked-by: Namhyung Kim <namhyung@kernel.org> Thanks a lot Namhyung! > > Thanks, > Namhyung ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH v2 1/1] perf test: Add test for branch stack sampling 2022-07-08 20:31 ` Namhyung Kim 2022-07-11 12:50 ` German Gomez @ 2022-07-12 11:56 ` Arnaldo Carvalho de Melo 1 sibling, 0 replies; 7+ messages in thread From: Arnaldo Carvalho de Melo @ 2022-07-12 11:56 UTC (permalink / raw) To: Namhyung Kim Cc: German Gomez, linux-kernel, linux-perf-users, Ian Rogers, Anshuman Khandual, Mark Rutland, Alexander Shishkin, Jiri Olsa Em Fri, Jul 08, 2022 at 01:31:43PM -0700, Namhyung Kim escreveu: > Hello, > > On Tue, Jul 5, 2022 at 8:05 AM German Gomez <german.gomez@arm.com> wrote: > > > > Add a self test for branch stack sampling, to check that we get the > > expected branch types, and filters behave as expected. > > > > Suggested-by: Anshuman Khandual <anshuman.khandual@arm.com> > > Signed-off-by: German Gomez <german.gomez@arm.com> > > Acked-by: Namhyung Kim <namhyung@kernel.org> Thanks, applied. - Arnaldo ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH v2 1/1] perf test: Add test for branch stack sampling 2022-07-05 15:05 ` [PATCH v2 1/1] " German Gomez 2022-07-08 20:31 ` Namhyung Kim @ 2022-07-11 20:56 ` Jiri Olsa 2022-07-12 7:59 ` German Gomez 1 sibling, 1 reply; 7+ messages in thread From: Jiri Olsa @ 2022-07-11 20:56 UTC (permalink / raw) To: German Gomez Cc: linux-kernel, linux-perf-users, acme, irogers, Anshuman Khandual, Mark Rutland, Alexander Shishkin, Namhyung Kim On Tue, Jul 05, 2022 at 04:05:11PM +0100, German Gomez wrote: > Add a self test for branch stack sampling, to check that we get the > expected branch types, and filters behave as expected. > > Suggested-by: Anshuman Khandual <anshuman.khandual@arm.com> > Signed-off-by: German Gomez <german.gomez@arm.com> Tested-by: Jiri Olsa <jolsa@kernel.org> thanks, jirka > --- > tools/perf/tests/shell/test_brstack.sh | 114 +++++++++++++++++++++++++ > 1 file changed, 114 insertions(+) > create mode 100755 tools/perf/tests/shell/test_brstack.sh > > diff --git a/tools/perf/tests/shell/test_brstack.sh b/tools/perf/tests/shell/test_brstack.sh > new file mode 100755 > index 000000000..5f6f40a77 > --- /dev/null > +++ b/tools/perf/tests/shell/test_brstack.sh > @@ -0,0 +1,114 @@ > +#!/bin/sh > +# Check branch stack sampling > + > +# SPDX-License-Identifier: GPL-2.0 > +# German Gomez <german.gomez@arm.com>, 2022 > + > +# we need a C compiler to build the test programs > +# so bail if none is found > +if ! [ -x "$(command -v cc)" ]; then > + echo "failed: no compiler, install gcc" > + exit 2 > +fi > + > +# skip the test if the hardware doesn't support branch stack sampling > +perf record -b -o- -e dummy -B true > /dev/null 2>&1 || exit 2 > + > +TMPDIR=$(mktemp -d /tmp/__perf_test.program.XXXXX) > + > +cleanup() { > + rm -rf $TMPDIR > +} > + > +trap cleanup exit term int > + > +gen_test_program() { > + # generate test program > + cat << EOF > $1 > +#define BENCH_RUNS 999999 > +int cnt; > +void bar(void) { > +} /* return */ > +void foo(void) { > + bar(); /* call */ > +} /* return */ > +void bench(void) { > + void (*foo_ind)(void) = foo; > + if ((cnt++) % 3) /* branch (cond) */ > + foo(); /* call */ > + bar(); /* call */ > + foo_ind(); /* call (ind) */ > +} > +int main(void) > +{ > + int cnt = 0; > + while (1) { > + if ((cnt++) > BENCH_RUNS) > + break; > + bench(); /* call */ > + } /* branch (uncond) */ > + return 0; > +} > +EOF > +} > + > +test_user_branches() { > + echo "Testing user branch stack sampling" > + > + gen_test_program "$TEMPDIR/program.c" > + cc -fno-inline -g "$TEMPDIR/program.c" -o $TMPDIR/a.out > + > + perf record -o $TMPDIR/perf.data --branch-filter any,save_type,u -- $TMPDIR/a.out > /dev/null 2>&1 > + perf script -i $TMPDIR/perf.data --fields brstacksym | xargs -n1 > $TMPDIR/perf.script > + > + # example of branch entries: > + # foo+0x14/bar+0x40/P/-/-/0/CALL > + > + set -x > + egrep -m1 "^bench\+[^ ]*/foo\+[^ ]*/IND_CALL$" $TMPDIR/perf.script > + egrep -m1 "^foo\+[^ ]*/bar\+[^ ]*/CALL$" $TMPDIR/perf.script > + egrep -m1 "^bench\+[^ ]*/foo\+[^ ]*/CALL$" $TMPDIR/perf.script > + egrep -m1 "^bench\+[^ ]*/bar\+[^ ]*/CALL$" $TMPDIR/perf.script > + egrep -m1 "^bar\+[^ ]*/foo\+[^ ]*/RET$" $TMPDIR/perf.script > + egrep -m1 "^foo\+[^ ]*/bench\+[^ ]*/RET$" $TMPDIR/perf.script > + egrep -m1 "^bench\+[^ ]*/bench\+[^ ]*/COND$" $TMPDIR/perf.script > + egrep -m1 "^main\+[^ ]*/main\+[^ ]*/UNCOND$" $TMPDIR/perf.script > + set +x > + > + # some branch types are still not being tested: > + # IND COND_CALL COND_RET SYSCALL SYSRET IRQ SERROR NO_TX > +} > + > +# first argument <arg0> is the argument passed to "--branch-stack <arg0>,save_type,u" > +# second argument are the expected branch types for the given filter > +test_filter() { > + local filter=$1 > + local expect=$2 > + > + echo "Testing branch stack filtering permutation ($filter,$expect)" > + > + gen_test_program "$TEMPDIR/program.c" > + cc -fno-inline -g "$TEMPDIR/program.c" -o $TMPDIR/a.out > + > + perf record -o $TMPDIR/perf.data --branch-filter $filter,save_type,u -- $TMPDIR/a.out > /dev/null 2>&1 > + perf script -i $TMPDIR/perf.data --fields brstack | xargs -n1 > $TMPDIR/perf.script > + > + # fail if we find any branch type that doesn't match any of the expected ones > + # also consider UNKNOWN branch types (-) > + if egrep -vm1 "^[^ ]*/($expect|-|( *))$" $TMPDIR/perf.script; then > + return 1 > + fi > +} > + > +set -e > + > +test_user_branches > + > +test_filter "any_call" "CALL|IND_CALL|COND_CALL|SYSCALL|IRQ" > +test_filter "call" "CALL|SYSCALL" > +test_filter "cond" "COND" > +test_filter "any_ret" "RET|COND_RET|SYSRET|ERET" > + > +test_filter "call,cond" "CALL|SYSCALL|COND" > +test_filter "any_call,cond" "CALL|IND_CALL|COND_CALL|IRQ|SYSCALL|COND" > +test_filter "cond,any_call,any_ret" "COND|CALL|IND_CALL|COND_CALL|SYSCALL|IRQ|RET|COND_RET|SYSRET|ERET" > -- > 2.25.1 > ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH v2 1/1] perf test: Add test for branch stack sampling 2022-07-11 20:56 ` Jiri Olsa @ 2022-07-12 7:59 ` German Gomez 0 siblings, 0 replies; 7+ messages in thread From: German Gomez @ 2022-07-12 7:59 UTC (permalink / raw) To: Jiri Olsa Cc: linux-kernel, linux-perf-users, acme, irogers, Anshuman Khandual, Mark Rutland, Alexander Shishkin, Namhyung Kim On 11/07/2022 21:56, Jiri Olsa wrote: > On Tue, Jul 05, 2022 at 04:05:11PM +0100, German Gomez wrote: >> Add a self test for branch stack sampling, to check that we get the >> expected branch types, and filters behave as expected. >> >> Suggested-by: Anshuman Khandual <anshuman.khandual@arm.com> >> Signed-off-by: German Gomez <german.gomez@arm.com> > Tested-by: Jiri Olsa <jolsa@kernel.org> Thanks a lot, Jiri! > > thanks, > jirka > >> --- >> tools/perf/tests/shell/test_brstack.sh | 114 +++++++++++++++++++++++++ >> 1 file changed, 114 insertions(+) >> create mode 100755 tools/perf/tests/shell/test_brstack.sh >> >> diff --git a/tools/perf/tests/shell/test_brstack.sh b/tools/perf/tests/shell/test_brstack.sh >> new file mode 100755 >> index 000000000..5f6f40a77 >> --- /dev/null >> +++ b/tools/perf/tests/shell/test_brstack.sh >> @@ -0,0 +1,114 @@ >> +#!/bin/sh >> +# Check branch stack sampling >> + >> +# SPDX-License-Identifier: GPL-2.0 >> +# German Gomez <german.gomez@arm.com>, 2022 >> + >> +# we need a C compiler to build the test programs >> +# so bail if none is found >> +if ! [ -x "$(command -v cc)" ]; then >> + echo "failed: no compiler, install gcc" >> + exit 2 >> +fi >> + >> +# skip the test if the hardware doesn't support branch stack sampling >> +perf record -b -o- -e dummy -B true > /dev/null 2>&1 || exit 2 >> + >> +TMPDIR=$(mktemp -d /tmp/__perf_test.program.XXXXX) >> + >> +cleanup() { >> + rm -rf $TMPDIR >> +} >> + >> +trap cleanup exit term int >> + >> +gen_test_program() { >> + # generate test program >> + cat << EOF > $1 >> +#define BENCH_RUNS 999999 >> +int cnt; >> +void bar(void) { >> +} /* return */ >> +void foo(void) { >> + bar(); /* call */ >> +} /* return */ >> +void bench(void) { >> + void (*foo_ind)(void) = foo; >> + if ((cnt++) % 3) /* branch (cond) */ >> + foo(); /* call */ >> + bar(); /* call */ >> + foo_ind(); /* call (ind) */ >> +} >> +int main(void) >> +{ >> + int cnt = 0; >> + while (1) { >> + if ((cnt++) > BENCH_RUNS) >> + break; >> + bench(); /* call */ >> + } /* branch (uncond) */ >> + return 0; >> +} >> +EOF >> +} >> + >> +test_user_branches() { >> + echo "Testing user branch stack sampling" >> + >> + gen_test_program "$TEMPDIR/program.c" >> + cc -fno-inline -g "$TEMPDIR/program.c" -o $TMPDIR/a.out >> + >> + perf record -o $TMPDIR/perf.data --branch-filter any,save_type,u -- $TMPDIR/a.out > /dev/null 2>&1 >> + perf script -i $TMPDIR/perf.data --fields brstacksym | xargs -n1 > $TMPDIR/perf.script >> + >> + # example of branch entries: >> + # foo+0x14/bar+0x40/P/-/-/0/CALL >> + >> + set -x >> + egrep -m1 "^bench\+[^ ]*/foo\+[^ ]*/IND_CALL$" $TMPDIR/perf.script >> + egrep -m1 "^foo\+[^ ]*/bar\+[^ ]*/CALL$" $TMPDIR/perf.script >> + egrep -m1 "^bench\+[^ ]*/foo\+[^ ]*/CALL$" $TMPDIR/perf.script >> + egrep -m1 "^bench\+[^ ]*/bar\+[^ ]*/CALL$" $TMPDIR/perf.script >> + egrep -m1 "^bar\+[^ ]*/foo\+[^ ]*/RET$" $TMPDIR/perf.script >> + egrep -m1 "^foo\+[^ ]*/bench\+[^ ]*/RET$" $TMPDIR/perf.script >> + egrep -m1 "^bench\+[^ ]*/bench\+[^ ]*/COND$" $TMPDIR/perf.script >> + egrep -m1 "^main\+[^ ]*/main\+[^ ]*/UNCOND$" $TMPDIR/perf.script >> + set +x >> + >> + # some branch types are still not being tested: >> + # IND COND_CALL COND_RET SYSCALL SYSRET IRQ SERROR NO_TX >> +} >> + >> +# first argument <arg0> is the argument passed to "--branch-stack <arg0>,save_type,u" >> +# second argument are the expected branch types for the given filter >> +test_filter() { >> + local filter=$1 >> + local expect=$2 >> + >> + echo "Testing branch stack filtering permutation ($filter,$expect)" >> + >> + gen_test_program "$TEMPDIR/program.c" >> + cc -fno-inline -g "$TEMPDIR/program.c" -o $TMPDIR/a.out >> + >> + perf record -o $TMPDIR/perf.data --branch-filter $filter,save_type,u -- $TMPDIR/a.out > /dev/null 2>&1 >> + perf script -i $TMPDIR/perf.data --fields brstack | xargs -n1 > $TMPDIR/perf.script >> + >> + # fail if we find any branch type that doesn't match any of the expected ones >> + # also consider UNKNOWN branch types (-) >> + if egrep -vm1 "^[^ ]*/($expect|-|( *))$" $TMPDIR/perf.script; then >> + return 1 >> + fi >> +} >> + >> +set -e >> + >> +test_user_branches >> + >> +test_filter "any_call" "CALL|IND_CALL|COND_CALL|SYSCALL|IRQ" >> +test_filter "call" "CALL|SYSCALL" >> +test_filter "cond" "COND" >> +test_filter "any_ret" "RET|COND_RET|SYSRET|ERET" >> + >> +test_filter "call,cond" "CALL|SYSCALL|COND" >> +test_filter "any_call,cond" "CALL|IND_CALL|COND_CALL|IRQ|SYSCALL|COND" >> +test_filter "cond,any_call,any_ret" "COND|CALL|IND_CALL|COND_CALL|SYSCALL|IRQ|RET|COND_RET|SYSRET|ERET" >> -- >> 2.25.1 >> ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2022-07-12 11:57 UTC | newest] Thread overview: 7+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2022-07-05 15:05 [PATCH v2 0/1] perf test: Add test for branch stack sampling German Gomez 2022-07-05 15:05 ` [PATCH v2 1/1] " German Gomez 2022-07-08 20:31 ` Namhyung Kim 2022-07-11 12:50 ` German Gomez 2022-07-12 11:56 ` Arnaldo Carvalho de Melo 2022-07-11 20:56 ` Jiri Olsa 2022-07-12 7:59 ` German Gomez
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).