[BENCHMARK] ext3, reiser, jfs, xfs effect on contest

[BENCHMARK] ext3, reiser, jfs, xfs effect on contest

[Con Kolivas]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Using the osdl hardware (http://www.osdl.org) with contest 
(http://contest.kolivas.net) I've conducted a set of benchmarks with 
different filesystems. Note that contest does not claim to be a throughput 
benchmark.

All of these use kernel 2.5.59

First a set of contest benchmarks with the io load on a different hard disk 
containing each of the four filesystems:

io_other:
Kernel [runs]   Time    CPU%    Loads   LCPU%   Ratio
2559ext3    3   89      84.3    2       5.5     1.13
2559reiser  3   87      86.2    2       5.7     1.10
2559jfs     3   87      86.2    3       5.7     1.10
2559xfs     3   87      86.2    2       4.5     1.10

I found it interesting that there is virtually no difference in kernel 
compilation time with all fs. However jfs consistently wrote more during the 
io load than the other fs.


This is a set of benchmarks with the kernel compilation and load all performed 
on each of the fs:

ctar_load:
Kernel     [runs]       Time    CPU%    Loads   LCPU%   Ratio
2559ext3        2       96      82.3    2       5.2     1.23
2559jfs         2       103     73.8    0       0.0     1.32
2559reiser      2       100     78.0    0       1.0     1.27
2559xfs         2       97      82.5    2       5.2     1.23

Not sure why the jfs load shows up as 0 cpu% unless it's an accounting issue 
within the kernel. Subtle differences between fs times.


xtar_load:
Kernel     [runs]       Time    CPU%    Loads   LCPU%   Ratio
2559ext3        2       97      79.4    2       6.2     1.24
2559jfs         2       136     55.9    0       0.0     1.74
2559reiser      2       104     75.0    0       4.8     1.32
2559xfs         2       105     72.4    1       7.6     1.33

Once again jfs shows up no cpu%. These results show signficant prolongation of 
kernel compilation with repeated extracting of tars and jfs, without an 
increase in load work done.


io_load:
Kernel     [runs]       Time    CPU%    Loads   LCPU%   Ratio
2559ext3        3       109     68.8    4       10.1    1.40
2559jfs         3       138     54.3    11      13.8    1.77
2559reiser      3       98      76.5    2       9.2     1.24
2559xfs         3       124     60.5    6       8.0     1.57

This shows the largest discrepancy with jfs holding up kernel compilation the 
most and doing the most work and reiserfs at the other end. Cpu usage by the 
jfs load seems to make sense here.


read_load:
Kernel     [runs]       Time    CPU%    Loads   LCPU%   Ratio
2559ext3        2       98      80.6    7       7.1     1.26
2559jfs         2       97      79.4    5       5.2     1.24
2559reiser      2       101     79.2    6       7.9     1.28
2559xfs         2       98      80.6    6       7.1     1.24

Interestingly reading has the opposite order to writing but with probably 
insignificant differences in time. Note the lowish cpu usage by jfs again.

Comments?
Con
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.0 (GNU/Linux)

iD4DBQE+OngeF6dfvkL3i1gRAvKhAJYtmOkN1kLVuBMKI7Ygm317nXrUAJ0Y8UWI
IIdOlvqomgW5eEL4ZQkyGA==
=YDEz
-----END PGP SIGNATURE-----

Re: [BENCHMARK] ext3, reiser, jfs, xfs effect on contest

[Hans Reiser]

Be sure to create the tar on the same filesystem that you unpack it onto 
--- readdir order affects performance.

A result that we are faster for writes and slower for reads for 
workloads without large directories or small files is believable.

compilation is not an effective benchmark anymore, not for Linux 
filesystems, they are all just too fast (or is it that the compilers are 
too slow?....)

I don't know what ioload does....

-- 
Hans

Re: [BENCHMARK] ext3, reiser, jfs, xfs effect on contest

[Con Kolivas]

On Saturday 01 Feb 2003 12:37 am, Hans Reiser wrote:
> Be sure to create the tar on the same filesystem that you unpack it onto
> --- readdir order affects performance.

The tar creation is on the same filesystem as the unpacking

>
> A result that we are faster for writes and slower for reads for
> workloads without large directories or small files is believable.
>
> compilation is not an effective benchmark anymore, not for Linux
> filesystems, they are all just too fast (or is it that the compilers are
> too slow?....)
>
> I don't know what ioload does....

io load simply repeatedly writes a 256Mb file to the same filesystem as the 
compilation is occurring in. io_other writes the 256Mb file to a different 
hard disk containing the filesystem in question.

Con

Re: [BENCHMARK] ext3, reiser, jfs, xfs effect on contest

[Hans Reiser]

Try running with the -E option for gcc, it might be less CPU intensive, 
and thus a better FS benchmark.

What do you think?

-- 
Hans

Re: [BENCHMARK] ext3, reiser, jfs, xfs effect on contest

[Con Kolivas]

On Saturday 01 Feb 2003 12:56 am, Hans Reiser wrote:
> Try running with the -E option for gcc, it might be less CPU intensive,
> and thus a better FS benchmark.
>
> What do you think?

To be honest I have no idea. I'm not trying to find the filesystem with the 
greatest throughput, but which utilises the least resources while maintaining 
throughput. Perhaps we're looking for different things, but I'm open to any 
suggestions; but they'll have to wait for my waking hours, and for others' 
comments. These benchmarks are run manually for every stage as they are way 
out of the scope of the design of contest and therefore time consuming.

Re: [BENCHMARK] ext3, reiser, jfs, xfs effect on contest

[Dave Jones]

On Fri, Jan 31, 2003 at 04:56:07PM +0300, Hans Reiser wrote:
 > Try running with the -E option for gcc, it might be less CPU intensive, 
 > and thus a better FS benchmark.
 > 
 > What do you think?

It's hardly a realistic real-world benchmark if you start nobbling
bits of it though.  Not reading the preprocessed output is sure
to bump the benchmark points on an fs optimised for lots of small
writes.

		Dave

-- 
| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs

Re: [BENCHMARK] ext3, reiser, jfs, xfs effect on contest

[Hans Reiser]

Dave Jones wrote:

>On Fri, Jan 31, 2003 at 04:56:07PM +0300, Hans Reiser wrote:
> > Try running with the -E option for gcc, it might be less CPU intensive, 
> > and thus a better FS benchmark.
> > 
> > What do you think?
>
>It's hardly a realistic real-world benchmark if you start nobbling
>bits of it though.  Not reading the preprocessed output is sure
>to bump the benchmark points on an fs optimised for lots of small
>writes.
>
>		Dave
>
>  
>
Sigh.  The alternative is to strace the compile, write a perl scipt or 
something to get just the FS related calls out of it, and then create a 
program with just the FS related calls.  gcc -E sounds easier to me.;-)

-- 
Hans

Re: [BENCHMARK] ext3, reiser, jfs, xfs effect on contest

[Dave Jones]

On Fri, Jan 31, 2003 at 07:40:22PM +0300, Hans Reiser wrote:

 > >> Try running with the -E option for gcc, it might be less CPU intensive, 
 > >> and thus a better FS benchmark.
 > >> What do you think?
 > >It's hardly a realistic real-world benchmark if you start nobbling
 > >bits of it though.  Not reading the preprocessed output is sure
 > >to bump the benchmark points on an fs optimised for lots of small
 > >writes.
 > Sigh.  The alternative is to strace the compile, write a perl scipt or 
 > something to get just the FS related calls out of it, and then create a 
 > program with just the FS related calls. gcc -E sounds easier to me.;-)

It still seems like perverting a benchmark to turn it into dbench to me.

		Dave

-- 
| Dave Jones.        http://www.codemonkey.org.uk

Re: [BENCHMARK] ext3, reiser, jfs, xfs effect on contest

[Hans Reiser]

Dave Jones wrote:

>On Fri, Jan 31, 2003 at 07:40:22PM +0300, Hans Reiser wrote:
>
> > >> Try running with the -E option for gcc, it might be less CPU intensive, 
> > >> and thus a better FS benchmark.
> > >> What do you think?
> > >It's hardly a realistic real-world benchmark if you start nobbling
> > >bits of it though.  Not reading the preprocessed output is sure
> > >to bump the benchmark points on an fs optimised for lots of small
> > >writes.
> > Sigh.  The alternative is to strace the compile, write a perl scipt or 
> > something to get just the FS related calls out of it, and then create a 
> > program with just the FS related calls. gcc -E sounds easier to me.;-)
>
>It still seems like perverting a benchmark to turn it into dbench to me.
>
>		Dave
>
>  
>
Easier is not always correct.;-)  We need better benchmarks in our 
field, sigh.... and correct ones are really a lot of work....

-- 
Hans

Re: [BENCHMARK] ext3, reiser, jfs, xfs effect on contest

[Andrew Morton]

Hans Reiser <reiser@namesys.com> wrote:
>
> compilation is not an effective benchmark anymore, not for Linux 
> filesystems, they are all just too fast (or is it that the compilers are 
> too slow?....)
> 

The point of this test is to measure interactions, and fairness.

It answers the question "how much impact does heavy filesystem I/O have upon
other system activity?".

The "other system activity" in this test is a kernel compile.  That is a
fairly reasonable metric, because it is sensitive to latencies in servicing
reads and it is sensitive to inappropriate page replacement decisions.

A more appropriate foreground load might be opening a word processor and
composing a short letter to Aunt Nellie, but that's harder to automate.  We
expect that reduced kernel compilation time will correlate with lower-latency
letter writing.

Re: [BENCHMARK] ext3, reiser, jfs, xfs effect on contest

[Hans Reiser]

Andrew Morton wrote:

>Hans Reiser <reiser@namesys.com> wrote:
>  
>
>>compilation is not an effective benchmark anymore, not for Linux 
>>filesystems, they are all just too fast (or is it that the compilers are 
>>too slow?....)
>>
>>    
>>
>
>The point of this test is to measure interactions, and fairness.
>
>It answers the question "how much impact does heavy filesystem I/O have upon
>other system activity?".
>
>The "other system activity" in this test is a kernel compile.  That is a
>fairly reasonable metric, because it is sensitive to latencies in servicing
>reads and it is sensitive to inappropriate page replacement decisions.
>
>A more appropriate foreground load might be opening a word processor and
>composing a short letter to Aunt Nellie, but that's harder to automate.  We
>expect that reduced kernel compilation time will correlate with lower-latency
>letter writing.
>
>
>
>  
>
I think the result of the test was that this was not a compelling reason 
for users selecting a particular one of the filesystems because they all 
did well enough at it.  Perhaps because of your code.:)

However,  it is rather interesting for all the reasons you mention.  
There is indeed a tendency for benchmarks to discount the importance of 
latency, and this benchmark does not do that, which is good.  It is 
annoying to be unable to work while a big tar is running in the 
background, but few benchmarks capture that.

We should test reiser4 against this next month, it would be 
interesting.  (It seems we finally fixed the Reiser4 performance problem 
that we hit in August, and now we just need to tweak the CPU usage a bit 
and we'll have something performing pretty decently in our next release....)

-- 
Hans

Re: [BENCHMARK] ext3, reiser, jfs, xfs effect on contest

[Con Kolivas]

On Saturday 01 Feb 2003 6:29 am, Hans Reiser wrote:
> Andrew Morton wrote:
> >Hans Reiser <reiser@namesys.com> wrote:
> >>compilation is not an effective benchmark anymore, not for Linux
> >>filesystems, they are all just too fast (or is it that the compilers are
> >>too slow?....)
> >
> >The point of this test is to measure interactions, and fairness.
> >
> >It answers the question "how much impact does heavy filesystem I/O have
> > upon other system activity?".
> >
> >The "other system activity" in this test is a kernel compile.  That is a
> >fairly reasonable metric, because it is sensitive to latencies in
> > servicing reads and it is sensitive to inappropriate page replacement
> > decisions.
> >
> >A more appropriate foreground load might be opening a word processor and
> >composing a short letter to Aunt Nellie, but that's harder to automate. 
> > We expect that reduced kernel compilation time will correlate with
> > lower-latency letter writing.
>
> I think the result of the test was that this was not a compelling reason
> for users selecting a particular one of the filesystems because they all
> did well enough at it.  Perhaps because of your code.:)
>
> However,  it is rather interesting for all the reasons you mention.
> There is indeed a tendency for benchmarks to discount the importance of
> latency, and this benchmark does not do that, which is good.  It is
> annoying to be unable to work while a big tar is running in the
> background, but few benchmarks capture that.
>
> We should test reiser4 against this next month, it would be
> interesting.  (It seems we finally fixed the Reiser4 performance problem
> that we hit in August, and now we just need to tweak the CPU usage a bit
> and we'll have something performing pretty decently in our next
> release....)

Actually the most "felt" of these loads is io_load and based on these results:
io_load:
Kernel     [runs]       Time    CPU%    Loads   LCPU%   Ratio
2559ext3        3       109     68.8    4       10.1    1.40
2559jfs         3       138     54.3    11      13.8    1.77
2559reiser      3       98      76.5    2       9.2     1.24
2559xfs         3       124     60.5    6       8.0     1.57

I'd say barring any concern about throughput which this doesnt claim to 
measure accurately reiserfs causes the least slowdown of the system  ;-)

I do have one more load which may be useful. dbench_load continually runs 
dbench in the background. I could throw that at it also.

Ext2 was left out for clarity because it wasn't a journalling fs but it's 
results are quite different to the journalled fss.

Con

Re: [BENCHMARK] ext3, reiser, jfs, xfs effect on contest

[Con Kolivas]

On Saturday 01 Feb 2003 9:21 am, Con Kolivas wrote:
> Actually the most "felt" of these loads is io_load and based on these
> results: io_load:
> Kernel     [runs]       Time    CPU%    Loads   LCPU%   Ratio
> 2559ext3        3       109     68.8    4       10.1    1.40
> 2559jfs         3       138     54.3    11      13.8    1.77
> 2559reiser      3       98      76.5    2       9.2     1.24
> 2559xfs         3       124     60.5    6       8.0     1.57
>

Here is a set of dbench_load results:
dbench_load:
Kernel     [runs]       Time    CPU%    Loads   LCPU%   Ratio
2559ext3        3       115     65.2    3       24.3    1.47
2559jfs         3       123     61.8    3       19.5    1.58
2559reiser      3       108     69.4    3       13.9    1.37
2559xfs         3       118     63.6    3       22.0    1.49

Note the order correlates with the order of the io_load results.

Once again reiserfs held up kernel compilation the least. Note they all 
accomplished the same work in that time though.

Con

Re: [BENCHMARK] ext3, reiser, jfs, xfs effect on contest

[David Lang]

I think it would be interesting to add ext2 in as well becouse a LOT of
people are still running ext2 and it would be nice to know how much
performance is being lost to gai the advantages of journaling.

David Lang

On Sat, 1 Feb 2003, Con Kolivas wrote:

> Date: Sat, 1 Feb 2003 09:21:59 +1100
> From: Con Kolivas <conman@kolivas.net>
> To: Hans Reiser <reiser@namesys.com>, Andrew Morton <akpm@digeo.com>
> Cc: linux-kernel@vger.kernel.org, Alexander Zarochentcev <zam@namesys.com>
> Subject: Re: [BENCHMARK] ext3, reiser, jfs, xfs effect on contest
>
> On Saturday 01 Feb 2003 6:29 am, Hans Reiser wrote:
> > Andrew Morton wrote:
> > >Hans Reiser <reiser@namesys.com> wrote:
> > >>compilation is not an effective benchmark anymore, not for Linux
> > >>filesystems, they are all just too fast (or is it that the compilers are
> > >>too slow?....)
> > >
> > >The point of this test is to measure interactions, and fairness.
> > >
> > >It answers the question "how much impact does heavy filesystem I/O have
> > > upon other system activity?".
> > >
> > >The "other system activity" in this test is a kernel compile.  That is a
> > >fairly reasonable metric, because it is sensitive to latencies in
> > > servicing reads and it is sensitive to inappropriate page replacement
> > > decisions.
> > >
> > >A more appropriate foreground load might be opening a word processor and
> > >composing a short letter to Aunt Nellie, but that's harder to automate.
> > > We expect that reduced kernel compilation time will correlate with
> > > lower-latency letter writing.
> >
> > I think the result of the test was that this was not a compelling reason
> > for users selecting a particular one of the filesystems because they all
> > did well enough at it.  Perhaps because of your code.:)
> >
> > However,  it is rather interesting for all the reasons you mention.
> > There is indeed a tendency for benchmarks to discount the importance of
> > latency, and this benchmark does not do that, which is good.  It is
> > annoying to be unable to work while a big tar is running in the
> > background, but few benchmarks capture that.
> >
> > We should test reiser4 against this next month, it would be
> > interesting.  (It seems we finally fixed the Reiser4 performance problem
> > that we hit in August, and now we just need to tweak the CPU usage a bit
> > and we'll have something performing pretty decently in our next
> > release....)
>
> Actually the most "felt" of these loads is io_load and based on these results:
> io_load:
> Kernel     [runs]       Time    CPU%    Loads   LCPU%   Ratio
> 2559ext3        3       109     68.8    4       10.1    1.40
> 2559jfs         3       138     54.3    11      13.8    1.77
> 2559reiser      3       98      76.5    2       9.2     1.24
> 2559xfs         3       124     60.5    6       8.0     1.57
>
> I'd say barring any concern about throughput which this doesnt claim to
> measure accurately reiserfs causes the least slowdown of the system  ;-)
>
> I do have one more load which may be useful. dbench_load continually runs
> dbench in the background. I could throw that at it also.
>
> Ext2 was left out for clarity because it wasn't a journalling fs but it's
> results are quite different to the journalled fss.
>
> Con
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Re: [BENCHMARK] ext3, reiser, jfs, xfs effect on contest

[Mike A. Harris]

On Sat, 1 Feb 2003, Con Kolivas wrote:

>Using the osdl hardware (http://www.osdl.org) with contest 
>(http://contest.kolivas.net) I've conducted a set of benchmarks with 
>different filesystems. Note that contest does not claim to be a throughput 
>benchmark.
>
>All of these use kernel 2.5.59
>
>First a set of contest benchmarks with the io load on a different hard disk 
>containing each of the four filesystems:
>
>io_other:
>Kernel [runs]   Time    CPU%    Loads   LCPU%   Ratio
>2559ext3    3   89      84.3    2       5.5     1.13
>2559reiser  3   87      86.2    2       5.7     1.10
>2559jfs     3   87      86.2    3       5.7     1.10
>2559xfs     3   87      86.2    2       4.5     1.10
>
>I found it interesting that there is virtually no difference in kernel 
>compilation time with all fs. However jfs consistently wrote more during the 
>io load than the other fs.
>
>
>This is a set of benchmarks with the kernel compilation and load all performed 
>on each of the fs:

Compilation is inherently CPU bound, not disk I/O bound, so 
compiling the kernel (or anything for that matter) isn't going to 
show any difference really because the CPU Mhz and L1/L2 cache 
are the bottleneck.


-- 
Mike A. Harris     ftp://people.redhat.com/mharris
OS Systems Engineer - XFree86 maintainer - Red Hat

Re: [BENCHMARK] ext3, reiser, jfs, xfs effect on contest

[Con Kolivas]

On Saturday 01 Feb 2003 1:09 am, Mike A. Harris wrote:
> On Sat, 1 Feb 2003, Con Kolivas wrote:
> >Using the osdl hardware (http://www.osdl.org) with contest
> >(http://contest.kolivas.net) I've conducted a set of benchmarks with
> >different filesystems. Note that contest does not claim to be a throughput
> >benchmark.
> >
> >All of these use kernel 2.5.59
> >
> >First a set of contest benchmarks with the io load on a different hard
> > disk containing each of the four filesystems:
> >
> >io_other:
> >Kernel [runs]   Time    CPU%    Loads   LCPU%   Ratio
> >2559ext3    3   89      84.3    2       5.5     1.13
> >2559reiser  3   87      86.2    2       5.7     1.10
> >2559jfs     3   87      86.2    3       5.7     1.10
> >2559xfs     3   87      86.2    2       4.5     1.10
> >
> >I found it interesting that there is virtually no difference in kernel
> >compilation time with all fs. However jfs consistently wrote more during
> > the io load than the other fs.
> >
> >
> >This is a set of benchmarks with the kernel compilation and load all
> > performed on each of the fs:
>
> Compilation is inherently CPU bound, not disk I/O bound, so
> compiling the kernel (or anything for that matter) isn't going to
> show any difference really because the CPU Mhz and L1/L2 cache
> are the bottleneck.

When the io load is on another hard disk yes, however the results do show 
differences when the load is on the same hard disk - these are two different 
tests. There is more to kernel compilation than just cpu usage.

Re: [BENCHMARK] ext3, reiser, jfs, xfs effect on contest

[Maciej Soltysiak]

> compiling the kernel (or anything for that matter) isn't going to
> show any difference really because the CPU Mhz and L1/L2 cache
> are the bottleneck.
How about a test comprising of lots of mail being
sent/rejected/bounced/deferred/etc.
Usually SMTPs like postfix store lots of directories and files in there.
And thus create lots of reads/writes. We could measure the efficiency of
that.
I think it is possible to DoS a system by thrashing its i/o by forcing the
smtp to do lots of work. With very poor io efficiency that is.

Propably reiserfs would have better results with such a test, whereas
ext3 could have better results on a differents test. (different
application)

I think the 'best result' fs will vary on the test.
Also i think it is better to testdrive the file systems using real
applications on high load.

Regards,
Maciej Soltysiak

Re: [BENCHMARK] ext3, reiser, jfs, xfs effect on contest

[Con Kolivas]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Saturday 01 Feb 2003 12:20 am, Con Kolivas wrote:
> Using the osdl hardware (http://www.osdl.org) with contest
> (http://contest.kolivas.net) I've conducted a set of benchmarks with
> different filesystems. Note that contest does not claim to be a throughput
> benchmark.

Ok to complete the picture here is the set of results including dbench_load 
and ext2.

ctar_load:
Kernel     [runs]       Time    CPU%    Loads   LCPU%   Ratio
2559ext2        3       92      85.9    2       4.3     1.18
2559ext3        2       96      82.3    2       5.2     1.23
2559jfs         2       103     73.8    0       0.0     1.32
2559reiser      2       100     78.0    0       1.0     1.27
2559xfs         2       97      82.5    2       5.2     1.23
xtar_load:
Kernel     [runs]       Time    CPU%    Loads   LCPU%   Ratio
2559ext2        3       92      82.6    1       3.3     1.18
2559ext3        2       97      79.4    2       6.2     1.24
2559jfs         2       136     55.9    0       0.0     1.74
2559reiser      2       104     75.0    0       4.8     1.32
2559xfs         2       105     72.4    1       7.6     1.33
io_load:
Kernel     [runs]       Time    CPU%    Loads   LCPU%   Ratio
2559ext2        3       105     71.4    5       6.6     1.35
2559ext3        3       109     68.8    4       10.1    1.40
2559jfs         3       138     54.3    11      13.8    1.77
2559reiser      3       98      76.5    2       9.2     1.24
2559xfs         3       124     60.5    6       8.0     1.57
read_load:
Kernel     [runs]       Time    CPU%    Loads   LCPU%   Ratio
2559ext2        3       96      81.2    6       6.2     1.23
2559ext3        2       98      80.6    7       7.1     1.26
2559jfs         2       97      79.4    5       5.2     1.24
2559reiser      2       101     79.2    6       7.9     1.28
2559xfs         2       98      80.6    6       7.1     1.24
dbench_load:
Kernel     [runs]       Time    CPU%    Loads   LCPU%   Ratio
2559ext2        3       102     72.5    3       17.6    1.31
2559ext3        3       115     65.2    3       24.3    1.47
2559jfs         3       123     61.8    3       19.5    1.58
2559reiser      3       108     69.4    3       13.9    1.37
2559xfs         3       118     63.6    3       22.0    1.49
io_other:
Kernel     [runs]	Time    CPU%    Loads   LCPU%   Ratio
2559ext2        3   	90      83.3    3       4.4     1.14
2559ext3        3   	89      84.3    2       5.5     1.13
2559reiser      3   	87      86.2    2       5.7     1.10
2559jfs         3   	87      86.2    3       5.7     1.10
2559xfs         3   	87      86.2    2       4.5     1.10

Con
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.0 (GNU/Linux)

iD8DBQE+OxEEF6dfvkL3i1gRAsnrAJ0YZ4FYDLA6iA0R37ccJGdIoqIcJwCeJVbu
vs0QdsTDhfti32WI5c9NJHc=
=qN8n
-----END PGP SIGNATURE-----