Re: CEPH Erasure Encoding + OSD Scalability

[Loic Dachary <loic@dachary.org>]

[-- Attachment #1: Type: text/plain, Size: 5686 bytes --]

Hi Andreas,

It looks like this code would be useful as a standalone program instead of being integrated within unit tests. There are a few support program of that kind. The unit tests are probably not the best place for benchmarks. What do you think ?

Cheers

Note: got the updated file ;-)

> I fixed some more issues in the test:
> 
> 1 strncmp => memcmp (object has now binary data and 0 ..)
> 2 I fixed the length of the second block which was wrong after the change from 2,2 to 4,2
> 3 I have put 1MB as default object size, otherwise running with valgrind is too long
> 4 I have simplified the time measurement code etc.
> 

On 23/09/2013 17:43, Andreas Joachim Peters wrote:
> Hi Loic, 
> 
> I have modified the Jerasure unit test, to record the encoding & reconstruction performance and to store this value in the optional google-test xml outputfile. I have put (4,2) with a 4MB random object as default and one can pass a different object size via '--object-size=1000' (for 1G).
> 
> It looks like this:
> 
> ./unittest_erasure_code_jerasure --gtest_filter=*.* --log-to-stderr=true --gtest_output="xml:erasure.xml" --object-size=1000
> Note: Google Test filter = *.*
> [==========] Running 8 tests from 8 test cases.
> [----------] Global test environment set-up.
> [----------] 1 test from ErasureCodeTest/0, where TypeParam = ErasureCodeJerasureReedSolomonVandermonde
> [ RUN      ] ErasureCodeTest/0.encode_decode
> [       OK ] ErasureCodeTest/0.encode_decode (35231 ms)
> [----------] 1 test from ErasureCodeTest/0 (35231 ms total)
> 
> [----------] 1 test from ErasureCodeTest/1, where TypeParam = ErasureCodeJerasureReedSolomonRAID6
> [ RUN      ] ErasureCodeTest/1.encode_decode
> [       OK ] ErasureCodeTest/1.encode_decode (35594 ms)
> [----------] 1 test from ErasureCodeTest/1 (35594 ms total)
> 
> [----------] 1 test from ErasureCodeTest/2, where TypeParam = ErasureCodeJerasureCauchyOrig
> [ RUN      ] ErasureCodeTest/2.encode_decode
> [       OK ] ErasureCodeTest/2.encode_decode (33009 ms)
> [----------] 1 test from ErasureCodeTest/2 (33010 ms total)
> 
> [----------] 1 test from ErasureCodeTest/3, where TypeParam = ErasureCodeJerasureCauchyGood
> [ RUN      ] ErasureCodeTest/3.encode_decode
> [       OK ] ErasureCodeTest/3.encode_decode (31917 ms)
> [----------] 1 test from ErasureCodeTest/3 (31920 ms total)
> 
> [----------] 1 test from ErasureCodeTest/4, where TypeParam = ErasureCodeJerasureLiberation
> [ RUN      ] ErasureCodeTest/4.encode_decode
> [       OK ] ErasureCodeTest/4.encode_decode (31801 ms)
> [----------] 1 test from ErasureCodeTest/4 (31801 ms total)
> 
> [----------] 1 test from ErasureCodeTest/5, where TypeParam = ErasureCodeJerasureBlaumRoth
> [ RUN      ] ErasureCodeTest/5.encode_decode
> [       OK ] ErasureCodeTest/5.encode_decode (31927 ms)
> [----------] 1 test from ErasureCodeTest/5 (31927 ms total)
> 
> [----------] 1 test from ErasureCodeTest/6, where TypeParam = ErasureCodeJerasureLiber8tion
> [ RUN      ] ErasureCodeTest/6.encode_decode
> [       OK ] ErasureCodeTest/6.encode_decode (31824 ms)
> [----------] 1 test from ErasureCodeTest/6 (31824 ms total)
> 
> [----------] 1 test from ErasureCodeTiming
> [ RUN      ] ErasureCodeTiming.PropertyOutput
> [ -TIMING- ] technique=blaum_roth       speed [ encode ]=2.902 [GB/s]
> [ -TIMING- ] technique=blaum_roth       speed [   reco ]=1.701 [GB/s]
> [ -TIMING- ] technique=cauchy_good      speed [ encode ]=2.551 [GB/s]
> [ -TIMING- ] technique=cauchy_good      speed [   reco ]=1.571 [GB/s]
> [ -TIMING- ] technique=cauchy_orig      speed [ encode ]=1.401 [GB/s]
> [ -TIMING- ] technique=cauchy_orig      speed [   reco ]=0.911 [GB/s]
> [ -TIMING- ] technique=liber8tion       speed [ encode ]=2.861 [GB/s]
> [ -TIMING- ] technique=liber8tion       speed [   reco ]=1.822 [GB/s]
> [ -TIMING- ] technique=liberation       speed [ encode ]=2.863 [GB/s]
> [ -TIMING- ] technique=liberation       speed [   reco ]=1.815 [GB/s]
> [ -TIMING- ] technique=reed_sol_r6_op   speed [ encode ]=1.194 [GB/s]
> [ -TIMING- ] technique=reed_sol_r6_op   speed [   reco ]=0.489 [GB/s]
> [ -TIMING- ] technique=reed_sol_van     speed [ encode ]=0.600 [GB/s]
> [ -TIMING- ] technique=reed_sol_van     speed [   reco ]=0.429 [GB/s]
> [       OK ] ErasureCodeTiming.PropertyOutput (0 ms)
> [----------] 1 test from ErasureCodeTiming (0 ms total)
> 
> [----------] Global test environment tear-down
> [==========] 8 tests from 8 test cases ran. (231307 ms total)
> [  PASSED  ] 8 tests.
> 
> 
> [----------] Global test environment tear-down
> [==========] 8 tests from 8 test cases ran. (31351 ms total)
> [  PASSED  ] 8 tests.
> 
> And the XML:
> 
> <testsuite name="ErasureCodeTiming" tests="1" failures="0" disabled="0" errors="0" time="0">
>     <testcase name="PropertyOutput" status="run" time="0" classname="ErasureCodeTiming" jerasure::blaum_roth::encode="2902" jerasure::blaum_roth::reco="1700" jerasure::cauchy_good::encode="2551" jerasure::cauchy_good::reco="1571" jerasure::cauchy_orig::encode="1401" jerasure::cauchy_orig::reco="910" jerasure::liber8tion::encode="2861" jerasure::liber8tion::reco="1821" jerasure::liberation::encode="2862" jerasure::liberation::reco="1814" jerasure::reed_sol_r6_op::encode="1194" jerasure::reed_sol_r6_op::reco="489" jerasure::reed_sol_van::encode="599" jerasure::reed_sol_van::reco="428" object-size="1000000000" />
>   </testsuite>
> 
> Maybe you could use this directly for QA.
> 
> Cheers Andreas.
> 

-- 
Loïc Dachary, Artisan Logiciel Libre
All that is necessary for the triumph of evil is that good people do nothing.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 261 bytes --]