From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail5.wrs.com (mail5.wrs.com [192.103.53.11]) by mx.groups.io with SMTP id smtpd.web11.1963.1592342392601645872 for ; Tue, 16 Jun 2020 14:19:52 -0700 Authentication-Results: mx.groups.io; dkim=missing; spf=pass (domain: windriver.com, ip: 192.103.53.11, mailfrom: mark.morton@windriver.com) Received: from ALA-HCA.corp.ad.wrs.com (ala-hca.corp.ad.wrs.com [147.11.189.40]) by mail5.wrs.com (8.15.2/8.15.2) with ESMTPS id 05GLJVCk019502 (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL) for ; Tue, 16 Jun 2020 14:19:41 -0700 Received: from san-mmorton-lx1.corp.ad.wrs.com (147.11.4.21) by ALA-HCA.corp.ad.wrs.com (147.11.189.40) with Microsoft SMTP Server id 14.3.487.0; Tue, 16 Jun 2020 14:19:20 -0700 From: "Mark Morton" To: Subject: [PATCH] New source files and Makefile update for Test Manual Date: Tue, 16 Jun 2020 14:14:50 -0700 Message-ID: <20200616211450.8079-1-mark.morton@windriver.com> X-Mailer: git-send-email 2.17.1 MIME-Version: 1.0 X-MIME-Autoconverted: from 8bit to quoted-printable by mail5.wrs.com id 05GLJVCk019502 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Signed-off-by: Mark Morton --- documentation/Makefile | 2 +- .../test-manual/figures/ab-test-cluster.png | Bin 0 -> 18684 bytes .../test-manual/figures/test-manual-title.png | Bin 0 -> 15382 bytes .../test-manual/test-manual-customization.xsl | 27 + .../test-manual/test-manual-intro.xml | 634 +++++++++++ .../test-manual/test-manual-style.css | 989 ++++++++++++++++++ .../test-manual/test-manual-test-process.xml | 109 ++ .../test-manual-understand-autobuilder.xml | 312 ++++++ documentation/test-manual/test-manual.xml | 103 ++ 9 files changed, 2175 insertions(+), 1 deletion(-) create mode 100644 documentation/test-manual/figures/ab-test-cluster.png create mode 100644 documentation/test-manual/figures/test-manual-title.p= ng create mode 100644 documentation/test-manual/test-manual-customization.x= sl create mode 100644 documentation/test-manual/test-manual-intro.xml create mode 100644 documentation/test-manual/test-manual-style.css create mode 100644 documentation/test-manual/test-manual-test-process.xm= l create mode 100644 documentation/test-manual/test-manual-understand-auto= builder.xml create mode 100755 documentation/test-manual/test-manual.xml diff --git a/documentation/Makefile b/documentation/Makefile index 3bdf32b89..ff0494b95 100644 --- a/documentation/Makefile +++ b/documentation/Makefile @@ -361,7 +361,7 @@ ifeq ($(DOC),test-manual) XSLTOPTS =3D --xinclude ALLPREQ =3D html tarball TARFILES =3D test-manual.html test-manual-style.css \ - figures/test-manual-title.png figures/ab-test-stack.png + figures/test-manual-title.png figures/ab-test-cluster.png MANUALS =3D $(DOC)/$(DOC).html FIGURES =3D figures STYLESHEET =3D $(DOC)/*.css diff --git a/documentation/test-manual/figures/ab-test-cluster.png b/docu= mentation/test-manual/figures/ab-test-cluster.png new file mode 100644 index 0000000000000000000000000000000000000000..6a6a7882b43895cd3e81e18b8= 829c88af5b2ca8c GIT binary patch literal 18684 zcmcG#1z6i(x9^F&OKEYpmH=3D&WcL?q+PLUROcWBXKArM@GLxED0sJ{AVzKPGcX zdx$q=3DH!THeq{?CH9mEH;k5Z~qNJ!Q3xOe90h|k#0itpW!kO+P~y^xb%5>O!_DJCk( zO6ho;94z40lKsw-IA&yftZ8{Kknx!u4Oy;+wME_nOdqQFHfDtHdABXmZ!~OuDV>Cp zNJ=3DP*EtwoyNUkO$^iLzn8m>Mzjx6!$^P$bcQ;@#&;O2gx()_~wqrcs*09?SM66 zhhQc~C_Qf{qQr#f%aM`nig?b5N|-SO85wa%Q2(>!f*DBUU6J#$vJeyBQlahKuz9^p zxgpk0Bn9-Nj>q=3D2w>F>`5`nD)3{btgPk$L}%(_sRp$SiE2Jr>vm#4nI?+6Ug40V{RPa}V+H+^B?eSZO4pwxVa=3D;~{TO9M`7j}H84%XVA*q5z(aY8Jya$i;ORK#(VZ#j@muj$ox+vPJ5XgUlJmZRsA_{U{lN7o&vVq9=3D0qP zn2hZulXwu=3D-%1hO?U5FCA+k#a++KRvooF;CjxDbKj&bNy0|u1*nQ*WT!@&P>>q2cN z0d$EMyG+F|3Jy5i9IDwvk$P4o6PUjD`!9=3D5tI3$A0*zHzCivBtc5zJu#LT^@_1M#E zc;*B@T=3D0xqcAhqmnDIbEvk%*ka2pot0c{8tOBpFdYO4DwGa0)-`4mEZ+numFbrz8<3(>YhvWRedD^W*f6;QjRE|)MGMUf zqOKoz?-#t7&|Wa`scSTu5JEb7X`{gk3?~5=3D*lssbDrPJ)8u*2{>hT`w4ylh*(Y&db3Niju->Q?-WnFOP;q=3DbEr0G z8C69!!FC0;>5Rupgt(kNLrbrmU9(VX#%p+@bkphK7_%XnR$^i9I(4F0A}g5I77`R>8hMhXeJA;D~n#W%xeW6Io3)V64d9a|QY;nY5J zezdeS@6tm9U^_zWj)v=3DLQ!cbV1-gI<2VLv^>FQ0dj$32RpQvM#XifL+F1g<{UI4@#w!WoTI0Am_LKqVwww? zfj2vwjcai)!K(%5;r=3DW7gz^Dtx4Kb+Bg6{S=3D3%3aE^E<1!yCZ@V5rQ|V%vf1)^L_* z)LAat7b}pRd6Yv(Gzt7$JiO2+M~4F(umZA(vAHa-#huIbj$;pUhABdcJLd)g z_4bL4)L1HkXTT7wlo$$b{&>;TNHNyFYtVFiK#TEV&4{=3Ds_M4BxwXn)_1Hp%Bi+C>T zczCdA;%0!>Ys4bd(f0<5WV>*=3D95!t|s~_89 zSmv#m(6PeGXR^{t14n4%#p-koDGvsq2>Plj)M*H!39ekE%dE5#)1vj>?Q^@qqWk5^ z+yOrt-rxUnZIF|#qjxa8$Sr@tG6{|NNcJeK*!)><1U?h}E|duxo)AZUqij57l~Ws{ zgk20~-YQS7g@IZd%f{XwC4nr0^9QNkSbq1}t{g6OS~sWTW7Fxm9_cfu@zK(au$8Et zQ_U3o>xip!lo!EZKUY5lNXo3z8#7UkgR#htPLkilEOI^gC2t% z-)&uP?gV`MVNc*zm1(U9*!nn`%x^!fF@X{8v2L?16uC^_L!alxAZ(Lcp#qmzacwtH z^5fc*>%4#6mrza=3DDWwYdE@aGPz! zb{t3zqUGt`p9fW$J}w*@dzRjh6Qert?9F&dw1{1&wCt0JpQIssa&*P-?wQ`~+bzCp z7zVJI8|Hp4g7*dLcvPBRFuieGMRWVT{rB?Se$pBXq4T%Ewt9tSB%LRlWmK7g+bzfkAin7`x^g7<*X!g8rXZ(PlKA1ju z;c~l!;?jcV*2$=3DT@O;rp;?4c=3D^=3D5dX^1;b*R)*u-CF7R*gos@CU*FB;wiL1({U_Kn z*lyKn-W0J5;xw_wb@6qGxfKOe1TLs1al0sJzv+r(^QNY+6pL#nC}$UAw_)(!F)LBF z3?9qc7unw);T=3DUBpCyC zm-eU4AzOHdMsggtMBC@$;OF_2mI< z4cDT(9Iz{`zC0v<6v}mHnG8k^52$2UWMQd-?MxFh*^{$}7FR-LFPM_U|Eju=3DNELij z!4Cu6HRd#K!9bED3Ol=3D1eUkl*%10t(c+Rx7)p@Xj+G;UZn8Rjz3uGo>TE3595~9Tw z_164Kv!0T)s&IC%Ehy-0cbCtx9QAaRcGJ(T+qWUb$t$cRH^P$-jiI)U#Ya?fm>71w zwpF#M{Ma0QdI5R)_`x3iO3Sp#)k^l z5-NM7ID@f(!M3xlEk7OLOLIq?0$!YoqD@Zf0HiYybg>-eF`%~(w#nR$2(Q}UI-H-q2xwa>B4wj);>bn`4Q zLaDP~mKUwd{vx5aqD8^=3DeGL^;E+hN9M^so$LH`JHwCFlsCf1dgb|vKf)ZJ4rxYCq0*e!bC~Wf!Dt(d& zi}^TIKZo@p7j83MWm+}Dy(KA4UefLq0^2+5=3DM{cu(GjTni8&ZgN1{cGNTYwKEj&@m zQbPV-l8;&5gGS+=3DjY}csO!Jb;#VSi zPchC^H>J@J;*|S3dxY6syKW@xvU#wNE!=3D4yY`KuNq+a)&j%2H|>23xsvgB>8N1i1) zCNfr*jIG#62HvNyw3!_@lU4R6V{VWP7#B|E*-u>V#tMR78AIAh^QAflh`Mb_mieJ=3DIS+o(} z57%2Uz8k;tE2wM7!~>DK~pthmu4J%%j0Z?l1jj_R!nGvuaOSSxMm_# z*M$UG8=3D<8qO$U+}nWA&_AZ)%&?>n=3Da?Tm5z_D=3DrOQt{*YH1}?WpKoluO#n6tNr0PJ z+lDpf^4|v*pt0e$%?=3DDHbRQDQe&g;XwqsxW+d?5Up@7zoCAO`a9j$ntNa(k+uCZNqfIe4f3z`9tca1`e>C zfUCe3)r+$$5Q|(P6Wa5yhNTg0WuL*@?zz97L&VYe^4Rpz5xZB95Q}S1lSQ7ftRjWb zS<7I2Kb^#JbQn9EHnvwdVJ5- z7gUsILqlmjViTDZo;5G*efDPD5u(_7 zal$xm^hpr=3DZ<^cRZ;T17lt5!iXaE}Kd=3Dq_J*U1vMs~!!H^H)fqY{(=3DG!qA!AKN02pSmbZPRuNlg+ugFHL(c!We9^>oE_FMR+1ixrH}bJmgT}v|p-~oXSSGE9kiU{AM&hH%%>CN#C@55!P2j{^wt@ucRx& zvQ$b+ud`GVe2`ea!0&Ofu{g4uk0 zY1RdmTxmI8?20NiTe6-9G7sfgCYP~|PZgwr6wIc%UdZEO&lQ?=3DP{+Xfn9ITVc+SKH zIG~X#V!oWTCKBG+&9S&Mc9t4${}o(MY%GJqU#dTVGl*-ntg130=3DoPm1=3D=3DR@-r0DcJ zozQ1|#kr{S7BO2CzT&$&F+R(YB_~i{O+Au%c9kv+K{NsP`<%!9cn6RCg1&}%D^@&- zei}L13KgIO5nr{f{*a*B1+jf_H_!Vmm9{c%k#zTKHt1AXAMirdtyw!#bm=3DmT?yeJ% zXGBG9W%wPLAV0o0HDSZ7o!|INskz~1L^nIL(x$(%n>gdHu(!Ng!<;!(3qMPqXkztJ z+YOt)Gq!b^L?j&xC5kTLEn2=3Dh;S0TzpR%Rum@hj=3DjZ1ca$;@sba{Gxhmf-hZ}&y3bHdHT5}Vn+6yVXgrdj6LWhlO^UwG)%uq8&yFKzIKAVL}V!nQfB%Gy) zbPn&5INyEFOB=3D%B_}}tYyR7|za;ki_E1lafX+oo0ev8I{CN#>zi5)C_y{2gU-x}JF zQ5p`eHSfk~h)CB|MiFKN%VVE*sp~RM9;T8h1%;LW8WEjomBq2=3D#ilj21JGZ*Y^>o(1gLn!RJXyY5 zeo49HY){P_L139Wa(i+X6rhgb^G(Y$2KF5bB6{@t{MnJ!P5-c`zRT84jyXJW8ohQzZOHjqKb-&fR&$L9LrAvZ+qZF=3DY2_o*xG{2rI&-5FN$b$#&3^0oLE)UE&s&R z9l4IZuldnb(rR znbtI^z?oX4`1;5);x|izQ_rRh9w|gVUwezdF4p%$bI=3D-!$2DcrWH&)({M#7yN zBvz){7>%!%gM4kt?~IGB$;p?eG5$Zov41rkVSD`7SlYiOFK|3KIB=3DXyt*@`Y{DMo- zT;Xm#n7T6cv*sk|a-cA9^Z2oW1;XEA1U)|X%A%|odk2a!Wu@o{-_+CJa&skGW{#zk z5N{~KQ~Ukv=3D|@Mk3>;&su%*RNRE>t{RGQ`hiO~>>QF;2=3D{#?^}T-xi_AOOv>Yw3&6 zS2HK@>ZX&kawDB!3pUdKTLAcPjsCCj!vAt|4?8C(ukt^&2%^bt)Hv!;2=3DQw(%-7T0 zIl>LnkJdeVRLl6E+nzxfBw;Y}l>?Sxk~2N$P#Ce4HvgEP*EBhKok@w&ND)}y^D4`) zaK~q#3?fru2J06p?a0(`KQTO8Zr!;v*5tevlE>!gRHX`vorCJ}`2DThOnKjs-qsw{ zyHP+&zKk%W{-2;yGM(U-HPG6SN4~gC{lkZ_0p87!nKgUbOYD948RwTX`nCqY@_{_t z6Ssm_I}*9^s#|K52d=3D`s?bh`0=3D8Xj|0Gq!QdLt?U%?F5K&~=3D&dPs6G$I@=3D&8{ho-= M z2r-d^Qwy=3D%85S#C359+rMPO_Hw;+eCxiUPHczW#vANI6dlzD~AMHnFHF~jR%?9t8$ zh?Wu5z6gLnPSCd$*sp9UzUTA!W7(jWhF|410o1rhCttpyJ%7=3D1cVBr&v3hCR;Rwl* zuq!j}H@*8|8b_hNcs3-hOHNYu`)wbjj5kO^Q8#^YD~Dn(eH|j~^zx~^8q=3DKn!5OBW z)S)^4j2``0UQK^1+gJLfJ?Kg5SKNy6c?IUh$15M_8pj=3Dy*-o9mO48PQc(B*IWFC(k z>1P4k!O!x`^cZ#x@~~fa2CQPWfOo$=3D&en7THgGi_Rv)v#t!h_htPpH< zE6R4s=3Dj3n8yg_wcJa>In7sbi@@n>_bC1)Sz9$h~-p4C$Z#)0KbMYjA~TCE;bJzt=3DA ztv)EEG**b(D<_;Cz`TN5z@EH1VY`Lv11_3dJ=3DMZU(LP=3Dy2>X!B$Q2T>Ku)1503R~k zb?qkA2@Z>IzI?7@=3Dis}>(h|q5vcAS0xSmggIY)Q!7k1cVI)7?)ll($?XmTLX5s-QN zZQ)P#bX7pXM^nGA{@2LB7b9ZMHA2Uu`9sUvXF*MC2CgR5+}J(K1AnW0V}zZ`R}teE z3Nq{nxFrX=3DU=3DfLLcT|4?o zeZ^C&Qn<{7k0JTfircG9wSs6(mMHO{l(Nc#atq)wOaw!Z*^Wz13;WYqRiz1{)bak3 z+4~A+l+M$VEpfRq4hi=3Dkenk~G)qS8a7WxZN$o3644u92%)Ud>cIn7d5cdRe+kdCnv zNiTM`-U!_0IjiSHOr@^Y&**Z`uS{LQB+0tVo}OFuIT-n&1MIdENbBb+PrM}7!^n|N|u(;Y6JtPBW0FhXo! z8FLlJ$GVOAEb%Dd(OJO?jpYx~-6vC+OHuNNuY|PplV0)L>xQ?aIi4;n3N6j(d znB!4wxR+aw>p@<~jK;F_D%1o##*21f_en;E()%irpHR6k$x|4-x_;*^HY{ZS%d92# z?rouHsjXZ>O57_396)D?|BDd+(mo2-?Ryj&lyz>ekb_zp^n#bSWAC?StR{SrgRa@` ze>&(Qsw-C$3zW_4jtnQD(Gnih2$eyKwuOWE1roG!0rldBop8u!EGqOEFeBZ9h6E?rglI4`nmL&42jYkv`)RF$U*lZw1dp;~1EV7nfJ1p4nKsSOdi=3Dq;d2qQ68#X_x zmu0CYX%UpIz;uH>OA5p(K(_!Afiz``x`$%FCqi}{Oz7IoYuRi;V3}@lMrgsP3^B5l zb=3Dx!0k=3DKo+0)8n5`0yq$+x@xJPw_TWOdjf&*AMMlJ$s@}>RD)Ux&`T%-k&`wR2kHvy;3;44`2ub=3DP+pXqgQ{ZI$ z;9wN#AbGyRoJF;I97MFU`sQLbjw>VP^7zr^d`HVq)PJjpYRC&#$+gbF zPQz>e?(cOX8-R|lf3xpFyXPwhPMmaQl!fz<(@3U`X}PHv;{HAVS5|i2t`8{!i78Hq zWeJ_)2^f3LcpPY>V$KGSm7BU8vAXone5AO{ldTIlvv&J)eVvH0Jc!FWEz04*^}c3m zGfF0&+Oposx5vSwQ>@=3Dd;ySH+|MZ8UQ;qWXs(~y>WF)lCGI;NAOTK{yCIIaX$6qBr zGoc%!j+sO{$F0n+ta5fMI+U!%2oejj0F%D~IoXb}k(6C@uDD9^Q|dEgAY{iQlWR>> z=3Dzj`;iLK}`xZ)^Xaeb!Qj_CzaavmiGJ=3D*9|jE?SaGxYVnR74mOA})EBGN`6n>ik=3D% zWYNWH%Nelp4!HW(kX_>%hv_7OF8;yURULOALni2}B=3DhFu<2q3hRzl$3N|oS@jVBuM z@^Jgrhq-P|Oonq|*=3D$EbQuXY^QNWK}KUPM*zmcaaYy`^Q9l;k&17BIJP=3DMT?jqZWhUX zg;B7-5{crMps%g8N@rC|IJ)IhXo6_Z&MjsAQgk?=3DFb1HhW|mC5@63i=3DPhK%|uH~0q znVE>smv42LRw>!>SG^Zrjzy?3Z@*=3D%dZNVfx1BzIE_DFNvsI+@(5GMQ_0x_;MK&pDv+j^4iAhgHG0Aa3+|I66xWoqHEP8_#LJRhxE% zNA3>LZ4n~_fikHinCIT0lJ;MA8c@)#>Uv%++Z!A+EuD@E0t=3DJX)QI-`d+8kwlI<5w zCk8VqW!gcR%HZSmn;RpnX)M>kC?006nZD1FB_zI?3StNN*1s=3DNlSh-%`k4z$hQTCx z`xH!SqC*wB{F-GFQ8rCc8W#_v2wx_#t|SH^4p%36w;C?)`FN6$L9*L!K=3Dngexy(kk zM|8OXz7&FvS%?Ci_dL-Fg zZ+uv8SrvtVrY%TGCBKre@u(4rDs)-dc{_!>jVy80iY}g%8Q*m*tXgw2Ju@thU4)3e zz*ZeZS8&&-_C02U)6S0wPH zoO(m3v4q61+-8QqG2mu%uc5072riRn*X82qQ34?+2i&BVjHNEdGwWu0=3Dg#LKGAYJz zMlm#Av;5MVa789S#*ne-dn+_~jH&i#-dUWQ>DJ|&gdb99YNh;dVR?0A(PJ7uOd8_E zO?|P~jT`86%e1CH3QF8RBZZl_otMxjCy-8)-OXYvz1H zRMA_7-WJIkO$?8ss4sw}Er+Pio-IiW1kPr5-l2bS(X8b*BA3-I5gz#i6B!-()0huM zOJ?g!K>Fg2)E@Aa-+7=3DP;Jz2B*8-)z+$r_h^;hacLO7}ZF0QdMMq>u6=3D?4)fMGyL^ z9KTl*Rs>fxQqW#Em)AMevL3IaGW^t18ZWzXpG`Y-3ZI_4c@=3DKvGC?ch6K?9VanfUc zIg9m{dIxXcKF7B@cv-%nXY)EKv-%WD;Jj^cdZwu1!{Yj$HpCsBw;F`%S6c5Jp$YwTvP3kKF z+8mKmkd2L44jVHd(|HLcgC87uP7KN>^0MB4R%2xM$>Ss`>`IuZ%rh*UWKox1A ze3DG)(mOswQq+o$*|UU}i(P0KZ#y%FiJ_LU@+L_&%^(36K5jo9{ z9E9ULS2a{f;k#w^_&f+VpR6o7yFgbdcy*UA@6-HpU#hBPyrouo)Vqu)J5v+*iRvXq zV*wfDvKkpYtU;Nym^hmnmA`-IWK-1G6|igLWq|!pIwe0=3DC>i8gC`B3LUlA>R2}XRH zx8FYDWHSRVuJDYX)LIR;Du#xJw+K1bf}Swk|{Dc%2+-+-f#V8w9LRLa!INa9f3ghi0w+OZqqeeWxBiR`cWYgA`Zu^p=3D z=3DxedOIk{Dg1-TXf_vjz*#yEjGcAA$9qAt-hXA*u*6`$DH!QusJhP-Sik+K0-WVChUG7J^D-A)<`w!p|xwl>~y zbuCP5Ipmcvx`}KGx<~>PyJCSN2!wx9!@ndMsqypuKvVerE_APx#ONKRZm8W168w%Z z`!!6xb?}w(8F**1Dt<_2=3Db%i)sZ2?u`ELJg2>at#v*4KO-61i@ifZ4W{i+26i+hWz z4iCFY^WgkfAIPmJWR|b@zcXy+p~Gh#z>B^|um*N|hUFbH-NGozqWCT!4Pk$)NlGfo zYSZnt1n4k~Gn{5#j1)VB1vjesPs8DL-1Ty?_yR{;TgBQVV?CYku1&+_b36X_*`0PM zvyVi@w?aBn!Q0i7P0Xc$UW|0PrVeUAM_v8_zSJ)6R;Eq%(v5b;cX3}u7aNsO2zA&P z{qUR1A4>f1pl`j`?s4~dl@P4?BFv1(Vc204qAp?n0FTWf8yj<4_(ULlLguA&rg$0% zXmzySH8bwB`%zYiI?s?J)=3DKnABfXpd?mRix`gXHobMUO=3DLxa>*H6an1CQv4UtYlaA z__UlQDKkED*X{(~?qsdr^ox3wZHbv-Sae60>Up{<2(GQ91Sl9Y9gx`O3X#VwL&yWH zAY=3DTP2+d7ki(P{;P~#O#-*se3JHb8-_RZfXYmQT0|9Xc6A0>D(qM+~OUG}_Z$N=3D9a zDcil>zFh%OYpmXQql2Qqjn2sFcY2-mIgFcoUZKe^-WMO-41Z;Jjig_p=3DcvGB%Q`;y zXO7#+Un-xnlFYwLIhhWqH2sV%;J_atxfqc+`wBHfg3Sj;R0#XKt{8h#A%D19&W|3B z66LiY)JMF-AlirbcdMmz>8Q7tb-F38bQ_0ayL^aW0|+6^{z7@N@$NFq-j5?X-^=3DgH z9DT{4b$?kMQyGHCulJ$ifxTum%XtyFg^4BHcK%&4T@f_a5G{`g`s0gg)4Dw#_fF!? z3NFJ1y~W0$TdllZqQaVndPhF(cHhbcj8a+2HTD3pAc-h?Y$7X}nGx!vem)`XO(o zh-11>qLSdgqjtVGIZ%-%?ZG4*xD7+T>)V zt<`*Pu@HnrEyfIA=3DrCP*UT~tX#&6ng_AMZiT8#6r9|;IC@6FFbaY9j;ITCZpw)EL{ z$4od}Th+FSw3G`M)u)cyXbr?W^O(c4<26}0cE-3la6&W1F}m=3D1q;cv!h{k&uh&A1O zHi{AKq;k{G9L7)WW~2!Zbo{qb*zu z7iR++d*cK3Nc^2ehdFKol$Hpm{?bOC_1gg#M7CkQo~3^3p)wwtGCZA=3D7!PhPo}WKV z-Z@SPcOnB7H3{AG*;Z)uy}rgGBta{1)73nt{UFpmZs$t+)d zqI*d$3u>in(m($y+xgP#Xt@IexZp>E@H-OWIU`)|u=3Di6a)O-XzP-cYZDK7=3Df`60L$ zbS~MO+*xBUNEk84$k?Bq!qK%8635ckVxz1pRjETb2s9%QQdvXjh9~V1DC_G@JaW%Z`Iep$M@kt}@CB=3DK_#9$pff|7qg*rAskR+L|Toh)@MYVWGvyQx2RE7*fEzCJS zhO8!k3UOgs$%s5&fJQ3xEI++hk(E9~`EuaS1*^=3DPem)10ff6%}XfGr( zNey7IE9l-8Kg*_Ms&38Pd0ktu)%n`CgLyvwkyZv#C_xt@_RQmz zJf9TpWnL}B{Q+?Tyu#)@b#?>w$>q(vcps{qI^!*;z3xle>k5eHY8WXqUliaP5;|7N zT))L0f$hL4pefkSzSN3J><;FNTopa-G+_lrX+5g-n$6=3DdqZx*; z02!8sueg9D2d3^<_8oA#(c2xDLdA>q675r(37!?(!b$KAt@~*h_ab-Y3xp%)HKI@9 z{-UCQ9uxnYChNm|IQi@WaEK1`0nv)=3D_8h^VfI@@Qj?4AO+(b!MmWHLK-TpT5*}vTs zQdZqH1M_;c!=3D#gH@YQ@88vNRSz`vBFnBDA$378Gg}WzZ#mN1Mu(gKZ&w?~cVi$YlP` zg@`YL`pZ^+r7`|wQpTcpUnSEq109RNdcGLJzmUH|pHk?mzV$0UeeT<)CTGV7T*4mU zWYLx4Nc-h;><__Xt8Rqz_5`BmpWa+e_+#V2KxeDoN`e?t!&TC;2;wiTK(g~Yat3LY zSxik)HxJ+<)mF|SgAikBQY6K(VhOUBGNd92gXyR5;C|Ph20{b{yCgk84x%n~lb5%@ z8Dxa@qr`-dv|8@YzjM#BD3Wt1@-c(Z#M-Ao!+Zr25F@@xu+sAVYCr^HbxD`>TY50+Lg77x zbpx>@8*JHN$xHac84VTlWA_5QS~F_Fi>E)fccnG$qW{lI(m21XbMq zFjmPr(x&7LGGn^dBFBCQA(?J_BR^;RO9Vjc*x*rzkp0BphZ_TL(LX{~6WkPd9{wi7n_Py8HEWL9y^Q%7Ra-hz?A>?it0pC0CFag-IDzm**|72Pk z(qN;JmtssXa|1r*p|SSRD@Ca&wiBcT(w5Ow2dp@}+~>dAk{e@I&k?XRrMuN_!SkX2 zkN{^p5_qdhv8?eUQ7VQv!G(o|CM7Pigj(mM&+Jzo&b=3D7Faj@aVuW0EJxmVQ(XXQwR zvxKXg#@&Hg(9qn)D&^qmxE9)FqX%B;xa4)hRlLTJ0bj!%N`GvKy2{GZqyswJnys^F zPEdL3RA+60=3Df1LL6K9nS!QhND{CrSv;8*cbr1!Xj+{0{Jf5^ls4=3DX09-Ihnp5ng81 zXUhe?r)Q>vL0xpB%ip8RInaR`2>5eOocV!uky<{l7PZn8QE-=3DdR@%Bv%V=3DPGjswynWJ!dSy>-d@?{M%><36d zVH?NzadFQMP)n?o^dmAGBWK0`NeA&V>wj_pCk<96yA49B4ogyE!fX}6mZP3U9rK#H z5uUGVrIeJgLJ0G(l(7kQqdvWnF+-{~&*LQ8it=3DqUQXURD+tGZ?zl1bri_@O{>=3DV9% zrN9^o-4HG~@diTJWe}^4sP}^C_|fOS&5>*te6Q!hqcCyW@^zbgXal#H7zM}RWD8QvPBU+tQ)%xj zVv`3qoIwO~%Gp4V&Sd4*D;O~jSVU>v+$BK|J496x1Q*aACl?9_-lp zNSY3IJi4qZC|cCVk{(+W77n=3De6;(voGtZvx{Qn~`i6wVdOP)w7nrSlm=3D3U3lkBO%5 zLrTg~k+w0&01OnZrYLznyu8Br-}Q>F05t<3IAE#yW7_pHL$`2YLxY2tf<_90=3DZ&w4 z1EsUW$Mivn$Ujt%In5Ec|!;UFTr7f4fkwx)UVHYf~wwAtdfcgf5unDujr`(IDNG z6j~%iL2NY=3D(j;M|U#WcgNu!ig#lC80n)iTWDv|B`!>IPi%tqf<(82~~OPr_@yJoa{ zrfAux9QEPir1l66-cGp$Gje^J7aKztRZVqwZmz*+I<5eOt+%t*lS*zkHM8uza^Ys=3D z_z{bLD@O>b@g?e_RyP$kXlP$9onv&&)xs5`HXIdD0+3t&c@a+dU~f(bOzS1Zs?6tZ znflgz6!(+&9i8Xlm#zky*PU{mvIU-@<4rL}YT}Z4wwj?3^$~@J-6(QI+{V9ZWr(2r z5Y~MngOQF-$i)r*?9L-qx?NE*Hw3QopV>(L>6=3Db6-k500uJeTvi{U$CC(eF%+xgB+ z{`f_^xF>ihY42dVtp~v8!yV{h_v!iahk~q6wG*F9GHb!?)a<&5{l~(AMv5jpU5~is z?RGNk{osxYZ1Zibi$nU7yPo6*{mhOG>!4^QC=3D>=3DypFil0Y9SMZV2C{r=3Dso**jIqAM z{|5XADMZN$wTv-8j@)Gn#l8k{;9b*uGxSQe&D8>R*?^ss*jc7%Ko3wn$)|EO;gly{ zgr*yw84w>>rXcd*;4thbe) zBz)hdglSRGYA3+x^NjIz;|1Y{hV|s9f2YFTfOB+lUDY>rpK7{McS+o35HV@8U?i0A zT8ka`9^I@hs`)DSg&uMf7`o!lPA(Z&)|%WCe$`UXw;0arZvGQ-TiNofdA6o zc=3DjJ%0*h0FRzCU-jy35QW!KY<&uUv-h?xI)!a4JL!dG{k_gUE3MFZ-$Ep*Mcfda>8 znfwow4dhOjP`jZ!rTm3wR+FE7lm`N{uUnkJ)1e4B2qI1Lp=3DHKIi|Lc~xp8whu5o_z=3D&un6A!Hi<$VzAp{)JFf@R zenkT{zS?Bc}%A9at zBnyg&UhNIY0MIZ`uTc zT{x;#}yiFEL{9r|78ALUikjfz`@e#5v`hN5Wq3@ama=3DT?I zueEVt%TCZL8Ac-~Z>PspKLX9;Yws!MmdXXA8ljG#_2OY*6>JxBY@uTaif^mR>Ije+ ziTm|R%ve>>CnTra#YO)!o=3D7OM0KPn`4jmRoQ%uJ7Tc$L#xu5_Y4!mpO>s1q2^uO`K zf+9R;3;FuAdx@J;wVe}%99IQ!gVi|yZMwe6?7Lc8^y}&PHDH%ZH-SW-gGb};*d7l2 zdGc`}B6PJry&lc$a{A3s5qy!o;7F|Qmx}PwUR0P-$NlXzEWlIa%gG zf?mX)Q&Y2Z4tMKK?NIkUa1o@ydGsZ6IgRx@DREu2Ss6QV$>rV>thLp;ZD7aO&eVT~ zTF5eZZkN)K9|F)N(e&UktHv{U6-Ix%V+eOY?2824-}p4vf=3DqCIyz!)B{J21^CLL88 z=3Dr9ae@lw0v(Dbf^@d3MTtVWrER$Nl8+by7>xRxT@&_djJr3AfXb^yjii_=3DYKx^7dc~oj-TdfDab!%az^Iy?4^SE+&V*rGZAeOoYbYQ8Sq- zy6O$lZVD2y`P%g14_KOfbqKa}c=3D=3D1FQwp0zH^+uSBhOa5GQv*TkQRj(CAD(wm+;|9 zOMKv*iQD4BQdwDgQlxNq_cRv9cu%~G&Bu*~3{IZQiv-POzomCv&>+jp|>$G^Hv1wZmaVAZaS=3DYw-J9-7hiIi)MtT#2w47gUX8W(SdW7d zHu%onx>vFSRmviF8zFL+%;~vsuM22hw0SKm9EqihtEVHSQ|O%UA`lU*q%^x_2_~bN zjwjo}@+S()Y4Wx7&)varog>d~`Q3jsPO&z2c6O$X-+%Dp zoBj`pQ(yGb2ZVlfv!7~!@Qd69@80R0>LY1!s>H*LRl(|XtfwJZmvobh>-K+b^*0OWbR)q|pD){f?2G@w>!JBbn;alDF*_Rc zkZPzLj|_PAFBS+!oJQt0J4gz#h?D1iHxflk3cj=3DRzWL+URQmYvl82k!(2hid&LRvU zh}sQ7WaFY)F7ZJ`G>h?ceknv8l=3D0X^|J|zo1}!N29u3-MP$g&K!>sSp97i}Od=3D)8| z*{SWrm>3y>PpYcOP@Ni1pd@jk0bx0N>C`*98Q<3n2#K`LOFZX_h^KQ`7W>~7-iVyg zK`B3+I$A`?H8kb>Z3V&yQg?T;bf0W!Cq{Qz4SDL%Q@LdSe`u|>j8W!4W0R^hI4;;@hs2dj}0m0N;`6VU%kl$L1YYh_;wd{AmrmVQrb# znfkD_d^vml!TnSzP%khivRN8Y03A=3DfShE`KAU3JJ2&L!X9dwV zvoyl_kiS5ZTH(3Jzs9Wm$xNnCXB-lSf&=3DKZ3qImj(;mX5GNu)f7t&ZoO&f@Os zq)G08+Iu-970@aMr~7{Yo5HkqPqFC{?J!cbOT-=3DV|t&QDfLVI0q7T0)!XY&{JR=3Drcg0+z=3D3N{ai)N6j^gbb`Z1$BVPOakZ$)L6N@d4`Z;KHkSixsG1asf_ zP(n^~2R|r^fF}C=3DpO}>@^$uh7xn@}`cA-3jg3PzD(&Vk2p!V^5JvX{fduJmN4v!ww zmFHVza%Ih~rp}w&`SAcS(XKAKUf&2-heZkJGj^qknlii`^uQt%#8rNiac2udv*V4p zv9|Nn`;Q3aYRAJx)j^Q6ze!#25>1DDI6Sl9UTEkxf>{C;D(hZb?tx8k*Ku<(MyW~?z1~yb>NCwdHF)Hf5UQN_Sna(Xt-`#Al-|{SdrVIjWw_orl&RkckZaS=3D zVk^5Bl1sGM$leg!gXdnuP<&zmFMchAwVLFZ#>tJHA3;WYR@=3DhtAuVjj@fQsjBic*( z>pN$1_xm2%>=3DzeIlfKe%&&?XQ=3DwjIFT72;Ul zcaO=3D=3D*|yUm>rqVF2I#S0YF_}DulFW;M!EG-aZH+kHib8qtNxFLy`0#lQlI(kUvb&5 z?3?FTKeH=3DY&Nn5c;O(916Hc!@dtUQgm#FreATRD_cK&VNu}Q0@deyt_)KA+U7PIQ2 zeSg%&=3DU1*@`mQjkE>G#z_Z1gE{8s+nqk6CT-`q?8m)B+)?r?VHJ4WDUXu0Au zhIzXAx3}fy>CTgx{N#p3=3DDJ|cnvJ%9zgz~cmHaQif8KY+^{b!Nx*QY_c&DGYyD!VR z#_{E=3D!z=3DVxWLuzoMxnH&dtY+`BZiSl&vW@%aAss;@tycbBb=3D z?pDiP{WRy+mXm9zhIOCUdO5xBQni+Ay!NY_uRrbxFf0)f`Tp7cw{p;}xG9D9!ZE@3 zw>xO=3DyCQGd`)jXiQyZH_?5^EEpUu7-wJ$L0gdsCK-xlu+$4|vis%vLA+0+e8pUbn} zIsd<^ch&CK!|b1vRz73-P}` z6?<3vFMV;|3FHb-`|VqRJ9zS^?^qV!<&@yeE};_Z`ojB{@xk{z??TIi4!v*Kb5FnZ z<@wEV@8@d>{#_c{@2A#qTH)ofoQb~|>Ad+a{4I0W=3DZ5($^>&u-vtGNL{F}b*_$xow z!X+DRUfo#oU9s|4y4d%Bg|=3D&F&7Zu?n)yfXzok}IF0t{SJ`{@-et+=3D%^^^O{YxDv=3D z|NohMPxwUL&zrw*E6m^MSET>%=3Dc0AJOYFA??Kdu5^v+TJ@8(H+iuU_Ity2s1JW@N) z^PYVOFg}C=3D=3DlXDId-QS?EZDE3dK8#g{|o>8FX;IG`u~%cfhA9ToaevKQ-Al*dSa(? z|NqRN@9p17UHvns#m-&i=3DS-KqVZceqJ^X8ae^)$yqEP8&y6W|VKfj8v{Qs0Q^Sx)? zw}0C%JU_hk@j3Qi>6*v0r`GLs|2t&jM;YZ?0(S1y=3DzIix?o@jNS4u8%H!)6;H29WK48y}r~`MF_$3GK3RO~CwCqv8 zYoy47Iq!j|{9Kh&cTiq6fA`%>U0k3#*0&$L=3D?k*iW`f~=3D`DL$Wn}50>@fCPt5re0z KpUXO@geCxLdSVy=3D literal 0 HcmV?d00001 diff --git a/documentation/test-manual/figures/test-manual-title.png b/do= cumentation/test-manual/figures/test-manual-title.png new file mode 100644 index 0000000000000000000000000000000000000000..c709cb9d09cfa38b0909cdf70= b459946e3b4615b GIT binary patch literal 15382 zcmch;Ra9J0^e#vYPtZW{;O-VYcyM=3DV2sG}l4GA7BxVyW%h9JS+rEzcEq3PSn@4x0@ z*35mGb?2cePWL%gyLN5)>f3d~loh4lq7tGaARxSz`68i;fba?f{{JuJm+-%c^+XHs z=3D9ROUj5;zh^74k#I{cErMN->E&ECw#-Pj3;U~Xq`3uJaSaRLJEoGt8KPG5BiBOrW4 zkdY8o_eh7Vy6fW(fRWBx`X^%y3x3c^zQAJg|9VR3&Be|h#?HQT-*=3DVyAulY9o`$_} z0h`;+#AX2#t^fPWA6#f3J8qqz_pgN9s+f@UDM4$ju;z@k<%|}Cl+)3Xv076Of){XU z5V#kDS^l>ED&qC8;cfTpN$kJZI1+My8+lCTzqhf8{(t;Y_hZZ#bc5Fh|Mjz~s-f|o zU66*sEk6e>mr-9F|L<5{)u9{g!7Sg$a_mvym;NZZ)PKLG2~GUlB!AR}E0ql)=3D8L+a zVWshI#d70OJzF<=3DT{~RBE8G5+`yfflz@OXsmn3GLZqNq|5&OH1y|;RPkJNN*%D~M2 z{{C@IKTm#kl}v0Rp{I_fHJ9q9Q(ctHiwkG?=3Di*Y}x@1%jmTedOju;bRNi6pf0PV6#aqpH!%a2n3O`gpvj4k0lt*;KdoIIUMh*?*gbbz;)w5}Oq=3D}yjHldx zxUPUdnTmlai5`s=3D-p2s;dW2#Wji(fCfLga^4yKh@eN}#sz_r<57mFjc37_T_WE>*U z34qo4sI49sq#YcD({H@tPIICe>E%gB7ZhsiY7+~1bdQY4XI}h%xYlceNsewEZaP_jN#Dxl z3he8_S9+p}7M~^C4%Bucov>J(Get2Zo#|^#z^w;>2@lHbmE>TPQwcIL@g1}`CDC%L z7<;a1~)PkgU`(o=3DXUFt5XEF*V?3rwQH#A>S}5hUiY}* z{j;xEcg|b(7xT;R5Y@44OzQU5av>+#Hcz$ZrO{c}fyB|ns9N^{H+R{6HEneX-y7TG zh4Yyat+A}x;{A-K)9x3CyY?c`^-AH7YRJhnFTpN67@;&EWnS_&(_LnsJi@f znja2MaoP;>_{V=3Dv_@V!~YiT~cJp~m0py{Ug&ei$aoN>F#9%wcM2ziZU>6HC(h~kY5 z%+Sg^#$gwoJG!Z25@EzV%o9&pV5?dUdb}`UCH<)KVH53#S&^%6}3TIWxG?K?#YSSKk%aDDU z<8YD{X$xyL{KWmAfFlKU5XaB91_sOFJ`bo%Rj{nh3R|bBs3-$r7g^oZr-ryZ@ub`7 zl2?{2ZVMBNJAO`DkHwqL-l?q)5{Ul^6}B{w&1;NJZaVj!cAGu`DTpXrUe|TY%nM%_ zXlXgQLpj%E<>^ZGFE|CeoNPPZ9|Z>^1`)`KR6S;H)GehxudJ-(CP0h1j8_(-IO$uQ zS*z6U^8}X%xSGZ`*td^g{`Y9_hC^}Fw3<27D99$bRH?qr>AI9AL^-s3Nj`&1=3DDAfp z$(oQ)fX{V9&F^@D+z)cJwQBr4FP&_cEET+rcM6)C8d1+HQAzz5G>wcR4F8B7IN*ov}3@uJ|14!g zfl$PdoLX!&q5vo4ohq+A(3I_zkD5E7mB8=3DpmsrdTU(0WOXDTLK^yc#(%}wr^5P>_T zko)(UM;sJ$-53%73E{yg{P39^7?4UJx4XMLGg_3$1WfLr!V6Doze45i`;`N|;XQO# zdHwR2H<@3qqo;s5OmS%%ocmRp!}Qtr67SHA3;-aBFLwTKG2d8QGu|KRa29fI%ZG}{ zu15SIA@JJ_bZRsih{hnml2+i80Ni<>MnPsqaf%eB(W5PG)H@6(7ccQCf(N*`xY%&z zsjvqD`1tt4fV#!wUjc}Nywun-y=3DTd(OS(Hk@@YcuzfXxu=3DmlUaLk<#9qd`;M&S7}$ zL*W1G$dp7u<^}Z24||PdJWshl_jeWrxTpd#V-rD)(VZ4J}!>9D&J?z;Xs=3D#V(CO;8y z%5;&r88AbkRl5vVgql>Oa6Ex?_FsN7@|gKT;qv*nBY)% z4>8aJp3%C`iYHY=3DNaUPmXJ@wxIU*Mu`0Cx8%E}nU5mV&xc#UIQ*+0D@0^JEyr1QKo zXqo;pHq}G%H+BJGU?eTyYu6Cz$tSsqS0jJJBO9AlJ-g@k%~416s>de-1 zf8wlh*uioGJVO3W7Rcd!vVtIZkbIg6!%;U|zr0tYRs5pJpOx-nNDdE)RJ0jce72=3Dx zo9kz1@XUz{52~iS5eM+6_eynShkma@!M(5ZP!o_J{^nrxqu)we!~%}0ot8G=3DUSeMT zg$5rccFMF$a;B0PwLVm{Wk;#VXTqiPcy0S=3Dz2`3SOagR=3Dpeb$bcF3M4EXF9FHVhwgcK+% z_3rYY5N4-0A{`waZw*lQ5&p|gMa|6~OX;2}0hFDOR&QNntl7-|2QwJ`&8FE%d6lm~ z=3DOdOlI5>xGk&%&<14{AsrRg6vbanZGbHw+%=3D42o?m07P%tluUNw`lN$a5~fb^z>MV z#}scL6%l#wbvpRbkDY0woj;|&Q4+b_FlW>Y`aV#o zFx`#-qoJwqFsji;wQqPM9ySoIqpQoq%^jJ^6-3A|DAKku4`@1BSiDL7yR zG0!SesRnZAfXcKAz!o?h`$rVtf9~`P%B%qM+LM>^r+X3q3#~%SA*by-`|E5r`^v5a zH_!&(xh=3DG?)b5>&S<^!^Pftm6w6jtCGJ(gK%)Wj8HuxN})ewV%*aKw)Fp*n` z8Rza|pAsWgYU%}2P{<%mM_ZfDRJ&ty8tyS$N9z2fmeU?sK*;7%DJUhR;-)QuYQsbi5(L$Ve20#d5ebJuZNNu&+Gd=3DIWuCQzk_fC6h1cNF!pY zFt-j4P!}ncT5>R#^xR!0BS^8gtnuT zYUf@olp?2avol(StZcLdRuO{iR8=3D28=3D>1LzWG;BNZ&qEhFdl4$-q%-Rz{RgAHD?Mt z{DDDLc--oC>IBkI36IAV^VnTGVG$2_wL1``cxYRw6Gjbm=3Dj5STd7|xDHF?l=3DWJ-B%N%5OekenZZ%i(^iT?_xsrTU$sU8JpRz zNCCA(l(*`E77Ulk;L!HkNKF$Y8%-Ly&;d{T|1L|7-4!&r@K#1yn9I!-qccqv;#vbYs{49BdGJ9sPODTq zG$~SUw-@JF!|qVcayPNcY^I%aULMAVqFv!;&F4iW^HKDtKW`bT)*ZAlpPE@TQW*K} zUNS}zTPshl52jv%p-xuR z@|bQR+ixikSMam_0q+`C4+i=3DH&Ovtt^O#7wfn0`wMvBWBBxX$yF|`JHKuGYxmd7wl4W3n(hbAyJFZCseud&DUZnx_zH(;B||LEfJJ8MHSk z@@>;v+`vdAUNT_`J%jIe@1lI|d(e18rrJ#ZMuvrT>1*@X+?gVZj? zE151DfmmW+^1H@okZ*PDiGPNu<&M;etYJWhil9@Fhmz9~Pb36?>BKn@&!rH2Y318m zgsxBT+#9K;ciXhz>64gCpK8bA0fd^;I%gt}_TTu8gBGZ8#I*IhHq@e6F0zvve0;^0 zjQ~V4c2j*4OUhA$@q1)?yVO||!)}CZW_vc?OTs6Ht6FP|Z_?q*RS}0r1?Z4asy%mc zEtHVE;w6ebLf-JK)W6pp8@Oy@%6J|i3TN#k+`symv{p*r>d4QRmTH7`v1P-mCu{5( z^=3Dh8^EPK0nIZ;_G1*PaeFQJ3H<)~(&86{duN}c)%J+fVDeUTF#8IkACgk`xE7Wi1N z84kHWU?MBXA+`gn5DaQGGm#S}nd$mBF}fb?5MfCi-kT;0W1FpVWy{f-{WV0wM3 zKnMY!2Qskj%4^ORn&kxt@9LJ$>>oj=3D#p6f@>COtH`9|R{%A^#!WB$4zBC-m zv+4{7>LbPQ@E9qts$~Fro+^lTE6k^p*X20(7XFiNi5Rn*9gV10Ldpqu1nTRcB8xJ) zpPl6IhOkJNr(6RmTd2l$98aw}T(UEjgy3<#w7%7aV)*OOQ;u(CMn~S^=3DnGD%DAi-m zj}-~G(+5yY;a?|FrFv{}+27u6JF!z2@R#Vf_pSwEQWP`&DU2?=3DyJ^yztmtX*z6d)| za#P1ZTw-uY3dN{;(7(K&&NPu$tT!7v`GAL8{($o5q0G!$Qj~@Ho3%Yng|(H+2<^^Q zBUQdoWoe5xIY_aqh!g>VR7cm)Vrhoc3b@dBPcKW!jz?Z5U1lNjMx&tb3Kv63j(e18 zxqopk7MHiNsD4`=3D71$dn8JgXi{=3DAKv7#`k|x|7=3D`srP2C#n8BG#KoLAnDWK1n?q8o z@+K?PLo3TBdF=3DC=3D25Siav%a%i(3d%-s{aOEA`LatQA`u*)uQL~&|Uc;w=3D<`88jw*;BIfKwRmoXP1ENPIpAPMos~x{&ggWv9kg$ z9ssxg$1=3D?!sxI!$Z}$1Yt;KpxBN8X{=3Dlk#?@a>lSOvCTGJvicrcP>x2zZq#6d5s}l zQuQxiu#j*uF=3Db}-*nEfzt0>G<;@HhZEO6Js5H7e2mM<*|(Q~Qj<~QD4hi>ODU<^EZ zmK<^BZfF-{_{Z;;rv4Zvw2-H>5%`TLc#VK?R?K!h?vPlriH;)^@Wob-_B{&j$pZ)X zUM&iFprnq*`7aP*U#kUYgT^`wYAV-`D{CJgpAF0S(o$=3D5h+KAujCpXjwoOCKkG0Z4 z4t9;^m!tT8tv+ggqqIFokbg;cxOx(Md1!7W5OWBZz}T`wTS`5eA}sCD`hec;#x|09 zxipC&4RdHey1HDHVj83r)rh*nt)DgVEnIwdrN0pFdo;^K{So5GKrTh~B4H-A4kV$B zzaHm8;L$>$=3D8ezmC2I@{`0Q$E2~=3D_AWbWhmbN(wC2R!#HEVHe?Sq7g`)T*poO3{9n z>i~HAMGo`5CYk_ugPi?;mVlUvIqZIh%FdT1ebDWAQ)3xSpNmWHqlG zr_RM_?h@i_{Le_ihC>T0u7wyK9j!@T^|F$C-|L0l)$pLt^yAMe7+TaudaaS}1s!nu z?b0c3>h*f!lu1y1{Mw6m44OP1Ultftn0_Rp+8keLC^QxZHPYc54v#_V=3D_cjnjEvlWTUP&rd#YI~EhxV7v#Pm)XAY*VyKRp z*5mecn?{_}vSv_v<~KM;u^VlX8Hf5fOfK0vp2D`fTpI4bM-uJi=3D-n!H(Cm@G9o#rL zDiL|v)?q~OA$7Ws)}7^(u0E}OrRn#0oJSh6;#}Y+QoF!PqSF>0FB^*!0&~dK6q&cC z=3D};Sa{;ZDqp6yssMe06w=3DS*5H2Htooh%tFxYHExl=3D*_o!4I|2;CM%}^>9@H;5h+~u zj~nZSn>T}NQ-$sCMPPD}4;*j9(pR1*TJ5sDBL9KfR&NKn`wJx&z0B%{c)mAONgwiZZD z<^+5&d!>l^d^M2wvSgHMeq}uMr<3dtDp4+mEw996t3D z$OG;pBiHkOy0K#33sTlPCu_C(31n<<)0<6V^ENZCnCfc6&6wJ2OuR|BHGF7=3Dpo*e_$0sIcKI_tepCxHUhSUGyq=3D4vNEHUJb1>wi8zOq-`x^WXTA}8BvDlbYA zy+851AX+0MGcVm_?#p-`A$;x>8nSssXlG>O;irI_Q}B{Vz*i$~??dM$#8{N@N=3D7G0 zKEiKkUXkAXRI-!Cj2l%+B~{ydi{D`Ij0Hh)wIQA)|W z8JjSgwKjD-nz07KH?-?mrs$N87>>3ySOepep{&v)^|_{xnWGf2EE!WwUVmlv>+hHN(QnC4OP@K=3D}Q2C~y%_^|g5uj3dF)MowxHFUZAl6(Z+nRwUkqdeT9NawjK-*Z3k z-A(T|8Mt3d5Db4IeljN~g=3DZR4c;#e2QJqTNt6M*sEd4b+_n_udPtaQ`SSsK!2bmbl z$L+u=3D5`H+H>heY3{q#tj`iiSFoS>2)P94acagw=3D+dbs@_uVdv^=3DRtW7u5m!E{m0G; zp)4^^yh$mR(I}P(wFGysXEqk|!2!f3Ci6GpMW&R+ffsS`GGewgoc{It9hS~%4+KPJ z=3DD~?fBSAO|>GI8X$H;t~u>!ln_QLn&gOGs9%!PxV#&{Z>0A;h%9wved<&W}@jQpQE zC7kN{OZomk7cu|;`tfgFHA2^)vh)PJNdLb3}) zI&6Q+9{f&dabeUXO^S)ZyDZw3_x3BUDR2+ueqWP)B*6BT|9ZH6(rk$l%l}~^R6_xIC(wtL5p@z0LWIxeY{O|k=3DcWvFD8%t!eg?RB|I#8E zvQa1d5ynx8SHFJ@uiEoZHFQ!(*&hdH3F#FKoqi#DUi7{3z@ZL0z24m-Un>P!rMB@C z{I1_k%#8>vjH+|h>~h30Jg7LBALek~k1kQo#^-Nc-do#P)DK5MAjcgVWnkZZ_bVVKO74e(qxeuZQg6MRfd7Q?s3*BfFRDi)>o9R8Kh$ z(Xq(1WfmZb9P3c2ja58m5BKHyp%4qhQ$Y<>EZJU$CJPMuW=3DM<>49R{r#v|&srMM35C2i zICsR=3D53rV~`cs3ICh%x`gU_|M+MV|9@tnSE2N)@!>C$x|YvJcHn%2chY6q58Zrt1me7Pb0;7q zW)op?QMKpf)-rGEB6EE_|B!U4Q7;mLnZkBC6{Pg8h_p>@`6DIXO=3DVt)R0mAkBLDpk zMJ$RPG_C$ZTlWc)s!PA2y)Yp04&-My^Wib7GO1GkaJ8z|j@853o%N_x0BB1zdLQyb zG7su8)KJ`!}gXpNo^ajB^}pXuzv-kDp!4MSncI)wG1Oum$#{6<$&I=3D6;C~z#eQN zGp8PZvveB*8qR}_;5Kcqjo$Crjj?*K8u|jCwNyH0u9d>9PJB8+up-MJM)GqQ9=3DIXXuf1*DTvf^4g)?ZLp105Z#HaY&cGIyr zHFHCg;{TS$($?xHwiw~_v+tHkQQ6L*f(A8=3D`1vslt*Hql>>WcHf+?%pUA z5znPTof)Rop4@U6N4usnyNxZ@>R4i3HpFiNAux8^`x8JQb4_bE7xqLP-$SkPFiVl+ zGrYR2HgUe{$RWDZSkac0$)J%8p|Z98^XJR>pD!@8rrSqY+T-A7xq>dB)I))5o)m%+ z8)DWDbJ?gc=3DS3v+2Lvgh+GBO;?dA9!vPK7eW$XRSvL9WDtV;!qbQ!0WfnMRG$%q+~ zHZf^M0ajzgYm~P#czxVv3y4`mp)_IQ*F`B}?ZGe8zbN)prRNrtYj{WHmnX_NBpAtx#z3Vsv(k z48s`z!@P}b@l55-oEiWyf{P_4QG(P6KlLy~dI5gi+#i%mNYZIyakbUZkq+zy<(fa5 z?an^#4Al`z&JJ)aQNFJFY|ii?xmu2e z@Zra|JnH|8RZDC6A6CuEjB2*arwPWYF`8JLID(_rv@?4h=3Dn>YpGX)z$OFcSizj4;( z`)!Uo-r-TI`~2pd_(9%1x5JNDL*tbcGL{Be$E>`9(IWTZjzrSFP6BXrAW#3W+e!pC9>dNR# zFHsmB2b0aNy5@4PR{ou-=3D%d&FP;Ri89f40b9V*Y{b5lJ6O>-xmPBp$g9tS@ZER{gG z8B4sU+Io-O zby;{;Ef24EIBz+txG@eZa1q0-FQPOiRD1F|GPp=3D<{t||cXvQXoLEq&Y7O8I)A%MhE zTsiHQ2n;tzUaZ~KmN;bzzov%=3DHNV&9%y|mlQ$4qNW+GbAnS8%z5>cWu+nuLQj)DD} z$8#m|dTJI(htCCT5Qd|h@A#;pw+KpYzOxwoI}$-iC*fwVRi$x(vrU}S%e)yho(}eG zZkeG&RjkofdDJz>X7b^w8@7G6rJvj`o^E`GO03$??Zj4^4{m1c4fN*28eO{J{zMlX zk@z}j@Dz}9?RA^-G?E~>*2pA=3D9Cme^LbxG(6#43cM=3DIX)X5cXgaQ|nRsnwGuZE0dz zYE%Xa2&FBEyx$Cax)olz1Lg?kTqQXtRE;_B;xsOI=3DVYN%NhoXk@+9y_F50e7D7}Sk z?q{Dt$rg_N*F0C%k?GuH?0atn!u?6bOuh0gvfW^)odSGEkf<9aCn!j zX&4@gK68mX=3DDZBziNM_C66vkQ!Qs`u%Kb6q4zNg*9l1t3B!YaUu`TX2>9JqVDL-FB z5_sRL;KA;kxgan-a3K2bd^J2|z}F8-N-f>Dcn2_Vzl#!RP+oOkL{)TW_Xpw)XjLfiOSYREQPt&~`(`&B$dDKPlb3Inoyc{jy%hrHaHz$Gcg}?C!2P#FEEf*&q z?q3w!1B-@@@hq=3D1x4K=3D?1nKw=3D8>!>#F$iV!I4+8<-zSf!PmRYt(?;2DT(50@i^qDY znx>^v7JXuSp0RCx6b+c%2HPhMzxQ6zLG=3D}gMMc@T&Nw?oGTFP=3D)^YM2GyKK~Hn<9| z>k%y*$E_b9sl9p^RoL)tbkA;vPGJzU(6Es8=3Dop*pR}Lp6@Ti^opfX~ZHvoJaUw}IGDNN{6>d$?g-ZS@=3D!sOxw5OvH>hG`~{(!%Wl_PM8% z#hqLgGsp<&y$&LJ5*m+Hb80wVLDJ}T%6CWoHxa~wrY1R0yEjuuI!*Z3qziMr%s3j8 zsmc@@*YoqG_Q&?DH^z&L4=3DoG#{Mv4X%lUe>ioIR=3D#+u_{d6enp{B+4sZ#l|WC3#;8 zNr(6Dd|)Ol1v-iE@wiY!QkzC6%yc{iPsWAib)?tr4xtx(V;v>TZPlAuHcNlBPJcn4 zTD?`0S-m8fH@ zUrAYE0fRN#0E3%9H@%1UbWeHs_3sBkbQDRA7TV>aG3Pvr4Cc_%k}F8FC=3DRWkQrLTs z4j;scT))P;Dr;6%#RHnME)_~0i{C!?glBToDv=3D1E$tIoI#fvV1@n`#G6U4huViY#LM`+C11B^>>B`Tm!VF8cx1 zmAwJyh17vIa3JbecC{m`_=3DqV_o+;S<3E&QW5i9?Whsp^SACc0t017D`l`67Iu>Tql zl9ffA0Jh7;5IW@Rnf=3DMEH9%k5c25DZ%G5fw=3DmhK#Rnfuev#wh>d{j8UV$T9VTjA_DR+q~m-ZE>ud8aif)-^SVrMaEN5zA-httzl-^?0;# zgVQu}w{v0~QLGAS@Xde_yLvs9ZMijM37mI0gojnbMjnWnxavQ7dT-j9R%WF%)jxAI zSTY|cLSqvRdId-!eN=3D7k*wl|9{XO$rb=3DKaIrzl_tlrV?YoaWSJ6IExvJoXxhWT%$K zG*nwS=3Dwo4DXzfjYm0zGYhBrj)5G0VOB3i-2%Fjr)x z!vqse68HPU&Zp5|%t1?sM(lSwPgHxwN@&Gx=3D$azkoR(Us2rc;w|ASVSF;Ii;KG zoJbg5Er^Gh?EbjCv3RK=3Da!}r&V>~M2nm`eV$ulou{1ETY(O8f_432&oRd4~){=3Dutx zuD~bDZTjhFB8#i!QfaN5V>{AA3#8O%Axztm3vgCT*~P^vP$U#Sz{DCk$7sRaDhOH4 zLs>fZWr`VM*Rdz8bm4E{DJ1-R0?rL}Cv@DhUkqLMA3WM$q4nc)=3D@}uvAC#l|2 zG*pDEQz=3D_$%8#}n`V|`|SzM}~zK}OCyWTi%z0X!Fr^~A@*C5PVUX#Yd+}J7q6=3Du`! z`z#kEK$n38c{ZA|*}HLEn`_5`uC@NGy3)yr8&r35mhJ1&)Htw>Xq7^kz!K;FY{N*z z=3D{3z%ss?tm@&`h8VLe-QW_=3DD28?*UNRkD)fKMv%Ao8Nv#Hpi5m_q87d>S^qxb*fCpu>b&%KvS%kpDWm zoa=3D<}#b;^s8|T&hBq3icDiz}fWg<1Q#SZ-rv&iVj`$+2B#&-9kib>AngVXsP8ujNU zP5gZXc-XBL;m-L;+mTS8xY8-%Ym?&JbuiW%DJ2gvSlOo!x_@i6gb1$(aobW`ewHguznU}1c+gouFH!TPGSCmr6P@fwr zU0O9Yw^hpb%vhNLFc&N5Rb7u-b>(!=3DCjZ#o4}=3DR!1LER7NL82*bnhvxo0@F>akF^%XC`#$r<6MR}fzi3bK}JCA;nC1WZ21mY4rq02tk3<@Uy)P0zPHhZ=3DtxYiq&dY|FF#&M)$eJf zL4BH9VYTcn@g&uZTtEc@tfGoMRxT+dAVa z^@uMBwnU`9O02Js)(w(qVHDbzqAt)NkQ57&tN<$9`nBX^`u_>!zbENE0ajsdYQW;w zvN2i080~z$<2KHuG$;}Fxqd?p_ZAh1OglD_ckXp|1_4DF%YahZ?zV6l6i z<)hmyJQ3@08i{mI2j`iam-kcghOy6enGPB?QQ`Fs$v0G5okZ6>p71hO!oM=3Ds?$o3+ zfft_P1SVNp9Pdt@GzTkqDMjgCZS>Y~z3fqDfGNBtujA9AC&Uj1e3J#u;63F799{y$ zt<}x`YeMcplP89Ux&_f_YgAlA8Q%0-C69d|UB6MhAHd6FTpAKhycMg& zxA9eVEI5WjE-uyqJp&RA`~(}O0?o^**h|hCYVT+aKCpY{Ke2}8+YRi-C4DfqR0cYE9}?SKDVAWVP@GAs^g)Ux|OQ`yuAU z+I6m&N)lu;IUXa9N)Kh2T8VxhOKbi!kSH0Nf6$ZDp?0f9=3DPU08F9#X@ESWp?XmGfM z$ko~_$CXye-}ksIPwms&(I^zS`#oQtx>unfv)55D|9GK9L|F#+#6RUhKH zV1&T@PIl)(>m#>b#At^dv{JdK%yId=3DF$o$T)-f~(&xLZm+elptZ|9Duoa(9Aabf*7 zUM`E+u7I`z6eXI737u zypK07p1Z3c(%k^OgN=3D7Jo7d>&J`>EAdp#<4%5)3`HY zP7~GQd4}t1mREi&^4K znJzx2y6SAh?RU^IfTQbXf|VFD>v7kGvfN)H6hBs_{7@+AUKH;kwX6S}gwq*%ot2nB zK~vY5xO2ihta1Nhx;spO=3DKATn<}Vc$17h*Wg!E#Dj}H}#uOI&s*RSA)3ND){Z~qxZ z!aj&`_BB2+mW8G51zM6x8!S8L=3D5}y=3Df8sv6oX1?g?E(R_gKnOi9U=3DG%*AG`oK84cD z=3DTDmB`@UgwSu}aL;6GNc5i(5I-@rVgHUz+=3Dc=3D^ASXv%(uoh#i>6@}KWhKMxU=3DLWP= ! z)75w$`dz~ID$npkh&~Xl0bHM9*o}G)KM$?EUPn*&g+}ueFPp=3DfJY2J-a{NpDRt909od@R#t-|cAzCrGZ_q$Jm7C&$1c2miqOFlF7bct34LW^H#u4prgrq7fx; za|<@WJ%QowR;HnorDL_9^!0ihhW71!ZKQa4f?vk@zhdZ;^B$&k=3D-$4F!YtSd{HN3K zp+azY1e$?i$R{{G9z>rcxpMU;C978w<}O9n>Rw_U4OcrOI2+kYGUJuh&EX7y$Z(;Y zwh4X;@;*}o7K1_-L2T3iRp-I?A-PPL1q)}d39NG#8!{xG5@p!zp{37CnpReQQ5br2 zO!&_ogV|jmwKhJKO@25RnitoR__c~=3DNv5-?@RKCK&BGHs9K5HM=3D4$q_LoFK96iJ2j zG(HNJXP?8&BhREuMyE18vK8gNjkcCgLATc#Jc2Hu(2j&wgX%jK=3DPVgVfyZAtGdP}+ z6@i$IVc~U`>z`qKdGLDx9cyRDMIpK%Pt^@P;897yd;7Ubq{`9l`|yi?{1b&QtG>{+ zf=3D7*=3Dmtud*3isZv(dcL(EA@3uJq5B)^Uf}(s`I+^SB}5xn7zl36~=3D^peuSuncG;Uh zBDZwlwO=3D_&% + + + + + + + + + + + + + + + + + + + + diff --git a/documentation/test-manual/test-manual-intro.xml b/documentat= ion/test-manual/test-manual-intro.xml new file mode 100644 index 000000000..5e9def894 --- /dev/null +++ b/documentation/test-manual/test-manual-intro.xml @@ -0,0 +1,634 @@ + %poky; ] > + + + +The Yocto Project Test Environment Manual +
+ Welcome + + Welcome to the Yocto Project Test Environment Manual! Thi= s manual is a work in + progress. The manual contains information about the testing = environment used by the + Yocto Project to make sure each major and minor release work= s as intended. All the + project=E2=80=99s testing infrastructure and processes are p= ublicly visible and available so + that the community can see what testing is being performed, = how it=E2=80=99s being done and the + current status of the tests and the project at any given tim= e. It is intended that Other + organizations can leverage off the process and testing envir= onment used by the Yocto + Project to create their own automated, production test envir= onment, building upon the + foundations from the project core. + + Currently, the Yocto Project Test Environment Manual has = no projected release date. + This manual is a work-in-progress and is being initially loa= ded with information from + the README files and notes from key = engineers: + + + yocto-autobuilder2: This README.md = is the main README which + detials how to set up the Yocto Project Autobuil= der. The + yocto-autobuilder2 repo= sitory represents the Yocto + Project's console UI plugin to Buildbot and the = configuration necessary to + configure Buildbot to perform the testing the pr= oject requires. + + + + yocto-autobuilder-helper: This + README and= repository contains Yocto + Project Autobuilder Helper scripts and configura= tion. The + yocto-autobuilder-helper repository contains the + "glue" logic that defines which tests to run and= how to run them. As a + result, it can be used by any Continuous Improve= ment (CI) system to run + builds, support getting the correct code revisio= ns, configure builds and + layers, run builds, and collect results. The cod= e is independent of any CI + system, which means the code can work Buildbot, = Jenkins, or others. This + repository has a branch per release of the proje= ct defining the tests to run + on a per release basis. + + + +
+ +
+ Yocto Project Autobuilder Overview + + The Yocto Project Autobuilder collectively refers to the s= oftware, tools, scripts, and + procedures used by the Yocto Project to test released softwa= re across supported hardware + in an automated and regular fashion. Basically, during the d= evelopment of a Yocto + Project release, the Autobuilder tests if things work. The A= utobuilder builds all test + targets and runs all the tests. + + The Yocto Project uses now uses standard upstream Buildbot= (version 9) to drive + its integration and testing. Buildbot Nine has a plug-in int= erface that the Yocto + Project customizes using code from the yocto-autob= uilder2 + repository, adding its own console UI plugin. The resulting = UI plug-in allows you to + visualize builds in a way suited to the project's needs. + + A helper layer provides configuration= and job management through + scripts found in the yocto-autobuilder-helper repository. The + helper layer contains the bulk of t= he build configuration + information and is release-specific, which makes it highly c= ustomizable on a per-project + basis. The layer is CI system-agnostic and contains a number= of Helper scripts that can + generate build configurations from simple JSON files. + The project uses Buildbot for historical reasons b= ut also because many of the + project developers have knowledge of python. It is p= ossible to use the outer + layers from another Continuous Integration (CI) syst= em such as Jenkins + instead of Buildbot. + + + + The following figure shows the Yocto Project Autobuilder = stack with a topology that + includes a controller and a cluster of workers: + +
+ +
+ Yocto Project Tests - Types of Testing Overview + + The Autobuilder tests different elements of the project by= using thefollowing types of + tests: + + + Build Testing: Tests whethe= r specific configurations + build by varying MACHINE, <= ulink + url=3D"&YOCTO_DOCS_REF_URL;#var-DISTRO" + >DISTRO, other = configuration options, and + the specific target images being built (or world= ). Used to trigger builds of + all the different test configurations on the Aut= obuilder. Builds usually + cover many different targets for different archi= tectures, machines, and + distributions, as well as different configuratio= ns, such as different init + systems. The Autobuilder tests literally hundred= s of configurations and + targets. + + + Sanity Checks During the B= uild Process: + Tests initiated through the insane= class. These checks + ensure the output of the builds are = correct. For example, does + the ELF architecture in the generate= d binaries match the target + system? ARM binaries would not work = in a MIPS system! + + + + + + Build Performance Testing: = Tests whether or not + commonly used steps during builds work efficient= ly and avoid regressions. + Tests to time commonly used usage scenarios are = run through + oe-build-perf-test. The= se tests are run on isolated + machines so that the time measurements of the te= sts are accurate and no + other processes interfere with the timing result= s. The project currently + tests performance on two different distributions= , Fedora and Ubuntu, to + ensure we have no single point of failure and ca= n ensure the different + distros work effectively. + + + + eSDK Testing: Image tests i= nitiated through the + following command: + + $ bitbake image -c testsdkext + + The tests utilize the testsdkext class and the + do_testsdkext task. + + + + Feature Testing: Various sc= enario-based tests are run + through the OpenEmbedded Self-Test (oe-selftest= ). We test oe-selftest on + each of the main distrubutions we support. + + + + Image Testing: Image tests = initiated through the + following command: + + $ bitbake image -c testimage + + The tests utilize the testimage*= classes and the do_testimage task. + + + + Layer Testing: The Autobuil= der has the possibility to + test whether specific layers work with the test = of the system. The layers + tested may be selected by members of the project= . Some key community layers + are also tested periodically. + + + + Package Testing: A Package = Test (ptest) runs tests + against packages built by the OpenEmbedded build= system on the target + machine. See the "Testing Packages + With ptest" section in the Yocto Pro= ject Development Tasks + Manual and the "Ptest" Wiki + page for more information on Ptest. + + + + SDK Testing: Image tests in= itiated through the + following command: + + $ bitbake image -c testsdk + + The tests utilize the testsdk cl= ass and the + do_testsdk task. + + + + Unit Testing: Unit tests on= various components of the + system run through oe-selftest and bitbake-selftest. + + + + Automatic Upgrade Helper: T= his target tests whether new + versions of software are available and whether w= e can automatically upgrade + to those new versions. If so, this target emails= the maintainers with a + patch to let them know this is possible. + + + +
+ +
+ How Tests Map to Areas of Code + + + Tests map into the codebase as follows: + + + bitbake-selftest: + + These tests are self-contained and= test BitBake as well as its + APIs, which include the fetchers. Th= e tests are located in + bitbake/lib/*/tests. + + + From within the BitBake repository= , run the following: + + $ bitbake-selftest + + + + + To skip tests that access the Inte= rnet, use the + BB_SKIP_NETTEST variable when running + "bitbake-selftest" as follows: + + $ BB_SKIP_NETTEST=3Dyes bitbake-selftest + The + default output is quiet and just pri= nts a summary of what was + run. To see more information, there = is a verbose + option: + $ bitbake-selftest -v + + Use this option when you wish to s= kip tests that access the + network, which are mostly necessary = to test the fetcher modules. + To specify individual test modules t= o run, append the test + module name to the "bitbake-selftest= " command. For example, to + specify the tests for the bb.data.mo= dule, run: + + $ bitbake-selftest bb.test.data.module + You + can also specify individual tests by= defining the full name and + module plus the class path of the te= st, for example: + + $ bitbake-selftest bb.tests.data.TestOverrides.test_one_override + + + + The tests are based on Python + unittest. + + + + + oe-selftest: + + These tests use OE to test the wor= kflows, which include + testing specific features, behaviors= of tasks, and API unit + tests. + + + The tests can take advantage of pa= rallelism through the "-j" + option, which can specify a number o= f threads to spread the + tests across. Note that all tests fr= om a given class of tests + will run in the same thread. To para= llelize large numbers of + tests you can split the class into m= ultiple units. + + + The tests are based on Python unit= test. + + + The code for the tests resides in + meta/lib/oeqa/selftest= /cases/. + + + To run all the tests, enter the fo= llowing command: + + $ oe-selftest -a + + + + + To run a specific test, use the fo= llowing command form where + testname is the name of the + specific test: + + $ oe-selftest -r testname + + For example, the following command w= ould run the tinfoil getVar + API + test: + $ oe-selftest -r tinfoil.TinfoilTests.test_getvar + It + is also possible to run a set of tes= ts. For example the + following command will run all of th= e tinfoil + tests: + $ oe-selftest -r tinfoil + + + + + + testimage: + + + These tests build an image, boot it, and run= tests + against the image's content. + + The code for these tests reside= s in meta/lib/oeqa/runtime/cases/. + + You need to set the + IMAGE_CLASSES + variable as follows: + + IMAGE_CLASSES +=3D "testimage" + + + + Run the tests using the following command fo= rm: + + $ bitbake image -c testimage + + + + + + testsdk: + + These tests build an SDK, instal= l it, and then run tests against that SDK. + The code for these tests resides= in meta/lib/oeqa/sdk/cases/. + Run the test using the following= command form: + + $ bitbake image -c testsdk + + + + + + testsdk_ext: + + These tests build an extended SD= K (eSDK), install that eSDK, and run tests against the eSDK. + The code for these tests resides= in meta/lib/oeqa/esdk. + To run the tests, use the follow= ing command form: + + $ bitbake image -c testsdkext + + + + + + + + oe-build-perf-test: + + These tests run through commonly= used usage scenarios and measure the performance times. + The code for these tests resides= in meta/lib/oeqa/buildperf. + To run the tests, use the follow= ing command form: + + $ oe-build-perf-test options + The + command takes a number of options, s= uch as where to place the + test results. The Autobuilder Helper= Scripts include the + build-perf-test-wrappe= r script with + examples of how to use the oe-build-= perf-test from the command + line. + Use the oe-git-archive command to store + test results into a Git repository. = + Use the oe-build-perf-re= port command to + generate text reports and HTML repor= ts with graphs of the + performance data. For examples, see = http://downloads.yoctoproject.o= rg/releases/yocto/yocto-2.7/testresults/buildperf-centos7/perf-centos7.yo= ctoproject.org_warrior_20190414204758_0e39202.html + and http://downloads.yoctoproject.o= rg/releases/yocto/yocto-2.7/testresults/buildperf-centos7/perf-centos7.yo= ctoproject.org_warrior_20190414204758_0e39202.txt. + + The tests are contained in + lib/oeqa/buildperf/tes= t_basic.py. + + + + + + + + + +
+ +
+ Test Examples + + This section provides example tests for each of the tests = listed in the How Tests Map to Areas of = Code section. + For oeqa tests, testcases for each area reside in the main= test directory at + meta/lib/oeqa/selftest/cases direct= ory. + For oe-selftest. bitbake testcases reside in the lib/bb/tests/ + directory. + +
+ <filename>bitbake-selftest</filename> + + A simple test example from lib/bb/tests/data= .py is: + + class DataExpansions(unittest.TestCase): + def setUp(self): + self.d =3D bb.data.init() + self.d["foo"] =3D "value_of_foo" + self.d["bar"] =3D "value_of_bar" =20 + self.d["value_of_foo"] =3D "value_of_'value_of_foo'" + =20 + def test_one_var(self): + val =3D self.d.expand("${foo}") + self.assertEqual(str(val), "value_of_foo") + + + In this example, a DataExpan= sions class + of tests is created, derived from standard python unitte= st. The class has a common + setUp function which is shared = by all the tests in the + class. A simple test is then added to test that when a v= ariable is expanded, the + correct value is found. + Bitbake selftests are straightforward python unittest.= Refer to the Python + unittest documentation for additional information on wri= ting these tests at: https://docs.python.org/3/library/unitt= est.html. +
+ +
+ <filename>oe-selftest</filename> + + These tests are more complex due to the setup required= behind the scenes for full + builds. Rather than directly using Python's unittest, th= e code wraps most of the + standard objects. The tests can be simple, such as testi= ng a command from within the + OE build environment using the following + example: + class BitbakeLayers(OESelftestTestCase): + def test_bitbakelayers_showcrossdepends(self): + result =3D runCmd('bitbake-layers show-cross-depends') = =20 + self.assertTrue('aspell' in result.output, msg =3D "No depe= ndencies=20 + were shown. bitbake-layers show-cross-depends output: + %s"% result.output) + + This example, taken from + meta/lib/oeqa/selftest/cases/bblayers.py, creates a + testcase from the OESelftestTe= stCase + class, derived from unittest.TestCase, which runs the + bitbake-layers command and chec= ks the output to ensure it + contains something we know should be here. + The oeqa.utils.commands module co= ntains Helpers which can + assist with common tasks, including: + + Obtaining the value of a bitbake= variable: Use + oeqa.utils.commands.get_bb_var= () or use + oeqa.utils.commands.get_bb_var= s() for more than + one variable + + + Running a bitbake invocation for= a build: Use + oeqa.utils.commands.bitbake()<= /filename> + + + Running a command: Us= e + oeqa.utils.commandsrunCmd() + + + There is also a oeqa.utils.commands.runqemu(= ) function for + launching the runqemu command for t= esting things within a + running, virtualized image. + You can run these tests in parallel. Parallelism works= per test class, so tests + within a given test class should always run in the same = build, while tests in + different classes or modules may be split into different= builds. There is no data + store available for these tests since the tests launch t= he + bitbake command and exist outsi= de of its context. As a + result, common bitbake library functions (bb.*) are also= unavailable. +
+ +
+ <filename>testimage</filename> + + These tests are run once an image is up and running, e= ither on target hardware or + under QEMU. As a result, they are assumed to be running = in a target image + environment, as opposed to a host build environment. A s= imple example from + meta/lib/oeqa/runtime/cases/python.py contains the + following: + class PythonTest(OERuntimeTestCase): + @OETestDepends(['ssh.SSHTest.test_ssh']) + @OEHasPackage(['python3-core']) + def test_python3(self): + cmd =3D "python3 -c \"import codecs; print(codecs.encode('U= ryyb, + jbeyq', 'rot13'))\"" + status, output =3D self.target.run(cmd) + msg =3D 'Exit status was not 0. Output: %s' % output + self.assertEqual(status, 0, msg=3Dmsg) + + In this example, the OERunti= meTestCase + class wraps unittest.TestCase. With= in the test, + self.target represents the targ= et system, where commands + can be run on it using the run() me= thod. + To ensure certain test or package dependencies are met= , you can use the + OETestDepends and OEH= asPackage + decorators. For example, the test in this example would = only make sense if + python3-core is installed in the image. +
+ +
+ <filename>testsdk_ext</filename> + + These tests are run against built extensible SDKs (eSD= Ks). The tests can assume + that the eSDK environment has already been setup. An exa= mple from + meta/lib/oeqa/sdk/cases/devtool.py contains the + following: + class DevtoolTest(OESDKExtTestCase): + @classmethod + def setUpClass(cls): + myapp_src =3D os.path.join(cls.tc.esdk_files_dir, "myapp") + cls.myapp_dst =3D os.path.join(cls.tc.sdk_dir, "myapp") + shutil.copytree(myapp_src, cls.myapp_dst) + subprocess.check_output(['git', 'init', '.'], cwd=3Dcls.myapp_= dst) + subprocess.check_output(['git', 'add', '.'], cwd=3Dcls.myapp_d= st) + subprocess.check_output(['git', 'commit', '-m', "'test commit'= "], cwd=3Dcls.myapp_dst) + + @classmethod + def tearDownClass(cls): + shutil.rmtree(cls.myapp_dst) + def _test_devtool_build(self, directory): + self._run('devtool add myapp %s' % directory) + try: + self._run('devtool build myapp') + finally: + self._run('devtool reset myapp') + def test_devtool_build_make(self): + self._test_devtool_build(self.myapp_dst) + In + this example, the devtool command i= s tested to see whether a + sample application can be built with the devto= ol build command + within the eSDK. +
+ +
+ <filename>testsdk</filename> + + These tests are run against built SDKs. The tests can = assume that an SDK has + already been extracted and its environment file has been= sourced. A simple example + from meta/lib/oeqa/sdk/cases/python2.py contains the + following: + class Python3Test(OESDKTestCase): + def setUp(self): + if not (self.tc.hasHostPackage("nativesdk-python3-core") or + self.tc.hasHostPackage("python3-core-native")): + raise unittest.SkipTest("No python3 package in the SDK"= ) + + def test_python3(self): + cmd =3D "python3 -c \"import codecs; print(codecs.encode('U= ryyb, jbeyq', 'rot13'))\"" + output =3D self._run(cmd) + self.assertEqual(output, "Hello, world\n") + In + this example, if nativesdk-python3-core has been install= ed into the SDK, the code + runs the python3 interpreter with a basic command to che= ck it is working correctly. + The test would only run if python3 is installed in the S= DK. +
+ +
+ <filename>oe-build-perf-test</filename> + + The performance tests usually measure how long operati= ons take and the resource + utilisation as that happens. An example from + meta/lib/oeqa/buildperf/test_basic.py contains the + following: + class Test3(BuildPerfTestCase): + + def test3(self): + """Bitbake parsing (bitbake -p)""" + # Drop all caches and parse + self.rm_cache() + oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'),= True) + self.measure_cmd_resources(['bitbake', '-p'], 'parse_1', + 'bitbake -p (no caches)') + # Drop tmp/cache + oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'),= True) + self.measure_cmd_resources(['bitbake', '-p'], 'parse_2', + 'bitbake -p (no tmp/cache)') + # Parse with fully cached data + self.measure_cmd_resources(['bitbake', '-p'], 'parse_3', + 'bitbake -p (cached)') + This + example shows how three specific parsing timings are mea= sured, with and without + various caches, to show how BitBake=E2=80=99s parsing pe= rformance trends over time. +
+
+
+ Considerations When Writing Tests + When writing good tests, there are several things to keep = in mind. Since things + running on the Autobuilder are accessed concurrently by mult= iple workers, consider the + following: + + Running "cleanall" is not permitted + This can delete files from DL_DIR which would potentia= lly break other builds + running in parallel. If this is required, DL_DIR must be= set to an isolated + directory. + + + Running "cleansstate" is not permitted + This can delete files from SSTATE_DIR which would pote= ntially break other builds + running in parallel. If this is required, SSTATE_DIR mus= t be set to an isolated + directory. Alternatively, you can use the "-f" option wi= th the + bitbake command to "taint" task= s by changing the sstate + checksums to ensure sstate cache items will not be reuse= d. + + + Tests should not change the metadata + This is particularly true for oe-selftests since these= can run in parallel and + changing metadata leads to changing checksums, which con= fuses BitBake while running + in parallel. If this is necessary, copy layers to a temp= orary location and modify + them. Some tests need to change metadata, such as the de= vtool tests. To prevent the + metadate from changes, set up temporary copies of that d= ata first. + +
+ + + + + + + + + + diff --git a/documentation/test-manual/test-manual-style.css b/documentat= ion/test-manual/test-manual-style.css new file mode 100644 index 000000000..15ff718b1 --- /dev/null +++ b/documentation/test-manual/test-manual-style.css @@ -0,0 +1,989 @@ +/* + Generic XHTML / DocBook XHTML CSS Stylesheet. + + Browser wrangling and typographic design by + Oyvind Kolas / pippin@gimp.org + + Customised for Poky by + Matthew Allum / mallum@o-hand.com + + Thanks to: + Liam R. E. Quin + William Skaggs + Jakub Steiner + + Structure + --------- + + The stylesheet is divided into the following sections: + + Positioning + Margins, paddings, width, font-size, clearing. + Decorations + Borders, style + Colors + Colors + Graphics + Graphical backgrounds + Nasty IE tweaks + Workarounds needed to make it work in internet explorer, + currently makes the stylesheet non validating, but up until + this point it is validating. + Mozilla extensions + Transparency for footer + Rounded corners on boxes + +*/ + + + /*************** / + / Positioning / +/ ***************/ + +body { + font-family: Verdana, Sans, sans-serif; + + min-width: 640px; + width: 80%; + margin: 0em auto; + padding: 2em 5em 5em 5em; + color: #333; +} + +h1,h2,h3,h4,h5,h6,h7 { + font-family: Arial, Sans; + color: #00557D; + clear: both; +} + +h1 { + font-size: 2em; + text-align: left; + padding: 0em 0em 0em 0em; + margin: 2em 0em 0em 0em; +} + +h2.subtitle { + margin: 0.10em 0em 3.0em 0em; + padding: 0em 0em 0em 0em; + font-size: 1.8em; + padding-left: 20%; + font-weight: normal; + font-style: italic; +} + +h2 { + margin: 2em 0em 0.66em 0em; + padding: 0.5em 0em 0em 0em; + font-size: 1.5em; + font-weight: bold; +} + +h3.subtitle { + margin: 0em 0em 1em 0em; + padding: 0em 0em 0em 0em; + font-size: 142.14%; + text-align: right; +} + +h3 { + margin: 1em 0em 0.5em 0em; + padding: 1em 0em 0em 0em; + font-size: 140%; + font-weight: bold; +} + +h4 { + margin: 1em 0em 0.5em 0em; + padding: 1em 0em 0em 0em; + font-size: 120%; + font-weight: bold; +} + +h5 { + margin: 1em 0em 0.5em 0em; + padding: 1em 0em 0em 0em; + font-size: 110%; + font-weight: bold; +} + +h6 { + margin: 1em 0em 0em 0em; + padding: 1em 0em 0em 0em; + font-size: 110%; + font-weight: bold; +} + +.authorgroup { + background-color: transparent; + background-repeat: no-repeat; + padding-top: 256px; + background-image: url("figures/test-manual-title.png"); + background-position: left top; + margin-top: -256px; + padding-right: 50px; + margin-left: 0px; + text-align: right; + width: 740px; +} + +h3.author { + margin: 0em 0me 0em 0em; + padding: 0em 0em 0em 0em; + font-weight: normal; + font-size: 100%; + color: #333; + clear: both; +} + +.author tt.email { + font-size: 66%; +} + +.titlepage hr { + width: 0em; + clear: both; +} + +.revhistory { + padding-top: 2em; + clear: both; +} + +.toc, +.list-of-tables, +.list-of-examples, +.list-of-figures { + padding: 1.33em 0em 2.5em 0em; + color: #00557D; +} + +.toc p, +.list-of-tables p, +.list-of-figures p, +.list-of-examples p { + padding: 0em 0em 0em 0em; + padding: 0em 0em 0.3em; + margin: 1.5em 0em 0em 0em; +} + +.toc p b, +.list-of-tables p b, +.list-of-figures p b, +.list-of-examples p b{ + font-size: 100.0%; + font-weight: bold; +} + +.toc dl, +.list-of-tables dl, +.list-of-figures dl, +.list-of-examples dl { + margin: 0em 0em 0.5em 0em; + padding: 0em 0em 0em 0em; +} + +.toc dt { + margin: 0em 0em 0em 0em; + padding: 0em 0em 0em 0em; +} + +.toc dd { + margin: 0em 0em 0em 2.6em; + padding: 0em 0em 0em 0em; +} + +div.glossary dl, +div.variablelist dl { +} + +.glossary dl dt, +.variablelist dl dt, +.variablelist dl dt span.term { + font-weight: normal; + width: 20em; + text-align: right; +} + +.variablelist dl dt { + margin-top: 0.5em; +} + +.glossary dl dd, +.variablelist dl dd { + margin-top: 0em; + margin-left: 25.5em; +} + +.glossary dd p, +.variablelist dd p { + margin-top: 0em; + margin-bottom: 1em; +} + + +div.calloutlist table td { + padding: 0em 0em 0em 0em; + margin: 0em 0em 0em 0em; +} + +div.calloutlist table td p { + margin-top: 0em; + margin-bottom: 1em; +} + +div p.copyright { + text-align: left; +} + +div.legalnotice p.legalnotice-title { + margin-bottom: 0em; +} + +p { + line-height: 1.5em; + margin-top: 0em; + +} + +dl { + padding-top: 0em; +} + +hr { + border: solid 1px; +} + + +.mediaobject, +.mediaobjectco { + text-align: center; +} + +img { + border: none; +} + +ul { + padding: 0em 0em 0em 1.5em; +} + +ul li { + padding: 0em 0em 0em 0em; +} + +ul li p { + text-align: left; +} + +table { + width :100%; +} + +th { + padding: 0.25em; + text-align: left; + font-weight: normal; + vertical-align: top; +} + +td { + padding: 0.25em; + vertical-align: top; +} + +p a[id] { + margin: 0px; + padding: 0px; + display: inline; + background-image: none; +} + +a { + text-decoration: underline; + color: #444; +} + +pre { + overflow: auto; +} + +a:hover { + text-decoration: underline; + /*font-weight: bold;*/ +} + +/* This style defines how the permalink character + appears by itself and when hovered over with + the mouse. */ + +[alt=3D'Permalink'] { color: #eee; } +[alt=3D'Permalink']:hover { color: black; } + + +div.informalfigure, +div.informalexample, +div.informaltable, +div.figure, +div.table, +div.example { + margin: 1em 0em; + padding: 1em; + page-break-inside: avoid; +} + + +div.informalfigure p.title b, +div.informalexample p.title b, +div.informaltable p.title b, +div.figure p.title b, +div.example p.title b, +div.table p.title b{ + padding-top: 0em; + margin-top: 0em; + font-size: 100%; + font-weight: normal; +} + +.mediaobject .caption, +.mediaobject .caption p { + text-align: center; + font-size: 80%; + padding-top: 0.5em; + padding-bottom: 0.5em; +} + +.epigraph { + padding-left: 55%; + margin-bottom: 1em; +} + +.epigraph p { + text-align: left; +} + +.epigraph .quote { + font-style: italic; +} +.epigraph .attribution { + font-style: normal; + text-align: right; +} + +span.application { + font-style: italic; +} + +.programlisting { + font-family: monospace; + font-size: 80%; + white-space: pre; + margin: 1.33em 0em; + padding: 1.33em; +} + +.tip, +.warning, +.caution, +.note { + margin-top: 1em; + margin-bottom: 1em; + +} + +/* force full width of table within div */ +.tip table, +.warning table, +.caution table, +.note table { + border: none; + width: 100%; +} + + +.tip table th, +.warning table th, +.caution table th, +.note table th { + padding: 0.8em 0.0em 0.0em 0.0em; + margin : 0em 0em 0em 0em; +} + +.tip p, +.warning p, +.caution p, +.note p { + margin-top: 0.5em; + margin-bottom: 0.5em; + padding-right: 1em; + text-align: left; +} + +.acronym { + text-transform: uppercase; +} + +b.keycap, +.keycap { + padding: 0.09em 0.3em; + margin: 0em; +} + +.itemizedlist li { + clear: none; +} + +.filename { + font-size: medium; + font-family: Courier, monospace; +} + + +div.navheader, div.heading{ + position: absolute; + left: 0em; + top: 0em; + width: 100%; + background-color: #cdf; + width: 100%; +} + +div.navfooter, div.footing{ + position: fixed; + left: 0em; + bottom: 0em; + background-color: #eee; + width: 100%; +} + + +div.navheader td, +div.navfooter td { + font-size: 66%; +} + +div.navheader table th { + /*font-family: Georgia, Times, serif;*/ + /*font-size: x-large;*/ + font-size: 80%; +} + +div.navheader table { + border-left: 0em; + border-right: 0em; + border-top: 0em; + width: 100%; +} + +div.navfooter table { + border-left: 0em; + border-right: 0em; + border-bottom: 0em; + width: 100%; +} + +div.navheader table td a, +div.navfooter table td a { + color: #777; + text-decoration: none; +} + +/* normal text in the footer */ +div.navfooter table td { + color: black; +} + +div.navheader table td a:visited, +div.navfooter table td a:visited { + color: #444; +} + + +/* links in header and footer */ +div.navheader table td a:hover, +div.navfooter table td a:hover { + text-decoration: underline; + background-color: transparent; + color: #33a; +} + +div.navheader hr, +div.navfooter hr { + display: none; +} + + +.qandaset tr.question td p { + margin: 0em 0em 1em 0em; + padding: 0em 0em 0em 0em; +} + +.qandaset tr.answer td p { + margin: 0em 0em 1em 0em; + padding: 0em 0em 0em 0em; +} +.answer td { + padding-bottom: 1.5em; +} + +.emphasis { + font-weight: bold; +} + + + /************* / + / decorations / +/ *************/ + +.titlepage { +} + +.part .title { +} + +.subtitle { + border: none; +} + +/* +h1 { + border: none; +} + +h2 { + border-top: solid 0.2em; + border-bottom: solid 0.06em; +} + +h3 { + border-top: 0em; + border-bottom: solid 0.06em; +} + +h4 { + border: 0em; + border-bottom: solid 0.06em; +} + +h5 { + border: 0em; +} +*/ + +.programlisting { + border: solid 1px; +} + +div.figure, +div.table, +div.informalfigure, +div.informaltable, +div.informalexample, +div.example { + border: 1px solid; +} + + + +.tip, +.warning, +.caution, +.note { + border: 1px solid; +} + +.tip table th, +.warning table th, +.caution table th, +.note table th { + border-bottom: 1px solid; +} + +.question td { + border-top: 1px solid black; +} + +.answer { +} + + +b.keycap, +.keycap { + border: 1px solid; +} + + +div.navheader, div.heading{ + border-bottom: 1px solid; +} + + +div.navfooter, div.footing{ + border-top: 1px solid; +} + + /********* / + / colors / +/ *********/ + +body { + color: #333; + background: white; +} + +a { + background: transparent; +} + +a:hover { + background-color: #dedede; +} + + +h1, +h2, +h3, +h4, +h5, +h6, +h7, +h8 { + background-color: transparent; +} + +hr { + border-color: #aaa; +} + + +.tip, .warning, .caution, .note { + border-color: #fff; +} + + +.tip table th, +.warning table th, +.caution table th, +.note table th { + border-bottom-color: #fff; +} + + +.warning { + background-color: #f0f0f2; +} + +.caution { + background-color: #f0f0f2; +} + +.tip { + background-color: #f0f0f2; +} + +.note { + background-color: #f0f0f2; +} + +.glossary dl dt, +.variablelist dl dt, +.variablelist dl dt span.term { + color: #044; +} + +div.figure, +div.table, +div.example, +div.informalfigure, +div.informaltable, +div.informalexample { + border-color: #aaa; +} + +pre.programlisting { + color: black; + background-color: #fff; + border-color: #aaa; + border-width: 2px; +} + +.guimenu, +.guilabel, +.guimenuitem { + background-color: #eee; +} + + +b.keycap, +.keycap { + background-color: #eee; + border-color: #999; +} + + +div.navheader { + border-color: black; +} + + +div.navfooter { + border-color: black; +} + +.writernotes { + color: red; +} + + + /*********** / + / graphics / +/ ***********/ + +/* +body { + background-image: url("images/body_bg.jpg"); + background-attachment: fixed; +} + +.navheader, +.note, +.tip { + background-image: url("images/note_bg.jpg"); + background-attachment: fixed; +} + +.warning, +.caution { + background-image: url("images/warning_bg.jpg"); + background-attachment: fixed; +} + +.figure, +.informalfigure, +.example, +.informalexample, +.table, +.informaltable { + background-image: url("images/figure_bg.jpg"); + background-attachment: fixed; +} + +*/ +h1, +h2, +h3, +h4, +h5, +h6, +h7{ +} + +/* +Example of how to stick an image as part of the title. + +div.article .titlepage .title +{ + background-image: url("figures/white-on-black.png"); + background-position: center; + background-repeat: repeat-x; +} +*/ + +div.preface .titlepage .title, +div.colophon .title, +div.chapter .titlepage .title, +div.article .titlepage .title +{ +} + +div.section div.section .titlepage .title, +div.sect2 .titlepage .title { + background: none; +} + + +h1.title { + background-color: transparent; + background-image: url("figures/test-title.png"); + background-repeat: no-repeat; + height: 256px; + text-indent: -9000px; + overflow:hidden; +} + +h2.subtitle { + background-color: transparent; + text-indent: -9000px; + overflow:hidden; + width: 0px; + display: none; +} + + /*************************************** / + / pippin.gimp.org specific alterations / +/ ***************************************/ + +/* +div.heading, div.navheader { + color: #777; + font-size: 80%; + padding: 0; + margin: 0; + text-align: left; + position: absolute; + top: 0px; + left: 0px; + width: 100%; + height: 50px; + background: url('/gfx/heading_bg.png') transparent; + background-repeat: repeat-x; + background-attachment: fixed; + border: none; +} + +div.heading a { + color: #444; +} + +div.footing, div.navfooter { + border: none; + color: #ddd; + font-size: 80%; + text-align:right; + + width: 100%; + padding-top: 10px; + position: absolute; + bottom: 0px; + left: 0px; + + background: url('/gfx/footing_bg.png') transparent; +} +*/ + + + + /****************** / + / nasty ie tweaks / +/ ******************/ + +/* +div.heading, div.navheader { + width:expression(document.body.clientWidth + "px"); +} + +div.footing, div.navfooter { + width:expression(document.body.clientWidth + "px"); + margin-left:expression("-5em"); +} +body { + padding:expression("4em 5em 0em 5em"); +} +*/ + + /**************************************** / + / mozilla vendor specific css extensions / +/ ****************************************/ +/* +div.navfooter, div.footing{ + -moz-opacity: 0.8em; +} + +div.figure, +div.table, +div.informalfigure, +div.informaltable, +div.informalexample, +div.example, +.tip, +.warning, +.caution, +.note { + -moz-border-radius: 0.5em; +} + +b.keycap, +.keycap { + -moz-border-radius: 0.3em; +} +*/ + +table tr td table tr td { + display: none; +} + + +hr { + display: none; +} + +table { + border: 0em; +} + + .photo { + float: right; + margin-left: 1.5em; + margin-bottom: 1.5em; + margin-top: 0em; + max-width: 17em; + border: 1px solid gray; + padding: 3px; + background: white; +} + .seperator { + padding-top: 2em; + clear: both; + } + + #validators { + margin-top: 5em; + text-align: right; + color: #777; + } + @media print { + body { + font-size: 8pt; + } + .noprint { + display: none; + } + } + + +.tip, +.note { + background: #f0f0f2; + color: #333; + padding: 20px; + margin: 20px; +} + +.tip h3, +.note h3 { + padding: 0em; + margin: 0em; + font-size: 2em; + font-weight: bold; + color: #333; +} + +.tip a, +.note a { + color: #333; + text-decoration: underline; +} + +.footnote { + font-size: small; + color: #333; +} + +/* Changes the announcement text */ +.tip h3, +.warning h3, +.caution h3, +.note h3 { + font-size:large; + color: #00557D; +} diff --git a/documentation/test-manual/test-manual-test-process.xml b/doc= umentation/test-manual/test-manual-test-process.xml new file mode 100644 index 000000000..0b5036cd2 --- /dev/null +++ b/documentation/test-manual/test-manual-test-process.xml @@ -0,0 +1,109 @@ + %poky; ] > + + + +Project Testing and Release Process +
+ Day to Day Development + + This section details how the project tests changes, throug= h automation on the + Autobuilder or with the assistance of QA teams, through to m= aking releases. + + The project aims to test changes against our test matrix b= efore those changes are + merged into the master branch. As such, changes are queued u= p in batches either in the + master-next branch in the main tree= s, or in user trees such as + ross/mut in poky-contrib<= /filename> (Ross Burton + helps review and test patches and this is his testing tree).= + We have two broad categories of test builds, including "fu= ll" and "quick". On the + Autobuilder, these can be seen as "a-quick" and "a-full", si= mply for ease of sorting in + the UI. Use our Autobuilder console view to see where me man= age most test-related items, + available at: https://autobuilder.yoctoproject.org/typhoon/#/console<= /link>. + Builds are triggered manually when the test branches are r= eady. The builds are + monitored by the SWAT team. For additional information, see = https://wiki.yoctoproject.org/wiki/Yocto_Build_Failure_= Swat_Team. If + successful, the changes would usually be merged to the master + branch. If not successful, someone would respond to the chan= ges on the mailing list + explaining that there was a failure in testing. The choice o= f quick or full would depend + on the type of changes and the speed with which the result w= as required. + The Autobuilder does build the master= branch once daily for + several reasons, in particular, to ensure the current master branch + does build, but also to keep yocto-testresults (http://git.yoctoproject.org/cgit.cgi/yocto-testresults/= ), buildhistory + (http://git.yoctoproject.org/cgit.cgi= /poky-buildhistory/), + and our sstate up to date. On the weekend, there is a master= -next build instead to + ensure the test results are updated for the less frequently = run targets. + Performance builds (buildperf-* targets in the console) ar= e triggered separately every + six hours and automatically push their results to the builds= tats repository at: http://git.yoctoproject.org/cgit.cgi/yocto-= buildstats/. + The 'quick' targets have been selected to be the ones whic= h catch the most failures or + give the most valuable data. We run 'fast' ptests in this ca= se for example but not the + ones which take a long time. The quick target doesn't includ= e *-lsb builds for all + architectures, some world builds and doesn't trigger perform= ance tests or ltp testing. + The full build includes all these things and is slower but m= ore comprehensive. +
+ +
+ Release Builds + + The project typically has two major releases a year with a= six month cadence in April + and October. Between these there would be a number of milest= one releases (usually four) + with the final one being stablization only along with point = releases of our stable + branches. + The build and release process for these project releases i= s similar to that in Day to Day Development, in that the a-full target + of the Autobuilder is used but in addition the form is confi= gured to generate and + publish artefacts and the milestone number, version, release= candidate number and other + information is entered. The box to "generate an email to QA"= is also checked. + When the build completes, an email is sent out using the s= end-qa-email script in the + yocto-autobuilder-helper repository= to the list of people + configured for that release. Release builds are placed into = a directory in https://autobuilder.yocto.io/pub/releases on the Autobuilder which + is included in the email. The process from here is more manu= al and control is + effectively passed to release engineering. The next steps in= clude: + + QA teams respond to the email saying which tes= ts they plan to run and when + the results will be available. + + + QA teams run their tests and share their resul= ts in the yocto- + testresults-contrib repository, along with a sum= mary of their findings. + + + + Release engineering prepare the release as per= their process. + + + Test results from the QA teams are included in= to the release in separate + directories and also uploaded to the yocto-testr= esults repository alongside + the other test results for the given revision. + + + The QA report in the final release is regenera= ted using resulttool to + include the new test results and the test summar= ies from the teams (as + headers to the generated report). + + + The release is checked against the release che= cklist and release readiness + criteria. + + + A final decision on whether to release is made= by the YP TSC who have + final oversight on release readiness. + + +
+ + + + + + + + + + diff --git a/documentation/test-manual/test-manual-understand-autobuilder= .xml b/documentation/test-manual/test-manual-understand-autobuilder.xml new file mode 100644 index 000000000..754130535 --- /dev/null +++ b/documentation/test-manual/test-manual-understand-autobuilder.xml @@ -0,0 +1,312 @@ + %poky; ] > + + + +Understanding the Yocto Project Autobuilder +
+ Execution Flow within the Autobuilder + The =E2=80=9Ca-full=E2=80=9D and =E2=80=9Ca-quick=E2=80=9D= targets are the usual entry points into the Autobuilder and + it makes sense to follow the process through the system star= ting there. This is best + visualised from the Autobuilder Console view (https://autobuilder.yoctoproject.org/typhoon/#/console<= /link>). + Each item along the top of that view represents some =E2=80= =9Ctarget build=E2=80=9D and these targets + are all run in parallel. The =E2=80=98full=E2=80=99 build wi= ll trigger the majority of them, the =E2=80=9Cquick=E2=80=9D + build will trigger some subset of them. The Autobuilder effe= ctively runs whichever + configuration is defined for each of those targets on a sepe= rate buildbot worker. To + understand the configuration, you need to look at the entry = on + config.json file within the + yocto-autobuilder-helper repository= . The targets are defined in + the =E2=80=98overrides=E2=80=99 section, a quick example cou= ld be qemux86-64 which looks + like: + "qemux86-64" : { + "MACHINE" : "qemux86-64", + "TEMPLATE" : "arch-qemu", + "step1" : { + "extravars" : [ + "IMAGE_FSTYPES_append =3D ' wic wic.bmap'" + ] + } + }, + And + to expand that, you need the =E2=80=9Carch-qemu=E2=80=9D ent= ry from the =E2=80=9Ctemplates=E2=80=9D section, which looks + like: + "arch-qemu" : { + "BUILDINFO" : true, + "BUILDHISTORY" : true, + "step1" : { + "BBTARGETS" : "core-image-sato core-image-sato-dev core-ima= ge-sato-sdk core-image-minimal core-image-minimal-dev core-image-sato:do_= populate_sdk", + "SANITYTARGETS" : "core-image-minimal:do_testimage core-ima= ge-sato:do_testimage core-image-sato-sdk:do_testimage core-image-sato:do_= testsdk" + }, + "step2" : { + "SDKMACHINE" : "x86_64", + "BBTARGETS" : "core-image-sato:do_populate_sdk core-image-m= inimal:do_populate_sdk_ext core-image-sato:do_populate_sdk_ext", + "SANITYTARGETS" : "core-image-sato:do_testsdk core-image-mi= nimal:do_testsdkext core-image-sato:do_testsdkext" + }, + "step3" : { + "BUILDHISTORY" : false, + "EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; DISPLAY=3D:1 oe-sel= ftest ${HELPERSTMACHTARGS} -j 15"], + "ADDLAYER" : ["${BUILDDIR}/../meta-selftest"] + } + }, + Combining + these two entries you can see that =E2=80=9Cqemux86-64=E2=80= =9D is a three step build where the =E2=80=9Cbitbake + BBTARGETS=E2=80=9D would be run, then =E2=80=9Cbitbake SANIT= YTARGETS=E2=80=9D for each step; all for + MACHINE=3D=E2=80=9Dqemx86-64=E2=80=9D but with differing SDK= MACHINE settings. In step 1 an extra variable + is added to the auto.conf file to enabl= e wic image + generation. + While not every detail of this is covered here, you can se= e how the templating + mechanism allows quite complex configurations to be built up= yet allows duplication and + repetition to be kept to a minimum. + The different build targets are designed to allow for para= llelisation, so different + machines are usually built in parallel, operations using the= same machine and metadata + are built sequentially, with the aim of trying to optimise b= uild efficiency as much as + possible. + The config.json file is processed by = the scripts in the Helper + repository in the scripts directory. Th= e following section details + how this works. +
+ +
+ Autobuilder Target Execution Overview + + For each given target in a build, the Autobuilder executes= several steps. These are + configured in yocto-autobuilder2/builders.py and roughly consist + of: + + Run clobberdir<= /emphasis> + This cleans out any previous build. Old builds= are left around to allow + easier debugging of failed builds. For additiona= l information, see clobbe= rdir. + + + Obtain yocto-autobuilder-helper + This step clones the yocto-autobuild= er-helper git + repository. This is necessary to prevent the req= uirement to maintain all the + release or project-specific code within Buildbot= . The branch chosen matches + the release being built so we can support older = releases and still make + changes in newer ones. + + + Write layerinfo.json + This transfers data in the Buildbot UI when th= e build was configured to + the Helper. + + + Call scripts/shared-repo-unpack + This is a call into the Helper scripts to set = up a checkout of all the + pieces this build might need. It might clone the= BitBake repository and the + OpenEmbedded-Core repository. It may clone the P= oky repository, as well as + additional layers. It will use the data from the + layerinfo.json file to = help understand the + configuration. It will also use a local cache of= repositories to speed up + the clone checkouts. For additional information,= see Aut= obuilder Clone + Cache. + This step has two possible modes of operation.= If the build is part of a + parent build, its possible that all the reposito= ries needed may already be + available, ready in a pre-prepared directory. An= "a-quick" or "a-full" build + would prepare this before starting the other sub= -target builds. This is done + for two reasons: + + the upstream may change during a b= uild, for example, from a + forced push and this ensures we have= matching content for the + whole build + + + if 15 Workers all tried to pull th= e same data from the same + repos, we can hit resource limits on= upstream servers as they + can think they are under some kind o= f network attack + + This pre-prepared directory is sh= ared among the Workers over + NFS. If the build is an individual build and the= re is no "shared" directory + available, it would clone from the cache and the= upstreams as necessary. + This is considered the fallback mode. + + + Call scripts/run-config + This is another call into the Helper scripts w= here its expected that the + main functionality of this target will be execut= ed. + + +
+
+ Autobuilder Technology + The Autobuilder has Yocto Project-specific functionality t= o allow builds to operate + with increased efficiency and speed. +
+ clobberdir + When deleting files, the Autobuilder uses cl= obberdir, which + is a special script that moves files to a special locati= on, rather than deleting + them. Files in this location are deleted by an rm command, + which is run under ionice -c 3. For= example, the deletion only + happens when there is idle IO capacity on the Worker. Th= e Autobuilder Worker Janitor + runs this deletion. See Autobuilder + Worker Janitor. +
+
+ Autobuilder Clone Cache + Cloning repositories from scratch each time they are r= equired was slow on the + Autobuilder. We therefore have a stash of commonly used = repositories pre-cloned on + the Workers. Data is fetched from these during clones fi= rst, then "topped up" with + later revisions from any upstream when necesary. The cac= he is maintained by the + Autobuilder Worker Janitor. See Autobuilder Worker Janitor. +
+
+ Autobuilder Worker Janitor + This is a process running on each Worker that performs= two basic operations, + including background file deletion at IO idle (see Target Exe= cution: clobberdir) and + maintainenance of a cache of cloned repositories to impr= ove the speed the system can + checkout repositories. +
+
+ Shared DL_DIR + The Workers are all connected over NFS which allows DL= _DIR to be shared between + them. This reduces network accesses from the system and = allows the build to be sped + up. Usage of the directory within the build system is de= signed to be able to be + shared over NFS. +
+
+ Shared SSTATE_DIR + The Workers are all connected over NFS which allows th= e + sstate directory to be shared b= etween them. This means once + a Worker has built an artefact, all the others can benef= it from it. Usage of the + directory within the directory is designed for sharing o= ver NFS. +
+
+ Resulttool + All of the different tests run as part of the build ge= nerate output into + testresults.json files. This al= lows us to determine which + tests ran in a given build and their status. Additional = information, such as failure + logs or the time taken to run the tests, may also be inc= luded. + Resulttool is part of OpenEmbedded-Core and is used to= manipulate these json + results files. It has the ability to merge files togethe= r, display reports of the + test results and compare different result files. + For details, see https://wiki.yoctoproject.org/wiki/Resulttool. +
+
+
+ run-config Target Execution + The scripts/run-config execution is w= here most of the work within + the Autobuilder happens. It runs through a number of steps; = the first are general setup + steps that are run once and include: + + Set up any buildtools-tarball if configured. + + + Call "buildhistory-init" if buildhistory is co= nfigured. + + + For each step that is configured in config.json<= /filename>, it will perform + the following: + + ## WRITER's question: What does "logging in as stepX= a" and others refer to + below? ## + + + Add any layers that are specified = using the + bitbake-layers add-layer command (logging as + stepXa) + + + Call the scripts/setup-c= onfig script to + generate the necessary auto.conf configuration file for + the build + + + Run the bitbake BBTARGET= S command (logging + as stepXb) + + + Run the bitbake SANITYTA= RGETS command + (logging as stepXc) + + + Run the EXTRACMDS command, which are run + within the BitBake build environment (logging as= stepXd) + + + Run the EXTRAPLAINCMDS command(s), which + are run outside the BitBake build environment (l= ogging as stepXd) + + + Remove any layers added in = step 1 using the + bitbake-layers remove-layer command (logging as + stepXa) + + + + Once the execution steps above complete, run-con= fig executes a + set of post-build steps, including: + + Call scripts/publish-art= ifacts to collect + any output which is to be saved from the build.<= /para> + + + Call scripts/collect-res= ults to collect any + test results to be saved from the build. + + + Call scripts/upload-erro= r-reports to send + any error reports generated to the remote server= . + + + Cleanup the build directory using = clobbe= rdir if the + build was successful, else rename it to =E2=80=9C= build-renamed=E2=80=9D for potential future + debugging. + + +
+
+ Deploying Yocto Autobuilder + The most up to date information about how to setup and dep= loy your own Autbuilder can + be found in README.md in the yocto-autobuilder2 repository. + We hope that people can use the yocto-autobuilde= r2 code directly + but it is inevitable that users will end up needing to heavi= ly customise the + yocto-autobuilder-helper repository= , particularly the + config.json file as they will want = to define their own test + matrix. + The Autobuilder supports wo customization options: + + variable substitution + + + overlaying configuration files + + The standard config.json= minimally attempts to allow + substitution of the paths. The Helper script repository incl= udes a + local-example.json file to show how= you could override these + from a separate configuration file. Pass the following into = the environment of the + Autobuilder: + $ ABHELPER_JSON=3D"config.json local-example.json" + As + another example, you could also pass the following into the + environment: + $ ABHELPER_JSON=3D"config.json /some/location/local.json" + One + issue users often run into is validation of the co= nfig.json files. + A tip for minimizing issues from invalid json files is to us= e a Git + pre-commit-hook.sh script to verify= the JSON file before + committing it. Create a symbolic link as + follows: + $ ln -s ../../scripts/pre-commit-hook.sh .git/hooks/pre-commit + +
+ + + + + + + + + + diff --git a/documentation/test-manual/test-manual.xml b/documentation/te= st-manual/test-manual.xml new file mode 100755 index 000000000..9d3c0354d --- /dev/null +++ b/documentation/test-manual/test-manual.xml @@ -0,0 +1,103 @@ + %poky; ] > + + + + + + + + + + + + Yocto Project Test Environment Manual + + + + + + &ORGNAME; + + &ORGEMAIL; + + + + + + 3.1.1 + TBD + DRAFT - Work-in-Progress - posted June 16, 20= 20 + + + + + ©RIGHT_YEAR; + Linux Foundation + + + + + Permission is granted to copy, distribute and/or modify this d= ocument under + the terms of the + Creative Commons Attribution-Share Alike 2.0 UK: England &= Wales as published by + Creative Commons. + + Manual Notes + + + This version of the + Yocto Project Test Environment Manual + is for the &YOCTO_DOC_VERSION; release of the + Yocto Project. + To be sure you have the latest version of the man= ual + for this release, go to the + Yoc= to Project documentation page + and select the manual from that site. + Manuals from the site are more up-to-date than ma= nuals + derived from the Yocto Project released TAR files. + + + If you located this manual through a web search, = the + version of the manual might not be the one you wa= nt + (e.g. the search might have returned a manual muc= h + older than the Yocto Project version with which y= ou + are working). + You can see all Yocto Project major releases by + visiting the + Rel= eases + page. + If you need a version of this manual for a differ= ent + Yocto Project release, visit the + Yoc= to Project documentation page + and select the manual set by using the + "ACTIVE RELEASES DOCUMENTATION" or "DOCUMENTS ARC= HIVE" + pull-down menus. + + + To report any inaccuracies or problems with this + manual, send an email to the Yocto Project + discussion group at + yocto@yoctoproject.com or lo= g into + the freenode #yocto channel. + + + + + + + + + + + =20 + + --=20 2.17.1