[PATCH 00/16] tests: do more testing of block drivers

[PATCH 00/16] tests: do more testing of block drivers

[Daniel P. Berrangé]

 v1: https://lists.nongnu.org/archive/html/qemu-devel/2025-10/msg01650.html
 v2: https://lists.nongnu.org/archive/html/qemu-devel/2026-01/msg02083.html
 v3: https://lists.nongnu.org/archive/html/qemu-devel/2026-01/msg02939.html
 v4: https://lists.nongnu.org/archive/html/qemu-devel/2026-02/msg03245.html
 v5: https://lists.nongnu.org/archive/html/qemu-devel/2026-03/msg01252.html

The recent (at the time of v1) set of regressions identified in the LUKS
block driver re-inforced that despite having a hugely useful set of I/O
tests, our CI coverage is still letting through too many bugs.

The core goals of this series were/are:

 * Add LUKS and NBD to the tested formats/protocols
   integrated into "make check-block SPEED=thorough"

 * Ensure that all qcow2 tests can be run with "SPEED=slow",
   not staying limited to only the 'auto' group used by the
   'make check-block' target in its default 'quick' mode

 * Add  'make check-block-$FORMAT' to expose a standalone
   target for running all tests for a given format (or
   equivalently a protocol)

 * Add GitLab CI jobs for exercising tests for all formats

A sample pipeline for this is

  https://gitlab.com/berrange/qemu/-/pipelines/2258731804/

Showing the 'block-FORMAT' jobs which test the 10 block formats/protocols
currently enabled.

This series fixes a couple of bugs

 * Test 128 check for device mapper usage doesn't work
   sufficiently well to detect gitlab's containe env
   can't do dynamic devfs

 * Test 185 is reliably failing on gitlab

 * Two tests fail when run on CentOS 9 because they trigger
   python deprecation warnings. This will be fixed when the
   python-qemu-qmp module is next released.

Testing of pipelines throughout development of v1-v5 postings does not
exhibit any non-deterministic failures with the shared runners. Hopefully
that will apply to QEMU's private runners used forthe staging tree too.

At this point it is a chicken & egg problem though.

If we don't bite the bullet and enable block I/O tests in GitLab CI for
everyone, we'll never find out if they're truely stable enough to rely
on.

As a mitigation against instability a patch in this series adds a new
QEMU_TEST_IO_SKIP env variable that we are able to set as a GitLab CI
env variable. This lets us skip broken tests dynamically without waiting
for a git commit to disable them in the buildtest.yml config that we
use for disabling the 3 known broken tests currently.

Changed in v6:

  - Drop redundant change from gitlab yml config
  - Pull in rebase for qemu.qmp to 0.6.0 to fix two tests
    by avoiding deprecation warnings

Changed in v5:

  - Adapt for python-qemu-qmp being spun off, by temp disabling
    tests that are broken by it, that we formerly fixed in-tree.

Changed in v4:

  - Remove tests from build-tcg-disabled job

Changed in v3:

  - Test whether dmsetup create device appeared or not, instead
    of unconditionally skipping test 128 in GitLab
  - Use 'driver' term instead of 'format' or 'protocol'
  - Fix misc typos & rephrasing

Daniel P. Berrangé (15):
  gitlab: ensure all meson jobs capture build/meson-logs by default
  tests: print reason when I/O test is skipped in TAP mode
  tests: remove redundant meson suite for iotests
  tests: ensure all qcow2 I/O tests are able to be run via make
  scripts/mtest2make: ensure output has stable sorting
  scripts/mtest2make: support optional tests grouping
  tests: add a meson suite / make target per block I/O tests format
  docs/devel/testing: expand documentation for 'make check-block'
  tests: add nbd and luks to the I/O test suites
  tests: use 'driver' as collective term for either format or protocol
  tests: validate dmsetup result in test 128
  tests: fix check for sudo access in LUKS I/O test
  tests: add QEMU_TEST_IO_SKIP for skipping I/O tests
  gitlab: add jobs for thorough block tests
  gitlab: remove I/O tests from build-tcg-disabled job

John Snow (1):
  python: bump qemu.qmp to v0.0.6

 .gitlab-ci.d/buildtest-template.yml           |  30 +++--
 .gitlab-ci.d/buildtest.yml                    | 107 ++++++++++++++++--
 docs/devel/testing/main.rst                   |  33 +++++-
 python/scripts/vendor.py                      |   4 +-
 ...ny.whl => qemu_qmp-0.0.6-py3-none-any.whl} | Bin 72263 -> 72379 bytes
 pythondeps.toml                               |   2 +-
 scripts/mtest2make.py                         |  30 +++--
 tests/Makefile.include                        |   3 +-
 tests/qemu-iotests/128                        |   6 +
 tests/qemu-iotests/149                        |  13 ++-
 tests/qemu-iotests/meson.build                |  59 ++++++++--
 tests/qemu-iotests/testrunner.py              |  18 ++-
 12 files changed, 254 insertions(+), 51 deletions(-)
 rename python/wheels/{qemu_qmp-0.0.5-py3-none-any.whl => qemu_qmp-0.0.6-py3-none-any.whl} (64%)

-- 
2.53.0

[PATCH 01/16] python: bump qemu.qmp to v0.0.6

[Daniel P. Berrangé]

From: John Snow <jsnow@redhat.com>

This release removes some deprecated warnings for our use of the old
sendmsg API on older python versions.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 python/scripts/vendor.py                      |   4 ++--
 ...ny.whl => qemu_qmp-0.0.6-py3-none-any.whl} | Bin 72263 -> 72379 bytes
 pythondeps.toml                               |   2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)
 rename python/wheels/{qemu_qmp-0.0.5-py3-none-any.whl => qemu_qmp-0.0.6-py3-none-any.whl} (64%)

diff --git a/python/scripts/vendor.py b/python/scripts/vendor.py
index 78058183e4..127e442659 100755
--- a/python/scripts/vendor.py
+++ b/python/scripts/vendor.py
@@ -43,8 +43,8 @@ def main() -> int:
     packages = {
         "meson==1.10.0":
         "4b27aafce281e652dcb437b28007457411245d975c48b5db3a797d3e93ae1585",
-        "qemu.qmp==0.0.5":
-        "e05782d6df5844b34e0d2f7c68693525da074deef7b641c1401dda6e4e3d6303",
+        "qemu.qmp==0.0.6":
+        "5d7c5af0e9de427696e3bf72e333965c3a697929f77f6b7ddc30c989fc7b539b",
         "pycotap==1.3.1":
         "1c3a25b3ff89e48f4e00f1f71dbbc1642b4f65c65d416524d07e73492fff25ea",
     }
diff --git a/python/wheels/qemu_qmp-0.0.5-py3-none-any.whl b/python/wheels/qemu_qmp-0.0.6-py3-none-any.whl
similarity index 64%
rename from python/wheels/qemu_qmp-0.0.5-py3-none-any.whl
rename to python/wheels/qemu_qmp-0.0.6-py3-none-any.whl
index 6372b7544ffb015a5c77d9686e921b65c848d433..5754130b408e2f13081b368a5506101a6ac490f2 100644
GIT binary patch
delta 22133
zcmV(zK<2;4v;@1g1hCUQ4O)tRTt}0SbkZjP0AiEiJUA%t9NS~b>$~UmQeZboB5DE*
zfELZo<^J}2RRy4d?j}V$lkD!?>W9RWKs_aYZ;W=j&=tF?Hydr-U~s*XmaTPJSAuPf
zzLTY}jI*VOwJ>}ww%XjXg*MCzRZ6vFT$Rj-LdZL*mV-47VVS-YhH=F>Oh9U8oe_LJ
zWj8pZF!l|T7F*4&GrVxDsJOKQx8jb;^`;VQp`2xIC0L=2U~5sVxRQ2lnO-o?EH=VZ
zxjd#x;LH*BWB^l>mX*TFr4l7`n#~0hpPW#oD5rzL3&k9)dBclaz7!KC9dxK+dpHdn
zXThD+3Reza-~8o=H%}`ml)D+=oTr16las;V0ej(-H$F}ag`53W@KTtG|McFIkUmL&
zt#Aay+A1#Q>g7|K82#jS8^AmN>uA6jV;5>Sp?|<S5c#>&#(%fv(!o~J$EUoinopik
z=)+5C9sTR|1}4fY`gC1yLi_98M*I`^kl!h(75%f|&~-eZIZlN!+Jrg&<%c)V;j{m(
zcCyN+KiAIb^=!qf1wj*^3V}12*?PTy=EkpXQPp-u|J)ei+-?R`Bj&ugrElxaMi>ug
zJQxhF*?E}pXm~vw4`%PuZ)We{t7~p=FJH0qxHBI<e4I49_TiOQ;^Rlg9x#5(+06~&
z3M<6|(O@)Yr{6JXb8`a}<%ChfU6iH4!C~h<SZ5|oD)({lPRUQ<H|xyc3eP+Xe1rS>
zA-0*Y4BSVP;y)-Ky}X{ieSY!m`*)WwpTk1`;ScN)`{wI!|1cO3^|H5<DL^)V7tdZk
zXXote&<S}Da;rjEWM{quAh$M5SlE4OSIA_;1PIpC*RNhZe|r7m)jzUx_Dw#GI4sVC
z7Y=x0*8A<dSFfn+xBDm4EI_K|G_(ub#MXQVywnI+8_!yAV1+kqzGDTiDwy?617tM5
zAt29QTxGNWyLbdosU=d%D3DBla}+dnE`c|1SAr%im2{F<aqF~-Yglp17F<@y=#g@P
z(?1V{#dx*lJ4=j<adu*~cC6sF6(<e(F}#TiuGm~45D5rr^pe8{BMUD}NvwiNwa{jb
z_k<tv63+1xhbv@4lTOaqX~vn->QsD^)(I6D;Mo~<hCdEO$bMRHrePm{hf2HILf5LC
zLi6D`wtbN8;2FEb#ltWWm6ClrOZw%1AD$bdH2d}=d!?DJixpeQN&rfh(ul%ov!e-~
zZMflUhOBPpFa9(3=X%Z6DG(!3CtiPW2Ny=paosCHeWn?dPsJI#UI_;0aEj;99K$q2
z0^mHe865p=HnO5xOxkmQKz@S|<9Uyp1>1Tf%xFA~FtXm*G1=A5iiow-x*C|GT1>mM
zVCVVNQn2q0-)uz58pLZB+=82}1YmwD2FX@fXZ%heFSgD!izUFwY)1iDvpTt8DvaLr
zA*^_@fTJ>EXqnR4swW^J(K*IRDZUod3_=*WbC>nZgZKzroc!E>-N;|g*v|;-my^gV
zu7C+0<P;Z*UA%e03cX$f9~^nihtm~B?ckVQF=ktCnY6YBcIiM8m5OgRMsJJ+Hs>U)
zs$n1gTDQz3jEKikDtRkBQt!^z!mZA3{%<DlP8)UzP=HaU?5seEl1Ym|cm+6@aP;Y~
zraNNZcT!ipp(_u64}T*X<b|tw#cl$-zhPFuLLCBuO@S){{@oACh@Wa{L`l#RFyRN_
zN`X<M1nzh6O9(wfIwBfcNf>sL%qM*J8SJ11V|;+-_mVp{rOiJlfcZ~xU(Vw=#}c?F
zo$h1?gTZAg>|G8gYz`O4d6e^})9D0c=Z|9`__pF9)g7;YWO)jtJpkG3nN~x`-f$(0
z=>TTt=Q;{=p0#!!XWK}XmmeqW@?%b?&qSe7RzE|<4HO5?fOK@ptOkV)ZeOh`k~eCg
zP@p(r9G5D=t(-X8Gdh5WQr+oWL1;(=dt62uPgu<vdmjij_D^6+i@bC#Tn`o(D^c8z
zP&GYc!;B|?e1HQOMk%^ES=e`W4*HIuo`9q9<!qKRaQtZ+1XFf?-d=o0AcQW1!vMmf
zrj4kULlJAcWaq%IVLrbgFRP6a$~6)w$c-nlm#7Ypl*xs?gN@SBa`E(}%LCv55cOCp
zgu;1yf^nKeaxc!0kDjR)pYPp}4*MokNv+i9?F`I+^4`HxeFMjlWhJsf+7~>xRv5=F
zd1Xa|4sDzt!B~^l^d>C~8z;P~4D>t#<?xJMB4S^r;%3tZW{zf(_yt*hESuM|a;Yxi
zR*CQw8P7QBy5K0AK4D_IoF?NX-ToB%Wt%%DT~_PS_zcW6Nr$bhDrUr{;)P(th1Ns1
zm2TyKd4R1e2N&TmUA5NZ;k4P%KcRwwo6(>c?$j3LKXZsF^xSRm_lTRNJwu}hd-Ujb
z8$aS+U#MMv(kb&9_b>xe<MHPS!EV5g?^#J>3z*?Qf5obP%N3Fyb=*21u(zH=G?*z$
ze>FT|m+btzaBGS3!{x`M<2x&&&6C{;gC+x$RYg=4vl82{2FVx=#YCAt!<3P?%YC^1
z*pt3RTYoQHHn(1zyBEbR@yBR@mzFK+syZ#f0{02KfOw(=2;JOByr_*46`=7GCZ}S`
zq(c~%90DF6gZDrxlpf;<ekWhJ12%Tx;GBUyST1DIY&dN~@G;O1be3<?z_QX?y0v-f
zcR8H{2zFMmb!}ZR(Uf8#VJ09Q&PqKqqPPQn3x7J}7XV>2zJXf|E!FZ)R6D<#*P!y^
zr2|G)up-?F#)CPrHNgp+*A9;o%tE+cTt1m_C4!1Op$+;(j4Vr`f{Evp-OQqOX6CJQ
zHwp1x6GnFY5Y}<y9U}(K{ExS2-)D~7Tbn527tN-(K%iVrGxE|eEF0{!AQ+^^4YVS{
zc?HnnI({pY97kUPNt1j>H30&XvPYl+my=USDO2Zvzw#z#_NJa!vLK`Rf*0c6{#~<A
zW+#~tibn4oIF>%LbJUv~sWLe*7@D249xo8K?oqnYjxy^RCU4<&b2N=)KQx1*Ehl_P
z#Y=L9vlowFx0BCE7as!`^3ee7r&`o)sJeDbE!FZYc$TMsfmR>D2iJhQf|CVFv;mQm
zx=B5MTcIpFXCE5ACKm~7^Ty1aH#R+|pctxhv^as*=Re1eUzuMLQ(C)mI40jAI@9si
zJ?U`={qg6ppkLU}LpGfLsHGYa*zqslc^CYjz+oed+r?di9?cGAe7spsxNw=j(3LE1
zaq)!f*(Djpok7wCHpy>RWFMsjK|m;=H55sI{FRM#i#IaJftxy^lXM=bHGf8icU&R5
z*Q{{Pix1f%d#}ZwyX~kuK0$|E9*kI<fe{9vJYC7E96|fkj@mXk_Vw<$c*DJ#pDo(&
zn)sHu3jv9m;|`1>J83vzZ=z0(_Wn9Zq@lg15eD+}YE7jvgtd9mJrj%nexI`08xyC0
z=rkip6B1v8{TkSP>i+11Np`Omj}*XJrtr(CLvk5i_OEo05w*1oDHEM>ReDx8>LA1f
zLMV)|_KYRLK)H?3+A5WBVnA5NReKOV-HHnwADy)^Rs>l4Tr6db8cBp@l8bCY@aPct
zYBGkF*hzX$(z0dE4OdP?JBSsIfgvG(y(cM7X6hM;cXgpPN)*zHYR6!I?|22M3WrY?
z@TPx$Ki#l}_ILGsx8L#Xw?C?UtWj{n{%2c$L0`fIDhy9yT0LR!UcLBVK1_veIA}VC
zT|a#jKs*Ms(X<U8Pw(pLWe7V2n6(u>EcKAft?Oy?8`<uiMZsiidLO*?5dkoNj+G+2
z3a$Vs7_WjsCVJe%AH@Jf0TGp3Fp`4704y7YR)rBDWhya+psKaD;J~!P<_HZ7+WPp9
z7QDG&H?Oa7_BSxUo9C}D>C+T2kY{j$=6YI*J3&YZc4IW2Sg^8Hq8&)4#I{9)xc*5#
zCyaA|ATM57#AJ>kuYMwf`<P9ChN2!hCV&XzzSaRZo8QPcl`*3}AKYJyL5+@OM6JL3
z_|iQr=?bRc$xDWiU@-NQteJ@)U?(Xx;(e6IIok+f+9FACo|A>}Xy_40sRhy3ywb%j
zjuyMqfHV%WLGK_#pK<^o*K1LNs!{E{b#5*jyLB@mV8EW4OxXnkLv$m54j^==k6>uQ
z_{-M31WPQ_CN;7+%@$Je3Zg_QWU#aT|KC|6xfW0hDs)|uqrkS1Q~OwXzpj5{&bUY1
zLFz^iZP3n;xy|5WE=R+|A(6wD*DMvDz-@2@PrHwAH6aeivaY5_n}K-(h|=Z#qE9dJ
zf2I)h4s#=rsyT=(cM>dr4S`xavIzvahS3wdXO{%;m#$*>LsC|xfG;JWX1}{+6fQsy
zp|e;BCZupm4C`A+W8Hs>C%=z)qB8lUj|zf)BN|=c%)17T{HRAKPif;3EyJA?JXtvc
zq(MZ#DJSZYmp*?q-{jCx9na+NETV<5t(J5~St6;77NiLJ-^YD_GTW~d*PaGTtxj*u
z-&2R|;cmytE78cT|0V>M|74QC6_fl;`BtFkeZtd!FecV8swBkp2yrZN`T+B2)H#r$
zz^#Nj3np_}r#EfxA&fvtgR5?y5}Kh8yJ?&-YpGzN8za^jXIAYn^EIHZ&uxH#x^x}g
zV|(-n8W@m1#DK(q@oFVLZ7@Q$+D+O2)-}*;28Djdpb{YxssS#Ouw1Z)T2^l+Oq;BK
zLv*m84PKu5Fegg&u+!E?!fk(uBsWISE3uxio1lIAQNoTvc0@ltB_{7FTK`8USv)d{
zimBw_alcpz$itqct$WS7@Xx#zHap;D@3`r&Jo{YpVkMP-FuVI95JB{14d;~6B4VG!
zv9?Rh-JodBEM+(KneNeM5IDkPA?T6(FAyfC;Y70A6739X7jn)boY(YqMCeJ9A>((V
zAqjEFlKTVP5D|`zy#gmjo_3SoX*apa<>q^CMFeoe&_dt9!9YP?X(iYQuP>>JsxFf-
zo8UBj3a%!9z$hGLf`FPxtv5Tj(u)1v^uLX}B1E29laa%{sYP4lK1mMd3PRSh?8}6O
zWQ*-#BkkSrH=!g3g9i`BZm>>$w&0+4OjBH5$m9~L&@}|8P{`ncAyOoCrWJzx0Fw=b
z8G_fa6y&NW7MeEONl}&Y8{*&jhCtx~Va#678729Dj7}cfGxpG)5H(GDvh0L0<MCc7
zo*6x~XaB4o+J9CNP&=`F)B@63IrFi>Kp}Vogo`wgAyCP*(<R|yy*#ejpT6PVLv01f
zM)FriM#(mbMQ-leSbd-i4bEx8=7Lfdw*hri$=Ra@>huir0lG_i&<fuejT;7qRNDlJ
z`hdfK9uS!7M1t%oOoJ7b;Hut)N!E_-bj_52oA6fYZFe3aVWw9NI(ZkbVPAKEfkcxe
z+p<0BG{kz~z639}-;lrj@P<`#Za9Q_@9Fra62K3Ppn?#1!T%AW?@mj&U;m`ETB>@g
z&GI{xi1vVa`t*$b54}?CO6e{8<BwM9?VpT)C|BG~3%&lXCk<%a`6i`-!bm3zUVRA}
zuwW_}bl5=3f_;6H=ZSs*@+}2UF)P+wISH}q9E_|_TCfny*{j#rY|F_Rh?<IouyKkm
z%oyIZ)&14?XFbya4T-mSj<;tt7eodL6AeNj1Mro}o(pH(CDq<Nx`=G}R;m)5@WhUP
zeUv9eusJHhV)a_YMm=$%1TW}JbE7#5moIz`62Q=*_*4wp0vInU51%$wh(d<c>lP?t
z0aCFgPQv)m{po!h(wAtS?Ll7Oz%y-Q#>A_HG3`xy+r2C;tJSHODo|Y64WTFwEQjie
z@HAGfwf)^FnN~<Zqq}4=NNMU;_QPm@?4wOBv#aPgm<v|;^NWIqP>?%OV4S>8!6_1T
zSf%v_tT@1X0SSb%aGu)$3k&n8PzMOA5ClU+nyvLTw$<vi6qu6z2>9uvkh(k<D`7FQ
zgDW<SV(KtU{A5jH{GRBt?ycHs!_M2A6+87tm4tqK>}5%XnCAGbaI+@%Wzx=nd%n+{
zg#)(jGCm+CCJpp@$*v$nsj_*bN?3sxU-uhfgO;6dz~MC29?|6*(s8i%2D&GKiiQvE
z5JS5k+Trx%kn1*<Lx|&#C-f!$yRD8TVTe81+wIUAD_kc|u+;=EbrW>8&grK5lhd2j
zOMS0PS#MfC-^bo;4WE_R_IY1_>^&CqCbOx>Ufw#Xj(pg2bTCUS;zWQ`^_R6GE2l-J
ztr*=a$i^s64a_lxh~D3-Z&G1hX^drh7Qj>fthXR(p@AfN#vu(oikNR7zfZGnlCz6J
zqez~6u`z^w2Y!)N@7mid1b*ke-un>VU+E9T6htav)^MUdfd;rM&~hz*6(&T46ti9G
z{bOJkuNuPX_W?RTvW?0^J_UAhb@7JT#F~Xc&q2GltgH>;6<IWVeIWU-Ny-ywld(T0
zS^|2+>m_J$)^Sk&+G=t*tQTSaH+{!;)9w~dYhO1yx0~MMf|v_-?m<Ab<xG5XhR+~3
zaWsTqAzX4eVZ(O{6as}FJ%-~$%=1Oh!6QbaN(udwjM&Hr8slD;e9#hHu0WFkRM~&a
zqObl&qOQC<;S3A8QQB5?CD4Vl&x>}CU#TaBr4i<PV}A}Y^NWBBsaI67hgfv<B+}46
z`s`6vQr4eny3ujxAib%j=|Uxf1XM=L{brabAax0PrL8;n!+rR=$D<Di92<Po)4<y5
z+}f)914tH?UY>u`{i%{`>CRhue0hJfL-`H8J5Ve6>5I1kP&=&hcU)q6BSCC=W7FMo
zSFFgR2&OCrlsRnhbV%ya6SUhTP`rUA8u5vi0A(w_gCGhBrb#l_&@T!sEaiI-$Y!(!
z%0xSQa^)fKjT>JHu^D}xtpiyjf~3I4#GN<-HaPlrJn;->^zC?zvM4qO=XrkuCyfL8
zb`12f^|K*v9YqY{(!n*GdjeY82hAR^=Ma^94JV7pL}@LO_YpVj@j-(z9fit>9?%#P
zzWJ8IdMqt^D@gi-%W)@Mlw<b5X*9N=S|#OdadRb9Pk#MBTkc?#Y%m8kx!OWDY7iyS
z`je=3AY6qCO>tgnT4Xyxy1jotUnXTmnW&Y1lR?YX?fqQ4Ct)&2nWu?X7Y05#3Qzvp
zp;zqJOJx^SHr3bV#$qh!PoN4c6BZI&^XMQFX2_KAA0TY%-=oCww{i*Wk!3Oj9^Y%b
z3N!_7Z}8<{^<K0&Q1#wD-mB*CWKo_!EGKNi*RtB3KQt3e)jfY`4qAUKafLqi9t!d*
zJsMA%ZxC4X86f7$X29h}KG7SYW(}!yrIRCM1$IabfgS0WURu@mjukNvABWQiMBXpl
zf<q$c7fGMLQpCJ>2LLtB9zXt~jX+WL&~AL8IQn@q=RuQ05S4I_M_hf!U3lFX!6O8u
z=P?Qq12UT$>$x2vEz*Bvv%*MEn62kn`%GAj)AT_B&_Onm3>fs3SZd5D*CDC|Lk(g%
z_7Eu)O1syaw1#wDfI)Uqqc^9hYI~GWb5sILb6}xlNP<*YGbGg}eH9R|C&gMC;QgC{
z13H*}@LQ<c)l}gfl3_KmYWj*G#{q+yUkNZWw+{DU9Tu!e)<J(hNc?t`ETOGWCnZYu
zbWWz+89LMyNxCjMTy@=)>1>{po}ZJMG(UGn?79Cv{<I?v+jnYLCLp&~Qr7vi{Fpt#
zjz<vSi$>sUIzC~t6zh$4=z{FEz=d)RbM00^7QuO*qGo^*p;YQVRR*m1sTSbF){EA|
z1`{f{suCT8AklwU4G>Ny?OLt4DyuYXJjz79#oU?|hqnlP(PHqpvo(jR5EO&gDjgvA
zJN6pyb|}!&gGW6WlGW+G-z!Hai{XNC%lh4Yej0lyC(=l|^Y=6Mc5v*Gdan&uRiEDm
zk=Pzb)2BVWl~dZ)9!x{FOjL{>iQafC>&rQwQmjUR14n-e&Jxt5(P?EuPT3PI8CtJ>
z$unP-&&RV+1WR*!prQdc4IywEs5<q<8}sUgL>w*L#Vw|Zhmsj!2q{)vs-uI%FU(vz
zh!jf#*TcF?hG9tQ2pKJW;eC~%YrnSb5ifmo!=B+&Kp&bna466P%M^Xcx^?U4G}WQ9
z#q+J5QfYt1(btn!%Y@%n)yx;L+;<&3;@iAj?Q;s2Ny|oAFhYqZN!;0pG|q9bP$WVA
zXUWP_ZjUC4QMTI2I2pO)>G$0`ONhZteG!@oO*ChGz-RBnA^Rc%`uG*)mG$xoD$MS|
z9l;sseI4TiU8N1R_Rz$DaC~eF(Q~LW-cE5!LE?WDz*L~Lq}`ec!>t&#N@fjPHM3^#
zJ`O)-8_@fgS3l|n(sthdSyUSg*@9{KLSqy5{MFUFw;)*318Ix=!J}59D{CkW*AWa^
zV=$RunKMAnUSBA-14^Qfu#riJF)Bo?MXg`P0@{~O?<r)B`kkUj^iLWd78wt~NE)_b
zB!YkA0RRoF>0QR5qI-u+m-SRsgSgXfKwOiYhrv%EfHeWv)}f&XvL5k9OQ2!FUy8kk
zn3Q<N0WL|N)9Li9`!ip<xi8-n`}s56J9KN?W1#d)vx(%9iz=U=6;@ziZKxXy#*s1>
zSIi~CZuRAme~z#5fdUHX<t>nnJ^+<9^!$IvPGOCJ?>AW5HQ3Q*Q+731D?CRqLd05u
zKH(;WdBp0$AjyE9n;dp>(p2$Rs!%z(KbtHsZ~QA{ft&{;gSm(4V)1&at2WoV>MgMz
z_+4-Uv{_h6CNJ7v48PYC3iC)yDee!{NUgLKF&n%ZP$yFPz-lnuZ29iL!<c@1y^w!J
ztb?1>P#YkWn2H5a<BP<X<GWz#>s&^A!nP?GA0d!L=(c#odyEhoI($y5;bU0vOcYe-
zI3_8!N?yW?L7PhWFbU5{B;i-^H{pa=P&XWzMS&P<YLzZkl|ew44)?baZ#+{)L*!_I
zC74H)oz`W_lkTN3PNGODxyqjcFlv9qhHe+w?DV%@w(qsK9vL$q=Kq`{o@T?~2+6l0
za%aC#_kdkfBp*IL=`)O>6FM<A+FCiUL_75WwV+39zI)^&>LV(`jxV$+K+m98Vj(EJ
zc@++EwG(a+3(i*8jj~T^WWJi=bp84gS>x!GtM-vyo1{uEYqD%kn!M!p4&{HIfPY6h
zT)C#%D{O~E7SbMpVdn*Y+2}pGxe-*$OdmC&RJ>_3#l`@UwDMfCd$@fTUP?VdM@(}$
zw2cd_Pu7~!?b;GSTVam(O*?7zh12%8+u>=a!^CN~j|yWRb?p+C4kbt-wfmlf>fs`L
zCjLGNIBOU+abbbphVBx26v}_L6iO?2$*lwb6^2PB-oe0?+33*^d`F9T^woiM|M_Z$
zFlN+$l#%hhr1UI(w)NzQ@ccHT*(2y>2Dh}7b&m{j(4+s%aH+y$3-Lh#EHlzNf^<`Y
zZJx^KW;aPD=<lui+cT3(Saz668pPa#3jX;W=+SrY+_S;idD`R|z;J&R;DyG{=`E|C
z-S?Srm9}BNe!W;218Qu`LgLG1FjDHU*^>~|7x&!ruv)vrR9ajN){(y7ujl8R?rp#6
zv=w4}2!B!{&JGP|8EG#+y*)ObY+N+5V&}4Z(Ke>QzLyO!Vegchwm1U5l)i!_e~=CY
zJa@53a4mab4aaU5gLp{*`#)vR@x_P*+EO!_w|O9V5(GyO7+WxWCNSB9S8}-$<`lyf
zk~f9?1McXPYjyWrjLPbf-ZK{rysdbffX7c1fO=prYYR~?lM!At0Th!)UT^{Rleb<p
z0oIf3UOEA~lP6y;0lkw}Uo3xps57*m0x;F88AYh_R4+~;)~0tDZ>Tv~!-sX~@YhrW
ziBm&$5@z(p>SP4v<Mj@6vF48ky-)T0vaLCmMKve%Nz%)(;dqYWQfZLLXNE|#82%E2
zOt4r#8n0y<iSR79)p|O=>2S&M9+weW9eSnvWtMk*P;|pL@YZ&DRGNQm900uvsUh>8
z4NXIMK6gW7fOx2-NLKNx%Aa(jqE9uZ{r{%BOwS1Rt5?mFYPhA?sNaF$e~Rvh@CZ|Q
z3~Em=+U%o(X_B|;{7sA1wytsOulJM!@b4X-b>67MH{{yU4Qa>#KA6*lvC(#8nJ93J
zmL=oC=^tSLNb6%29a(?(*Ida>UEu>wEkdF<9`O;xMz_kJMG6^msE&eN8ee#WjAXwE
z2-F4Z2v^N4;*gjvwMg4{Z*1H`!?&INX&Tbia^}5Md)wVp9Pae=J7VEbivW<RQ#N94
zm!F9a{+Ylpz=qZXgTFsAw3)!aEi!bn-a)}Xp<u3)?p&XE-xz=QYnvv3gik?cCqk7Q
zEtNaL-Rl4W0R4|P_MvC%z`EG!qw_Gp=Zg^ZkEf@9H3UsA3s4d(sghe_zH`FfcTGUx
zEp%(bN#X_u15WSh(c{1H_-sk>euY+dLP-qp1IQE)h>Cs$5Y&AO$_t>(j-K(A%0Du^
zXE1VugqNsU8c2VLi)=6wvRY*J*h^TST!577OW{#66sqcv0thH(8DA=evbCTPL@*F~
zNc9aBWT7M}!Wfs6)u$z##aD+*)NdMzVkLa4#@{DT`2v_y6!40y3}rR3>eY^D(ATzF
z)X6iP$QVG7pd1R$Ht>clulhLf=O@O5D70R&{zV#G7Oj8G7Age64s!}NfEVx#dOK-<
z@daWyz-gDJu*Jj|WegTZe{e_Sj`^ANWsa@waz^8~Wf8>fNHpV)*)jIq<C*agxSI*R
z=<{6UTR7+;a)ST)q#mARj+DOV8VV2SoiPagZYbOzSrTN$d(h^egxRr5Z>faqlPty&
zOYAjq67zo@y9uvE?|GFCH{+FdFT!2pHQX;@bVn@xEj@PKT$c^Gbyqr_GGC|SEI-B;
z!;3qxF>*~0fCHzy=?LoMn+y!4tod+)?0rdekfD8oy|X?T6@pcO73f6{1m=SF7~~fx
z)(KvnbRMP!iZg47^1(;#ml=$;0586@>%gQeUx<G&y;y6k=Rs|V(as<N`^-z0q0J;_
zmhEBWH3&7?o8rV4j1+f10GDqY1(2(z(kCi|^nkoJYl|iPaLdxCpIdi(;4c@xb`JAF
z2w<>|_xQL6HTqC0NI&Q8qTDULCM!;sOwIS-6H3`Hdxp2xJA&tdB+p3nXlG1b;to~!
zk@|ms1)=-b-+yyL|J;>dfu-PBn(qXeN7SMcw{0B?(tow_Z>vWr*+b?z^6h0RFXgRh
z*rm5R_b{0(5^cU~F$0mh?1?X+B{iV6jZ0{8fS?6DrAP6wDi@||R6F$G6w(y++Se3Z
z$|Ar0nnU%E;QDe;D5KD0dr)04*F#e>W@&%rs^;1CgTjih9!_Fj`(5yW&b=gyHi$@x
z@t`uZO{C+-qH`~5CfdYjAm~#%OQ#uWJYI|{gxpX?Ma+Why`KtRqq(E!e!|BXL{YmC
zun4=9CBrnuAwiie?;viyg-+bXaya___(A_Dz2Ge{nw20>LCt4K0+$p9h*`C2*Pm6j
zlRYla&^6$Zv8g&h*bTc@Ia~%JCSvOxY~j|O-nQMJ7*ZbUsX^pEhSew0R=pbF^{T5e
zPof77^|<nJxI_AJ?g22rmCq$Dbx2F%o-LyyF&+Frlh|ZC4F{oXTW3N<_nI0201%TP
zWhfVNO-ahhOftJySEZD$))RYXeY?)X$?TI~Wf)5`o7<b&^M^<jfd<g%cQ=sHXms@Y
z@)dcdq>;HS$wgI{tYX4Y!=*Sndb{JARI+GFMrtK*dBHR>J4RNoFJC<`ITL0@@{(#j
zBX^U+WhM_YMRb$z2-W2E`Q?updApPMWmF~KFcY4PYI1XA0q$!U$Gw>x5kklr&&>?K
z{e^4y?UL5e&_2AZlW=Ake{=Y81{lO|zUeI!M*Fts_~c$yOl!Jj{@ta0^e<(>%C;Xs
zykfeRLbDn9K`~|k4YqCVNy)W71)TY5&B#?Fw4sL0h+^*>uKW;<;bn@TwKPUn%N;E@
zcHQg)>>QV3shg^zYHxq86*FePG>YbIP4gT3Yg5-uxz5w0qa#a4lL%)LOVhOm$fh0L
zk>p0ImThnkv?ZK7MQb3hbKfC5LP+zPDq7j5ONZ_``BPI-aiSP4=(=P;H9#o^)4Aen
z_zKL1eX<0~hkTAaqNFq@lVxW#C__#b@cU6T0d15Ue@agN6UJ<ZP|P$+kj?0$?H>>D
zX-+<RsK3*nMw7v3Bufjnm4>4%{))?tp#ExBdj5gsjbV0`UTk$17O0ab%0aTP_mLjT
zm<2JJu80)m>eX|yp}cIAUmohR7BgIXFn2p9a1~AyDV8&S%abr@9sx#^PiP!}prD86
z^P|8b^p0{v-dA;4=6GGo{N|+K%Ir_p+*Gtap+zyCg+Beik7FSVHlCC5M#}Mwj0LUm
zvDz;!i1DYEnUBePRy7$6`o5}{xkYolQ}d`T{5_{aNJBPU6r^CcJZB3Wi3q6(41EFv
z+TR9fPC2Dw2?uu2DmVW-k?Ws-S#A<K(@YrK9=>5Gugg#zBjpvi8-X06?tZTwt<t+>
z*t*<@7N03q+cl(mvcD^K=9gZMwli@*JuHJ&P{IJK1rcpwgqoFuyE=rJREj6lBMXGl
zh3g0G!sepjIW?>p2HjgKSp-nsA!@2%nkgZ4^bL|?if>Rr363ne{jpYmvS!Nc!zS8R
zD1;7Zr_pHStx-D*Z0rdk(qHXl+`rgx`Gs1r2L(iJA}23EDygP6toU+y8A-_rGlZ5?
zEB~k!nOs|;XTc|u)7qJ-)TAPTh^8I54<wgOSr8BgYr9*y1d#?X9Xzn<Q9F?}v+BW0
z%wUT<p?OC&0x(QfTp&?@t=UE@MyTV^JE~hQMWQjPVV1y&+yGpJ5Ecf|?|4~)qe5>f
zFHy7Ia!OVO*KP`v=_)MW?tU<B3&}LDt)e%Ka6@XTH3t|*k{coK2;a22;mC4uo-<ak
z0^AZ{3eW-@6Z{@Q&^Ui)k${rcPEjyaH3;N=!_+?3R7R45S;Sa>^x+1$HTy<VIDL~-
z%}&<)6Zj-+s`-G<nUVA7FJHX4I)8KimQ;MZ12;$BkdoiP2wcZX$*BE2k!u553g>)+
zx@tU48Xpf5^PHT8Er_In)!V~84oo55%Fe*g1C?Dd&5RrP`dld?$<rr)co($*p~DRD
zPR*F=87|99z5OL1?nEm|(3Pl%T!|gM1uAcatJi-mSDts=60WV&I|TD?lLl)<e+%6o
z93bf8M^tU~+@bpme4$EjUFV((KBOv#XCHK1JTJ0<0lrY3jQ%D@+0T-T$#ud6?NTg|
zX=<TtMkEY-M>RNMMos*brl0;MMs0_8t%v;vWzH*5+RkI?@<)dw05@-31PFv)OzzH%
zK#z<n4WFq=&oh>hQ>O-YR1>Nce<In%aaafj0U2`x>``&8J)w;*&g?8UoRw(rhXH9A
z(rwGHg{VEIzoW$!to!rOj7}klWGm6U+Kk;6qGW@P4&Vy3TPp_K>Mu4fWEY}Et>Ptj
z+@KyC^I~lAi5^l}f;u!-a#;T<8is2kA(nT?HESiv3^h&Ut9fN@O%FF4f4}e1WC8?G
z!0C3om?0o^Y*}OMCD#}+JSI16uQ6=Mt^V}C-_o*SNM0Tow1kEgtJX|uA~)7V*)ejb
zBy9M;{$KH`1Ie^2wMej`r`Ql0c9Ig`2q^%V4xefzD^RLmRgxxN#Ft7^=={}u8Slb)
z{t1DDNfM&Gc97rAT!6G?f20r`*NzRGoz~i{Zp)M4cp4Aygs!FXvFT1Y+zMA<5nA5S
zy(Tn=VQuie9Xq8laIRYv`v5W#JimcgXcVMI_AE@S;F}FoOh90$#U6)ft;m_S=Dv$~
zr=tQ9ubWNDL_4-6(GR?8Dza_>GAWtZnjOvodT5BJkUPZ<v$+5fe@X4vm#?A?v=W>P
zvfdk}Gjhf@v?-ChpFX+#E(5s~qukp-&T1y;eMG5(ZNs!7bc>WPiBVK&nBj((Oj|c#
zg>6pY7i7}a>T!UTyu<~pRxrI>G3qo6*G_0ubdPe&#=5=a`i^Rz6JYh)oM^Kzfj)pM
z!PM46wst20;dC^hf6Iqxe3O6&-xd2H^H!9L$uu^Xq2D6n1c(FOa%CD?E-RYvxL`{R
z5yb{{YhioydNO|c?SEuXvM1To@r*qE_V<67#vQUOi=GtIWbq}QLfZX+eIGBD%S<(5
zVwXGh%A!J-vA6|tF1)+ray^DBgH&k1p~FT7_CszGl;7_Ue_63&F&QIyKyD(Vg1DTf
z&WjUQ-ED#9)1w#(VNdU<$#+Ye*s+FS7h=yzi$MT4VrO74Keo#g4fP_JaN)vCHX{^|
zbMg^eUyt!){O;3e+Ue(9H@`K{VFSu?(0TI|0_u5qy|A$shqNdZ(|R#}Y^wS(vVqOO
zje97tG!vHRf2qbn=+4(q6!_V&rJ?%98omjxr@e0JAj&1Wmf%-yS&_l($vvI=<ot#)
zvTFBZg}a4hTrfj<37);(yd4!qY0YF~m=fNx_p1ANPW~ddTL=cAOVj#@1zR^L)4gS&
zEwh!j`67t?Ei)FxWQihSxpcEUCRdCWUdk6vHkZ8Of94s=XUi)kUxQ@v4Y2F7RUKqO
zo<1Ru$kQj^{=t2C@Nf4uBedkydCE)_Xwdnv6G#+P2ZmKPRJ43$+q@+Un>f7=_Q+5G
z;a%)fIX+G7ht|`vjO#V)wKnP<S&$b}L;>DNA4159EaG)yD|FVN(>?YOj5Ld2wUlGc
zk}MG^f01-Jo|B(;46O_dx?81(bVkY6XSi(aiD^UECD%La@x+Eq1kUMO6p(LK!=91X
zKb$(f3|4xLniZ6u4G(-zOU*idEKXqQho6(RlqFdpxCf?QGX<6|$!Sg3yyS*M@GVMV
zHMJBmh-ej9&j9UL>G-Q`cu_H!FI%6V5IeLqfAXB>J641o;|#nEDbTyN+v9Ea05d{Y
zpg7;%lU*``%Qjt{LHHGjyaUg%%#s&mL7v22FMKTo#DEjGN_Qva_-~FC3~5Zk@)~*p
zFt>MUNMY;+>+6O!?7B_d1$KQmZbJ%WK|X#u6twq<X+s}<1SJ9d#4ug?Hmk(qY%R?V
zf1zTZwfKo?WwnB(fyIaRt8+{tLTFhk;xVoxnc%$SC=DBhx@#rD4bRAhu|74zGd`rN
z+Qf_85JsWQrtJ(Ozp70(xZXIa5a`LFwXKoB=T^xsgIri@rZtzMMRY2*;-M}CM-GVa
ziG3qP{sdCUdm+%Il}Dm@kkcbTbBqL}e>DIoJZvI2+JasrqXVLwVxNOI0We|_15Tyx
zbv=LieRp6Na&Lr{8mz#$l-O$0)V=;M-?;7&f5$17dn{lwius6v9^Te;KfNp%WYma}
z;E!*H1Tx6mjvPy)h28NY!)3W;EuhJ?Khy!3F^JG4?tg4U+{k0d1&Cr$iA+64f5%Hj
zMydnn^=O}jrw@{SvXDUWp2&u&{_9cVKZ%^8ZYUe(Y3YkI5V!o6fx{h5;~q-1)?$Xn
z<YaUTI{~o5;<GFpMH}05Nh;l?DaG>Jj<xJWnV;zd;?SXib228K0=Pf3z551Mqz#Nh
z6osjd=fUh0MrXW;_X)(3A8^#?e+aj3MD1ksjh>Tl^k{~_X8n3Kh;en?h5_t-qvwAU
z-)IQ1J3!qEx32FoK!FoC1*N7|$1$b4xcZK2)~1h!ka8Soa^|TE<D*(u%%IKKQf4-Y
z&hKre(@3uYe<Rp@P*k)6>oYXtE2r2nZui;Bt*@C-F8uoSQn~|S8Bl2(e=oix6sX!Z
z7+iIhC7O{?8;f3+vj=@+uUeLg0Wp?xJr0qK@Gg6O`6`Qt?&u*uaHg4Y?&)OA1O(>C
zGjjcInnBH-C?*KMr4~!$HYY}JSTL_Ey`7A)Vk#PQJZlG9gl~zqyD(#`BsAS%DS@q-
z^KLF=Z)ZudM5+oSNK`Q~e>M;-fr{uXEy5*_TsTrcvf(dE1iSOrS_CDOIRKeC^R_=M
z4CY7r-C+?Q3wJ04PsKNJb#?oP)d1C+Dbpr#xXWh^HgXvpSVXVg0hMuzT|Lv?uPt75
zc###qZn4%cDE*b_iQgI`oqmXF)Vm5|y<0{Ry#q?(5SMr(HU-koe`CW16l$b>gG}@u
zdM~s(uznYf#n@pISmRRYPAC_%o{Myl41VlzC^&dGY!4g<Z@+XNZJv7kjUJC?0Hhl?
z?f5v?H*Ehds~fWv6637B<^=2E25RR<ak)`1c1=hUcORTOX!RIK&}v=t0qG(y85K=^
zh$6z)`*!V&oJ>One-SVKVOo^JCstkTJkHTRV7r_aMRHgsB#<TQg18XqJ;Ze$V=S7f
zzCF_|4RPj)uQKs9quLuQSoxV2?RF;@1+IL!3e^!REv#v<<)e>pX~_$$9Rw}<EKzbq
zigHg%%<a^S-C%si)$#4*9U8AtvViMtqhhfw;*uXg@qpTce~J&A0ZZ4AN*GdsY3AdY
ze8)<8HzRipw!f(u1pL+B3e}QrFkD1^Z3lY{pC?kTL#1pqPk3iTe{h0mk_54lBp}8=
zf%1)M>csnw!LS+>F!4G)Te_aenxC`UpO~+C^JI1hb)0C+;1+;Dc7^w}OsS0Pr7fyk
z1hsFsBed92e{I)~B`WS^ywCswj7gv&+oP-OFe(8lUZ}=|<VLwCG^X09YAEe_VbG|8
z^>DGnbbAR2;!;|gqhj9|!#yBE8_2lGjgOe2X<agRc0t0tX5{Bao3?1S3j}gYqZcpV
zo{s<#SQ#tp$M389MAs~zkrA@!i0E!UfCO8jSL)T1Or4WaQrRWC<@``GBpZBlaW*5P
z9sn|OCn+Fxi+oKk&Q40Eb)=;{#<D>Gr@%HLD-Ya~$QKm^%xK!L(T}Nzq{`r!cw}Wt
zn;98B_mcs3Ivc;bdU=%)J1aE{uYWqddU5gMhZGf~AKkDQXOmWTC@}Y_yh2*vA*flW
zX6_B*1;-Z^6HOdmARQ)LUWI*L^v-q%nltde*C+8jQYjOFT(*<0b{hedlgf50e;?dp
zzq#l_ez7s*@cvh*&2hW+UvFN%ARbYz0}_{zxn3cH_$8fw5lO{Uc2=K7w{^KaEeyo`
zZG#LyCW!a@7$^k29lsu^ByJ6d&1kuNU)9GSyZOZVlYcKhb<-13y7Au@<Y{-)9oPp$
zZCvZ-py6s$j+lUlNgv4iT>uDIe_z*@4uY01B%<=pC#Kw)l9*raFV>azSi3WaeWbk4
zZP4}N%pc8Y0cfn^JH&Im7_K%*jL+7Z=~!fzbh!3ad_W7=(EJ@DE49Y#aJI(WHahC;
zr?={COPETM(VbNEgXVu{i>EE%d&5AoF<S{rr)M2nXFly&uyS}FhrDx=f5?~5p)}n3
zmeLUW-JJHr%H-75TDZboFji~DZn<o9xhF`?MO;NaNNYHDN;7>R0{~)jvlHl)!uJ)|
zgGXwDF0ugQqg{Yt#n~+$${ISwl>mZLhf;#I<bp%C_#4vS>ylN&Y=B;6H|!Ah{xGGv
zs-RG62w>2|Ny#29p}-)vf75W8P_L}a8^X5JfNRrzGZ?lxOxF*Qs}BwLMdJ*3pKk?H
zQB;reF!8<tVb&FtiEu-)Nn;w1=+E94!pu$<<dOAQ^&_&E4Vv%Txi+}<H*iQ83M_;7
z+KQWSj1Z0AN00%26f#KtQIo?B7iaC+K16~&4G<6l+H(cSSYf$^f4f>F>;~6hh`Qqm
z=rj0exeJ!#2zb^&G<aIdkJ8Fy>!4+rKj}3i7iW5gD5=0hL!(an(|b@H65x!jvK@n4
zXwPnO%ZC?dU!tEJcC&7eh(s<nNFJVSTt*<q>5z;$?;I%ktxKAiN_zQes}h48imj$d
zZ<Sw9ANv%BCjytle;Cltt$h}2S8FyUYSl@l)PM(-=O7&Nvm6IZ{N*j#)sHX0OPux;
zD7{O)&a`^cp>p0W%5?YMSS@P{5Z=vfwST9EY*?(>HV+FoZwA6K|DJS<t|Gcm!bR+s
z%<MFHYQrWW$QpMVAUfg(Rb*-A%He`cxX8-}#U-Rer%1^#e{p*V;xIF(vvT|_2vuut
z6jl3FRN^F2W?NH&Tx=2mD=c<g&FrC|4_w<~mcj}zcbWy(w8j|E#7536x22Q-$DDhy
zEkuc|xRa|>f+o0?gLtk34{@a=;9k$*T@azRJ_ygCMPjp^3yz<_8J-K?mg1svq;bbN
zsQIN5@V6iWe|zq2e?F=5^du2fAr=eSf=GMl4(1!S@gDE{b4T*`so3GH$^<q2ECF!t
zsnOzwhl3p{S%Bkh<K`uVFCb5K(&&R3hNl@|_!eaGmI<bK9(YtL%Z=^fEq`xT+x=VU
zUhji+u;NV<E!`PB7L+cc6O$26bU?>;TD`!^bVvmxe{uX+wc(xw>|(jZAw}g{M~cb^
z^ajtYcsn>y&}1{ZI3ttsN7wDscxpKng`XpfQO^!{tuwo8uxS&&X$Qw_iD@|JcW5RD
zWJcBxA*Z>JJ{CVOJM!>sai-Bk?NuB=nK(>tuhwx#2|M$g_B`Gs>*V|dOmBzDFT%Hf
zRPD!ge=i5ZG?wazwX(ZV$e*cnMeI^^eGa!WZrtIzUt|OKWO#zdx6z$V@o{M!*q-R?
zig<h92pmz7spfQrl;O^&f&pP@zc4>cIEZU~`qbQk^vHHcDyh--gR2q42E$z?xmkU-
zcy(pXDQZHA4<;DmGtv%Y;gGcd8~YTaJLC=Ne_AtW>W53BdSz&K3A&#V`UNh7TUJyL
z56CRIwmsZkF&*a|97%iR<pIy7;~oH}CM{~L>;yFsVa5J9eSrN(SKh0PF&)Ek7TGAt
zIXwqe5~l^c8gbe&IGO8x8y}d)Ce$%0Wlqb;@B?T4nLViQk0w~d5YII*ft6Eu@IC?(
ze+q)My&=S9@+w*I%|4uPz#Q#ZlO5a!@Srwp_orV77Rj(wAV;}g?3dzw@bF`LF%(&(
zH%CnnOsbrGsn}<i6FVez(w$Vtx1@C2TN+8Pbr{rXqxLE)i>mnMgVD_XXphkMwOL!{
zH9(-rEqvWO#!!7Y<Rm-bEAiqgyCL#_e_(02tad@}ss$Q$7}&!PWiMVB_1AZ3?R^c2
zEw4R9A8o@!yp|{M2YYV1Q{0)CdPef9us0qI?h*b)C9NTdKjiE7&k*+hq+AD=7p>xl
zz1`<{ezlFS#R=BH-onJUg2C!mq@*VGU95J+1;;8HjwtpW(RR(J{wtZ-BURIke_T2<
z5He^dA%nIq`+#7@T}f}_b~CTe$p~bQ+>kXD#VE4}X1lcvt4=_|Dv8m{#+=B_$vRQ>
zqv1U1;QcIkt%IA0yZt0aBVHdiv@NON`xtAE4J_K`<AYY11KvL+mV7{QoCq8UScP^N
zw+c^PMmU&!?OAxvx_gN7zZTgKf7ARw$CUhkO6>n~*6_cgA$s4rgS;xDg9bAotc8^8
z`(nUx_tw1nLT?bH1VlZTn_{W+%1Me%fk)tnt1?`w39eCb7a!F|Dj)k$tdhoF%J}B_
z<?`y{hd;f2b0E_cyqLU{E-2Q4+Gp^M%dZX$@1wVP6MbhyTW%Sy5!o<mf8YvVxv}JE
z(=;8Jc-LhBvxYlOwf2{kxjTe#SkFS--Q#ByM|f8Q91;&8G|L;s4V=-Y8iqHLa4pMl
z_!g$(&J^2G)(ol4UQrp~$eLLp2(?{!`;8SV;f^6+(zaS{nb{PpRk#G1;6*sJC{FO_
za5DuLV1{;2G@-1rw{v9ve}jnvt&O3sUM#%Yn_U#v<jMa|fIWRmv4?VrvY!0-;^M#E
z&HVpheT~d-LiWd%+2GAVaO+NH(?LejC9GmH_BWBFhhb%YlWl0l%RTws!q4jT`z1_r
z$YA#Ox=xT+2A3I<lj%}Zuko*v+sm=;s8UpzJ<*2eH?e#<CZ~84f1Ia(kTG2N0uR85
zhU*KO(%l;Z?0HK|Xk#~t{62g7H0z-Ymb}CVa!l%d$>GWvmt<YQ1(jGbe?oEaq_?M#
z)KR!zkIA@`6uO?cRlj-pm*tz6&;NY>cIFY$%8r@)FB!5L4t0$G$j2a`rqi~5WP0>J
zP)h>@6aWAK2mp71vk-s*Jr0V6e_XkuGwc{6001Rpmr)P_69G1tei#88vrfGtBO;20
ze_S61F7BBc002lv001rk0041iZFOI9ZE!6xE-)@Omwp%lAhWXIqZ)sTg@0Ufo#&pz
z6aWCgH~;`40001SWo>m|acyudFfK4IHZEjob960fZf0*UO+{2eL_t(Rty}MM8#j{w
zj;H7-RVyjW8Bvt{C-Ta-itYGxv1~a?yjPXViQr6=M1%px0EQHEbyfTAp5dzQ>fUXi
z;2z<s8<^oxwqwhdJ|urK2{ihD|GEL4VUVQ^GJAv0D{GEmlpl(hvce;Hi&c}qt!mMI
zH_V6meR0uLm2|5kc>V0`XE?J)1?%A4*<cH+L92<AZYBQNR+!athDR_Dq4r06dovZZ
zoaBYA_K0{^JNp|JVXt0=xiuMoMhBiWVQyVI;5;r$A7)rdr6YfHS&0kOe>K+3Op{wT
z6K7JHAeBLP1pi{^1}=<U_C9TQ|K^Qd{?%bQmx0(3>5%N_A72~+aZZd6I>wjFWzOob
zZnn3%_^B<L3QdqfS+m~3s~yTR%(1kE&z0Rv;`*EN?xES!GN2DPXg`@tGed1>Ure`>
z%H;D<>932H=H`D<r`|W{2e&oVUbd%r=O(oa+qeR6O-($}(t9;kWWT3mTw9ZF;UdV;
zcsM$O1IXY+*K@hC_S6J4C6;i~lnPCOv@{Vw=Qq~APz4&#D_&ik!bx2_yTFqE{ru(6
z@B+O@2hY)<lN!#OiB?6oX)yxpfRyTB=f(5$7mLxKcW-|~{=;;4nA#?>mohZ8`h_&J
zM$VY(Gc(i5&o|b+s?o`y%nUA8K42A3hf`BxO^PvLWBIwWGbgKxHuOoKvpJ%giGtCq
z+wRLB-g?Kkx8AWoy#1E_+iw}&e#^n_w;XP4`N`I*pl{ZOd{#AD27AMee_GW<_K1JO
zQv8T-8|8n{XOZ|3JU%94JVjqPRa0`xo>&T#kkN)C2zVX#t~2L@NM4wIuQVRPi_<61
zUS2$V|EHJF&tHrv*cYlqpFO({P96a#;x!9Tl@CWyuPF6AKE9Wa^8H7Ev=okyVT)I9
z-CSPS1v)vy$H({c{e1ZG4O1<z@FCG3=7%3&uDpMB%*}p2`uOT<Rj(c&@8^e~!b!c-
z%7n+qqkR8Ce(>@29z#<HYqfuTd_PGWQ{m+z)>?Rc3|r)aTOdvOQG&r<lezaFehRmZ
zTdI<o{S-y;>WnImRt6s*5A*#`knnspqr~m$m~Y?9)J+!(jrvqZL!)lZ=KJ>i0h7G^
zCX0WQM(Hx6s{4*X>$x)5<Xi`vbk<wmbpqaV=|XuK=2#(l+x}*t{L${tg)V(jJwASz
z4?leJ)0L5xDvm&>ke(`yLbL$yLrCsLLUPBS_9MjKP#3nsuV6fGX^+RCJV+S7_XzQq
z_aZFfdasf`pxaX>ptJSf+p5ldjRm8a$K!uXvGb3=|M%;&^MCyPe|B5U2}`F!Km*F8
z96Nx<Q<@x0zcPig>jN1)#h~Le5B3s~y|lDFD2LNPGBh4b2o^k=QoKGphYG!yGxYSK
zYmDJ@Sj_iJ^>};<*yeJgDqON@TM!QNhkxdHpGyzMLM2T}t5?w0@j{``h4^W%d?$Z&
zAq`Bh<FBw))fiRCfUs0y9*x@_Jb*CA$ftPTpbu~|BYg0eKSG#mAlaFNAlpo~N`jr&
z&rfy-qN5EL!BRGdrxGh`K9xLsbR_O0fvjW#YqP(;c)B|vZCqtk6x`aKp_E2CWGLwn
z1tg_a29Ow9T1ptYL1kzVL~3XU2ACloB!)&Bk)e^0PRXGpCFEQ0z3Z+U@3((GYwdl`
z`LloQy`DqQH%R81Ss1}5fa#1=`UHJmZ9qlo!(xMn!c9O{c?eio{rq?kjQ8_M#Ibk}
zuf~2(oR$d8(0qSFo#)#-H-d6=#{A?L*uHr=Q(6yOBc30ENI_`1Hh{sV6wBvRF{lyp
zB>K%g!w<5NiX7-Czu<tVpNui}$+yyEjGpr5ZZ7VsCI<3`qj^wa@x@A&lkK(`hp_i`
z#gjlb0&J5iGgQh<V9F;McC4~7G1(EA|1qx(2Ii%#H3&(ZntNX^`nl}I=Q>k;zZ_<N
zt31U1!;Nmqx3!f~A0@L<3oLDlvP#!PKHiw=*TT>=hPrOoXavu$zn5jI+-9V4DWw!Z
zUU>FRtrpPF0Z*ImV>DpC`!r2cdUa^H3-_oJ&KC;@d2@9yuBVs4JWQc3^GfgVgi5OF
zAw{^~$RMw7qq)wc98U{<%s~m$CP#A3oke<C)vVG~#fGBW30>}uWS|pz3U06G?$yeL
z-*ER-yQ94e^eZ=X9l}PhHmCPvpHi_^w824dFh+o1?(z<dGKj#v3r-nzT>E6&ri|SX
zTJ5w-P;PIiNjVc1j{1g=HTXs*mw(vJpzWGjb0pkFG0PWd^zz1ZlRbyrD=bRtC&2K>
zsXn~%<aU1}@d~#y-2T$hWnMbD+n%^uWJ@hVi_7wX3D7J#MA_U>rOou#_pkKyhAF=&
z*=*jusNq88aP54O*PByFB6`P96>8V$w21aI!G43BkjZGluI~!_$E5`2d${6)+`dz{
z3a)o6$29=2Mzjx}Up*%w)pcbJIS*^sTRSKdKsm7-mthhEF$Ep()Oq}j`(9R1bPnaz
zSeB`IGx|hOS2z3$H-Pp#VY~8Rfj(brW4-Nk@hUso5t*@`V*`%3fk{Z3i=5GuLN%&l
zc<m?;{n!yqMpZ1f948m)#-B0E8Bg$dYHoAUN_%9<kU?{sEw`+I@RMsQ`JpPr@Z_qX
zPD_YAH%)gTzf{QU_}LiR+f?sf$~1(J36i$<20Sg^p*j{V-phdU9?IQXOYy0$$?o}3
ztpA9{TduZZZ3Hu!)#(~zelOmt6?tV^U?LI0OU?%y#IX{&iOSQ%i1NMonmZW+%h@VE
z9RJRdsH$^YaG}5Vs(C;%D+yxdMwylCwl`uYHEMW;OczU<M|~I0FPq6fa|!dTGF<3-
zS8nX-Fi!80i57lxxXFCv`~EAmYy~CuLI;R8sXSR1F_5rBNZQ7yochmQ>kQIbYPiBh
z&-Gx_te&G}X7*i}e(=rrg_Vu_)`-Uu`*^7ft1W#Kp*$9IuFndk(6(N!K<b_xnw9Z$
z&RV|W{n^A?NdHP4Hc9VeoK_!v$k?-0PANg4!uVt~nFf%!H5K8%9IPqaJxuiM3HKk>
zM@1S-iKH4O+!r9gdQA&2V-+JcTlG%N*?rhP2SLIL^1=4Srkl~m0;36GqI=lhkL(Ue
znWt;9fyz}O3xJnEQ_ZkvG)6^8D2rP7c?mp{$6DKwizCsgysS=4Ievgka}M`C<oW`C
z^{y!ViFu%?EgLdyeP}Nk@F)MIcLyy$J0WCln5Ey7mdxP3U*wn4P&^T2o6y0HcZJn+
zyYI(gV<BT+9)ebOqeOGlA+tzxhg_#*_P!ZGI{zYbk0H(wAEKMRk(+A7<n;xIIr~qk
zLCEj)h<P1x2cM}-guslup?KIgVm`!E&&)@vs-jA53fCr_q~f7<M3-7gD(`zS_N#}0
zOh2nX>#nl~&8m1VX+l06psJwW1}1<khbK4r$hqrN0Y3Ax$bB{;K=S);hepyYnlsV)
z$8BvAh5$5TU+7PRrBWowGUA=pjIb6Obl<G;hWDJ%6kl;!@=(`RAQMp|$sKL19J9Ha
zn-cum_KM@|Zl1Vg;)t=hCP!kbQr(~5x7SN{5T|*~<D0ByQWd-FFDO$aVLfwY;Y;t3
z-uQ78Y0h-0s{b`Zp?}Olo&I=Ft=mcC)2%xtuVcUxN2P%^B)R1Wsyalvt6VZ>7AFQX
zZoR+X4+|>Qf~A^342yEg3gkk*5^H5Kx!Mn%3E;lLlA8ILXIx+Eqv9x8Hu9<Jd2#rN
zNNV=*dWMIz@yz6o2|~#tu${!nAH(Lvs5o+SCX17Y@(4$|ZO*k%5(FxK3{8BGW60IX
zx)QoEY0eeJb@`9X(JIPjrWFbJPMkKTocnA}U+BKf6HoL;nP9g7gG@~AN!;ZL8Xdg1
z_-Jg{V^c6(hSXjzK{RiYX1;0k;to<96HT_XQ8&sP;6~H@sPo7S=5kgUiHQ8buIVx8
zF4nC=^*J5SG)nbRg(SzY5qMzH!E<|YDcKZ!%K&XeWQQwKc5ftvaO!ad$gHM#P!gYo
zuZtOEmO({UDM^#{DK^{t$)txh<P1QXs=^%v%eRCi#Pvb(ocwd+8>)BxqdgF$TQV+P
zm-Gf%<jPQRp`{@VATL$ra|@`V7}<1s*ub*5A?~~9z7vQAU6ym2P;Ad@v!2MOtE*)T
zd+T!_!)`9EZv-O)XTlr~J?0Z>K5x^Lkn#kIYb;6Uce!t>V?<lr8u)?ED661}Q$_j?
z4pnALnVD0h`v5jPtLFK<4m7bl(4zc;edLfFcD)n$ksS5_bQX3K_{K+UVvq9ecp-&N
zX#-&tE?vNXW^GjFnNFNz{zoZ#=jMY1$)D2V%LU3G(z3_@c)n-Tjo|j%hPKAZLIhj-
z(x3$qxG2NyY0fgm;<`R*IqYoT+Uf&gsZ;B!#rRaf!dSUu?*4H@n2Et&P()?vPQl@P
zfDC$VB?ksZ6U1}M9-Hfe^b(2{>uV09>CGyq4vLjj_ER-kVmz(`2WUctKM`BgX+J98
z{PRu)w^{&wvD7Ds`KZ}5NEcwGK2k4hEL*8Xm+Z~sw+Q-^7+sp5zao0FKrmR4Lz`0l
z(&ZCo@_Bn^xxv2h8&SLDmRE&&0jRTE(&X!ohjTE%_k;}+bV{39h6(NfM($+$CTxPu
zcNY+@;jbF_iScdwVpQ+Ai<@lWsb*@gW;T+H+Chz~uTU2({rxo^1+9xOF1^y{se&(I
z;_}`orE!EBi-Hp7{EOvRyKYgwx{0^xT@tt@k!R9i*X@*QTV4~I`=?pUsYeES(MuT=
zE(I_X?8KIli+>Qaff&3urQPazVOkdIfT*QttCVa$zpA80t5T`==oKr&$eQX$WehrB
zPIBOEia;G(S;6+{Nj9{DQP2qr&?@`wt$DU-n5izQOIwTx)3Nueg^vpdRNJiGE)>%4
zg&af6jOq5IXODg`vROPYvGJ4@<a&to;)BH-Ss5U2$1u!K_1Zou)~4p_#1Yq{awR4i
zb)(E>No;NlZ4UJ+E}2%|T0L^wh>7W{-I*};=+;f}d!(Rn_gC<gUNifliRI1&&!%vH
z{CR>EF3kxoB~jL)s5vr~ye}9kEILfyeTk)TmPu{FNqFyCeK_v?kVn)p^h%~Fj}k^*
z9dSKxMQWZ=O%Y=M1ybS9oCE*CIwSbLS5gvlCGv7p>^2@7HHH^O-p)KYe~@(dJ(D(b
zsD-J~SCQ<=`r^t_^r}!+l8w$l3l@pV%8#fnb>Wz$2Dv@(<$JDgeT<(tvk#Vxa-*pF
zI5N;3Z6gd6&j`K{FqUM^VJ71gc?Cm@z9hCAG3-nUro+A7VM(Uo6fO`By@!xJX)cP-
z<xF#@-i~%a?!He@zG&^eXwBHAu36r^zHqNfBeK>XO_(C@+m~G}-+_lWWPT+WEehMO
z!s^6v{PeECG3$^b+1N&(ddL5$z8_oKrSLG|hut3GP*S`18rr`sMaMa}eGDd9v;HVP
z<#~W`9ZV*4Y&v)qI{NmSvniPM)^EdMH!iJN+sM*X1_)vqE4DHl*=j!*{7Ef8Lvd=X
z&c=WKLtLi30JzdFp-pb?=5A7-@@ci(ol~kaj>P-U4OQt?bO?e_xV}Nkc4^7yoH(vf
z5iJq(y6Aw8N%5)^b(h`M+F97X+pU>+Jn7h6I&ut~<2oR2;k_SHiopc<%Ld=Z^FWR|
z!O}fJO<k+P*jh_EE2)Oa@I-1!32snkonHoV8e~o(DoRPFg>adgD)m6As?uQkRh)oX
zCyFcS)(d@T58X8Havf60Wq-hMkd-N)*Gq)FvnV~D(ekZtfTjpY7xp+mUD(Y5EIX%$
zc08h?!(FJF00tVik|jW}F%3=`X9cgZl(ixz)M+~WK>3aFjgJ8(bpyl9Ry2bt8Il~H
z3NbzymrE=+-VIpxxrG@{=q=SrJ>TL|x%|L+zsx)4sW|;zD(Cgy%@75#vR`9;yl?x=
zRth5vwiPwrw=Q8jV3Dy+LcObdV&c2+76$sfBBG?eejttF4#hrupxhX<Qa`WpU}C-!
zevPVWT@v5vmyTt!VT<OfX&J3HO!M~aSS9mdnl8`kjB0}*5PRGX?0PuXB;!^&=6T{T
zquC1`mLu|}-<t!Nj`uKI^KIn8XH&<Z1>!F;g6>1``+87kSdxX4<<acdv~nKc(~3rt
zjGfb1&dku`7PllT5z1hXM3Rx5GO8wN>BE_bk#=BVScL02=PXAE0c(;2n1}axAIQIp
zFUAKriVcu!QHwm$dYQnwb!{P?y|a1wXUc6<`}}2s;NIu9WtAaF)JxFf#YZLi;k{iY
zK-X@2_NbvV>{-CFW#Q|-U|j@e*|K62^;zkD+f%=aBcZgKfPUsL$}3nDMVir~BYO19
zrBcbh2IXa!#0fj$1Eegs)J|1+k9nQjA_MqTvRVK9t|meh^S!D?7}fZK75NG9nY`QE
zj;h<@r^xB%?0uTo$8Ng#J+kOK#8bO6%cnc!hk-SnF!n~tO2l2?9B!jJJ0RIzxK+=k
z{jc_h@^oBPzq&mE@uJRdPru}nhmT{_SHJ5F;?nn?i8^K>R}-&%`6pgWuJmf-ivyC4
zp4|c?7R#XmTPJAIidT@<)}`>TBb+;6B?02&Bye7Q5M5*Jt$l{Ao4&xyuC9o#u3Dm)
zfWlNp*bht3q+%MFb!6{`ACCT*C<^g~A0B$UKZv{Z4X6F~W#2;*8rDUbPe${)sXRKV
zg|6q~Z1ZhK-{~a+rZ3G&<>IZ);W@A@)ujyt3dhweGiTc5mG&g|?C#E#-XXC*6~A~!
z(Z@$`@_H<PBGnqne@e5WdAkB#lvW~SE+V2bVW2k9TVpS1VZEpezjkv<(ej}Tf@Ky@
z`i@<TrENK<aI(!_i+{&@^^m%C?fe#|7-byKG5ssg($iVnDK+c<W|ml&H^=a`$orAp
z*Dz84Go9RZBe#Ag{N?`I%4nWZ(oEs3>bP@iT3=&IE;KN6v^|nUpj<|B>#@MnEKjQy
zOe^cM=Atq^wTo(GL<72uyGw8wdVRk%_OdItBwwE+X+LA6#ClZ?n&-%E`?i$!H**O1
zos01JPjuIerIl)wt-IFZPR3SMf+cqU?dEgk)d&m{sYK(P_TY-vEdb}P8XP;RSn;it
zE3P(WEP(CT%vTkofMr&R*^r!RkbNPw4xpVvXA3Pv?0r>O0{`j#16(sDvwt+WR)4(5
z1-h81IEVc^fc+~g<=0-1vj6}9<^TWy^tY@uV~`riVGeF0vPSv}^WLi=&*QT}arUdZ
zFP_!0msEGo+GYo76F$yApk2>mE6B4oW?Nhrq;zii-Yc>;u{O_sa_&gAk3?TR%$rgh
zr-vMKUl9KH@fpu(fb76W84ff+Kcc_xld<s6!4(Zq@b+H=O1r3o4ADD47TA9#w~!}l
zx)AlW@oD!$4QAq{%MeD5A7uFWRzQSV_@Fz5bnGCWzJXIYH)Na#t(FK+Ung$$8Tt%K
z2$F95Fn_@M#K`)&{_#UsW+6MWN~NH{7O^-2KYfVqotPMZK5+beC+h4|lm5h{IQ8{G
zCYpf9Ebj6FEac_yBhozicMR-A?2`q{n(OPkowW6G-oJh+lRbr~EDV_Lqey)5Xy!h4
zD*3I)`~29#&c3y-jV#aaoxX|(n7m<!8cV0V&wWNb6n!FBdn-aaF_HR`xBM|PEwSI-
zoPgD(5a*JYH}2a%!@$WYOSub*?h6tM42|vS{W;_MFcA5JlzXq%`?tL#SGX5#D<4iv
zBWo-CPi9~H96d{nd^XNGTIM+&5_e5;!nd%%9$PhV2QQNOb2p^W%cAk8X7NBcZLjgb
z&?NXKlF?}S-r3{FC-n^+yx<xy0z2mDi-KtWk2Ykp?2b5U@p6l;{2e0Hq3jDVsu{rD
z&R9`C0s}QCSim15mTf5xd%xMf&Z^@YRuqh;<As(`$1QaSu9YDY33uVc#_rPW4OS$B
z)DBI)>oqZVtOtHj>dzPlDeb!a_Lv)xo|#=<;peu?V(FTAo3x{mh@@Yvc7a4nad*B5
zXC{7isss``@^6DEym~(@;iS-v*0t2zB?6j+!rrnAJ|M(n0B)NsmPyexg#vMff7Xf`
zXKddm##Fc5t^f6jZHasHDL7Q#BiZU8pJWRAqPV8kl#BY#Q+<(+5ZfceT`1-8HxfSt
zxs(2APvS{o#+pmDmP1#sJlbvHPTF3Cs#8QS^9SCWsDdgs&y}b^TkC4F=YbPSOHgxk
z5G>LAxU8IiYg;3aw5nYkyb8ZYBuUKAoLfzGixm+-x#6Jk&O(9L;203YRnzJ+OkKUY
z)F4l`Z)Je^=L>?+<yn`h00IY3PfD{<F@>s7TkON2Esn4ryL$l&ki`v)dl^hY88_E+
zcHzW`*t{0?^}TlkF^<v?>jhZtzEC*6gw^P5^qlY`$i-8D<IAsSOzwei-exP5l!bH7
zDl;lYcFGEr!=%ZATdPoSKDiJM5-!~3ddG@C&uCt|KAtN88n=ZMI3pCVtJ0ZNc>8|`
zdkDx`Jg*;(W+p4f@ETs`5ASAMfB>RQ)I3{l*xV(W+TX7i_D*O!ezWSQ>2}wD<k_7P
zIk1_bo}$^bxZ_zbZzk^iEP-OkX6f~8x=ys{%O8m_K>Z3m@B{F_zb?&q%Lp_84rYup
z0y#PVYpKN!0AK|C8>p~9_8E7WfT{!ouvE@arVLjmpb)`Wc19)>P>En0n=#D<1pU(_
zE}4L=w8Jg`Uk}8lpr-{SpeOui$Mlzj{BL*kFQCH2mi|8l5%IrR|63yc3#c%<%fGQ^
zn1LMsaI3%UGy}*2WTa)F0se1GCCCE2`wwkbaUxbd000mN000F4B@6%nP+^$Av;0_q
z{QrD4k_E_0tA6Kix&ti0d;ie=%PZpu0f22{0D$8^bg^NHe`j&B0=Y<x8Grx);Qs*8
CtRyo4

delta 22056
zcmV(lK=i-6wFJks1hCUQ4R?VXTakJ5KHVn(08W$OJUA%s9NS~b>$~UmQeZboB5nc<
zfELZo<^J}2RRy4d?j}V$lkD!?`iDf5Ks_aYuZ?!P&=tF`Hydr-U~sdNmaTPJSAuPf
zzLTY}jI*VOwJ>}ww%UAT3vHMcs+4NUxGI?ug^+hrEeC5F!ZLj)4C9J%n1IyGIwSac
z%5HH+VeDHbEw-9lXL#XQQE_VrZp9sw>rExrLOILaO0Ys3!PcT!aV72AGQD7&S!{$;
zb9qdYz?mcL$pEG%Eh~kUOC?I?G@A=1J~^REQBDVgE5#hFdBcm3d?_YOI_OZt_HY_D
z&VoCs6|Nk<zWvJ&ub);@D0e%+IZp>CCntl!19s(;H$F}ajhp>O@KTtG|Mbq2kUmL&
zN8t#FwN+fq^^2!8G5X2vHh_2j*U^A6#xB%uLVv(I5c#>&#(%fv(!o~J$EUoinopik
z=)((X9sTuc0~6&HeY&YPq5aKnBmRkd$ZwU@ivBD(bR7?9j#FWbHerr``Qi0*`0RhH
zoviZd&$V-UJzMc=LD0mfLf{N$wqCD)x$&!8RJC2vpBp2b+s%M##GDr&>Dzj<5yrzA
z4+eu9b{=Lt8r}@YgW22ko7r3V>W15omoM3Q+?n_9KTMikd;d}^@!<nw4;cT*+3hXj
z3M<6|(O@)Yr{6JXb9)OE<%ChfU6iH4!C~j#TW2OrD)(XVR>@D{H|xyc8qYine1rS>
zKDL>#4BQ8k;y)-KzPOpad4BQi`?r@bp2I@_;ScN)`{wI!|1cO3^|CjUDL^)Vt7k8s
zvvc-!=!Conxm6)7vNPWSkXsujEbP9tD`c`^0tD;ntCugIKfSqn`H$?JeUnck4vX{P
zg#(_L^?vjA<xA@N?f%I$3y^9#4ei1<u{GZTFEzr|#<SL2Sm7<3?^waB3TA!V02z&M
z3COdn>umOa7mwg6wM0r81(Iohj)JDnCGh6$O3;L*l1}m}Zk<+f4J&Thg3AgSJyI@k
z`sabL7_YW`XNhq!&Q6ThjupJN;-n!zhBr~c6`KnLA^{<dUUJxAWZ`8giB&ME7TT=w
zp729n!a088aD_~0(#aV+%{Wt9or+J=I-vpsJUgS#@Q(u#vY*zQY1qeqq0(-)(6uV3
z(0n+KZ69Pic*ZVq@i0t8rDUJZl79KW_s@+{ntl6$z0}Os#fmLtB>*K$X++_)+0g{g
zHr((vLsmEQ7ylXibG_#36o`?i6R$tGg9{_)xbBspKGO`!r{av=tOSE|IK^{lj$xW1
z0dSt#432&_8(C2;Cha+YAiqI~@w~^)f^EGKW;C8g7+G)ZnCxn2Ma0@^T@6f8EvDUB
zu=9LsDcE<0Z#JT24dOKmZo$n~0x&-ngJdhLGkzzK7h7kV#S&m-wxa;7S)E)k6-ICR
z5LR3*;HZokTBdZi>Iq0lbdGUSim$~qgAhjU++{uUAU?ttCqH+8H}aP=_A|oz<s|Zo
zYhXeLImLxy7q72aq1S8RgCmdmaJr(X9UQYO#%#+ilh)S2E*(gsQt{2k=#7!U=A498
zHSEJ*>z0{>5%D-mB|i#})Vs5_aI3T1|C`CX(}o=a6kwDoJ1bD4WYQuKUIES}9DVw$
z>5iE9ozxX?=*q)?!{3ModEshavD?7zZ<!UaP=`QZQ{ak#zxzQM@l!30C<$5uCj0<g
zDKKi3!2J$>38805M?^y_3Byj3`GoI2gB`SBj1SQKUUJ8#wE5=*F#jp;%Xu8<SOWK?
z)1Ay<Ft|*Gz02W*&EeuWk8<90I-P**{9z0P-&Q=Ny5p69EKh;72OxVr(`xA0Yp!H5
z9l*@|Tt{Kfv)0bzY#XWa^23B(e#q(cnJ6^M>Sw69f#Sd!kd7{y)u52U?W<Kq@<t65
z3KS=d<5DHKl@mvMMhEavsyqEr5E|0J9+#2E6IOG^-UUL9{S%ndA}?JF*Mr5yN)#VQ
zsG6R!Va5}G-opV5qZHkoEbKcw2Yp9SPry<5ayCmDIQ}#Zf+;&cZ!bP05JH#1VE|!K
z(?(Rwp@_9zvUA|qFrQzLm(|7y<r)bT<i?ZOOH_wP%H+b{!A9w5xp;cg<pFR2h<Yp)
zLgBnU!8lDKxfkchN6*xY&-ZRfhkcW&q*m(lb_QmDdGBDUzJcS&vJ%-K?F*h;D~w~8
zyt1M}hc?cSV5~`NdXpB0jT2r~26`TWa(Kor5wR~*akFUyGe<K?{DLe$md$Hfxm1^M
zt3-H;jAxv5U2v35pD;09PLuJHZhs2>vdtZnF01uud<JHkq{CKL6*FQ}@j|fSLhB*h
zO1JWVJiykKgNtyOu3GEyaN2C>pHRWT&1g^zcWR6BpE<-7dhRy(_lTRNJwu}hd-Ukz
zHh#ptzEHdTq*LZI?qLR`#^cWug57`}-?Ng&7BIs<f5obP%N3Fyb=*21us5DVG?*z$
ze>FT|m+btzaBGS3{pE+G<6A4D&6C{;gC+x$RYg=4y%O862FVx=#YCAt!<3P?%e}w(
z(38GJTYpzBn_I8V-HYOu_+vD{OUo8@Rh^b#f%}AAKs-?bgl=vmUev~j3eflolT$He
z(jg2>4grsk!FwPTN{?{_zmr$)fQ=nEIA>rFmJ3-l8%~=Ld<?V$o#k6Ju&nf!Zf#!r
zT~6l!f}Is?U0WASG^JQbm<dRSvr^BDDDFVtf`88V1wa^$Z{Ze0OSQZc)y}Wx6{x&;
z>3~rctVnl)@nBADO>n~IwZo$Xvk<Nqmro{KiJ;<6XoEfxBg;~#VB$Gtx3g%SnRzSS
zZ9=>^gpnOTgmv6_$B02Q|Kly%_nG7NqfHd?i)PbXAW*KR8F}d!mJN1V5DZe|23ir}
zJO$`*9sekk97kUPL6dw(H30;ZvPYl+7n4&+DO1n?e(g=n>~%e_WI;yr1uw+C{kvwL
z%uX^P6ph|Fa4dab=cqR~Qe|>rFf==7JzgMe-J^7)9c9)tOy0ul=4cwperN_qTTb|p
zikIXHXIGD3wUf_C7asx_^3ee7r&`o)sJeDbE!FZYc$TMsfmZLq2iJhQhLZ(Jv;mWo
zx=B5Mk3v~?&fYhCO)e7F=8c&-Z)|!_K`~V2XmJ9q&wq{`zcRlhrnGkBa7?~Kbf)92
zd(z_$`s2@GLBFt{hio|gQA;%<u;X98^Dg*5fx|`^w~M<3J(?ZL_;|CNaN#n4p(|N@
z#KjY?XP0CYcLqrp*d)JMk$sdB1OcId)=(sW@mDs|E#Al=2X5+wPSSa#*8CY6-f@NK
zUbDhEFFs_8?7bFy?zW@u_yiqtc`#ya21Xcw@^mGuas=&DJ8IkH*w?$~;tls|ezs`8
zYvNntE(9cMjyo`l?4;p<y@@(C+WYGuk%so3Mi|J?t2LFz5Z2~J_e?DQ{Vrv*w<b=1
z(P>7ICM3QF`!%rp)cw&1lk8qC9w~scOyQSNhvYK4>|f~~BWi0GQYJd%s`RXG)Io>|
zgish^?HNmgfpQz6wN)zN#DK7jtM(v#x)m2VK00e-tO&67xmd~=HIfL+Bp2C);L#!O
z)np7Uv6J+iq-D#R8?Ky)b`UEZ14BZ8dQVcE%+xaw@9IKplqjSX)sDgb-th`h6%L;)
z;7$Mhe!67~?eFUOZolK%Z+}$zSfk*C{m-`ig1&?aR2ZJZw0gqczP$QhK1_veIA}VC
z-8_9AKs*Ms(X<U8Pw)ErMF=|tn6(u>EcKAft?Oy?8`<uiMZsiidKbL)5dkoNj+G+2
z3a$Vs7_WjsCVJe%KZ*f}0wOB6U?c^D0a!K)tqLPR%2Z+sK~-yQ!GURo%@G<FwDs{H
zEqHUmZeLyF>~CRyx6fZ)(x)k4AkW|g&GobrcY=@-?8az3v0!DZL_3g7iEWDpas88g
zP8jC^L0-JFh{+s7Uj0M{_c5D)3`IS1OaKwaeXRp-HouW?Dq}`{KDfUYgBl&nh+6;d
z<4gCjq-&UhCodU7g2B{JvSucJfSsh&i1$$*=WHW{X^SMmc}^C>qoGG2r4~eA^GX*V
zakSW-2BdM24SEL|`ji6zxn7GBRE=uqt#fnP*sYrh0R#5TWXdiW7@`}0aR8w^eFQ@b
z#$UGPC0JsaHmQ-tX||AxR}dviA%mUu|NqVs$+du5P@(IJ90j(8oZ83A`*r;rbH+X5
z4pKLIXoGf!%xwl2b2%Cw4v8GLyk@EJ1a5;Pc-nn@s|j&9mUT5f+6>GSK$I@;7kzq(
z|1*W4x0oA&RLwzTxszajX$aKXkxd}THH@CvJ-Z}$zjPJ5ACj^n1$-$1HT&Hqqi_Lw
z2%W`3Fd>CgVp!io8teX3Jo$aZ6P3v)eN+(a8`0<jXWlh%<VQU^c}g3PXc_LD;K|An
zAPpk=O*v7Iy!82_`6h>k>Ubu9XAv!gZMCE`$`VOsv>-*$|32)0li7Zyxb`$yYIS;J
z{+>Ey4|h9GUWrC#{Wl@7{3nzAt(fF*%C`bN?-QQ>gE6s&Q6(X!M~GvI(+8MGqt1a0
z1#TtOSumN)I=yLg4`BpK8eDbrl+X-)*iGYvSxW^A-59aPIJ0VpnXdtLeQpB`)TQg_
z9^0cw(7=H7AqFIWj#n%3X@e1})o#lEx2}O+Gbr>s29*erPz`XIgyn)Y)UtXrVcKN<
z8=`~#Z1D2bhdEKIhn==I5^noLB)KtqUWxUD-3IN`j}mqavLpKGDKU9Z(fU6+$>Nbo
zR7@oYkNd?+KpyriZQX0ug+KFF*zACpz2m08_Uv=bi<MM=!tCycKm^g3HJnpMi->&^
z$J#D2cY~rivy|P`XSzq5LEs3Fg`h|Bzd)Fnh7-wdOSChnUC23$a9+{Z5uqnZhK%2d
zh9txxOYRSFLqs?>_6nRBdD?Ayr`_fzm)q~T6%oKKLkoQa2LlCprIlbKyuPF=s=7?V
zY=YD9DY%+{0Hbh}2?A;&wO;SsN-Oqv)BiT^iV%5bO-2s)rWS3H`y@G-D+pQ3vM&=B
zk}bA}jkI^e--MDF3?4ifyTLm3*@A=GF->uKA(Km}Le~(WLLq|(hDednnN|q$157p$
zW(Z!xQjn{jSZLa8Cq-4pZ-{^A8v=y~gfV+PXO!fBGdg)_&)7qILew<r$+8p1jK_PS
zcxLp_p8d0WX#ZJ7K<&ixQ42_C<;=$h1BKuX5H8X{hCn6LPM3s-_42r8fBJ@d2elO-
z8_8c88712!7P+}=WA%Y9G&rXTn+r-=+y>N5C1;NssM9md2k0*8K`VS?G;SCaQf(6?
z>H`jcdq7~S6A7}XFb!5zf~$HHCRsbS(=}59Zo-dBZ@cpd2{XNF(8=3)4g0zS3?!N)
z*_Q1|ry<q@_a%6-{f7MIhu5r<bHgFbdr!yLl>mNV1Qmq93;vG~eRo>I{rV@R)l$_{
zZI<7mM6?IY)2C<bf9REB*Gg~MAAht;Z~tU}M7iQ_TIlt6J!wGW&NnFy6h=B(@ajv*
zfCW>*pu+}I7VPVrJWuokkZ&n)idnJd%1MY-=U`-g(t?Fp&R)K{VOvhlK-5$ugpE^l
zVaD*Lt?n<sKkJzeXh^)pbG$vHxgat~m}n3J8Gx@$_FOpQE~)nJ(M4p#AEhe62~X^Q
z*hhIn1e>E0ELN{YY}6AMO7MctG&h=~aCzlxkN}1b#iwG(7QlE>dHA%cLKHHjUbjFI
z3y_K}aT3Od?oaRAkiJCoY!CAK2A*jfGbUaojA?Juo9<<4S*=dJRDt5kZU{wjU^!Gz
zgr~7;t?lnd$+SWO8r>y}K}u7%vL8l&V;^m5nO#M{!CbJypI;O_go50O0^{U$3Qm!z
z!z!&eV8sF63rHZ8h4b77SXh`xg*re`g&-Ir(rm4#v8`68rNETzN5D@Xh1BJ_SP6@X
z9bB<t6jO&;;wNho<M%|Db#K*98+P8_tk|hHswDK=V=qf8#5Bieg_||8FOznE-t&Ft
zEF7?Hm+=8HF=?RJOLh$zN|ntcRl*9q_`2T+8?@|v0}iLD_J}SwkdA}3H_$x^R5W~O
zhZx%Z&<>|3hg`R@96}s_JfSb~@3uOYgdz51Z?{8htZ<z;!B!Kv)J@RUI;WfJPfl-A
zFZI1HWxZ+nd>?zWHGEcL+vk0MvG-Wao6M#jdwJ`mI`U!9(ZMXSh!X)$)nC?%teh5=
zwqkUzARD7NH895%B6@$TzDb33r7@Q2SpZM@v)+QFg$9!78HY6VC}O^O{4UM9NzN_?
zjUsvO#l{f!9r#67y=!l;5cr+<dgnuUf2BVVQxK_yS;L9;1RCJ3K+Cm%RG1JEQp|R#
z_m6>HylM!i-v{XY$Tlhu`4rg2^~Gyu6KfU*JqPXHva&XWS7g!f^?~HSCMi##O~(G1
zXbI>Mua}_3S;s;7YpcoOu&%=VZ~Knzw%sk9*1m3ZZnwS11u+-w+=GB<%bED(44*-6
z;%ErJLb&8`!iH}ZC<F=~dkn{inCDf`!6QbaN(udwjM&Hr8slD;e9#hHu0fLlRM~%*
zMPL1mL|u7z!WkBFqqME)N}vm8pBL>Ozfw;MOC!wp#{L{)<`)4MQm?3D53%UzNu;5D
z^x31Tq^v*Dbfe?WL3&e5(}hX|38;*g`^_*>K<X0oN?UjChkO5Zk4GO4I5zmEr-8NA
zxwTdI2aqf(y*&S>`%@*?(w(>R`0{^dhw>YGcc51C(-&_8pmtd0@3_SDMuOP##-_XF
zu2_*r5lmSMD0A50>5$Z;Cup}xpm+mKG~yF20m@c<2SF4NOp|1;p<fhOSjzVvkj-cd
zl!<oq<jO<d8#lfZVl(<WTL-d61WAF7i92xwY;g4Lc;Xq%=-craWl?Mn&hvi+P8tXF
z?HK4|>t{pUI*J&?rGslW_XM=G51Kt-&mk)J8cr6GiPBmm?;~#5<AVlcItrB$J)kip
zeDe*3^;lZ;R*>`um*Y;jD97x9(`al#wMxp_;^s=Ip8V>6w%oxe*<cQ8a<zqQ)F4Wt
z^(Rs7K)4DOn&Q0Dw8(aXbbEh)zD&xBGEpo2CWDr%+xxk8Pr_u5GEWn&E)0Bf6rTLG
zL$BDcm&z`vY^txzjm22dpFkB@CM+bl=FveU%#bPJKS0>jzekDVZ{-r$Bg<q6Jiga<
z6=(|F-r&o@>b+=lpz6JQyjRV?lSO&{u$-_3U(0HD{?JS?Rrma%IcR^e#1;D3dnm}O
z^k_V3zCmEkXMmV1n*o;_`9yDonl+@-l}?V371$v$1a_ofdTCYLJ66Ozd>l?65P83F
z3l52(UnG70N)hwk9RSogd;Iu|HUdS}L%Z>X;^^neoCi$~K~%yy9&z;@cj0wo1dkAq
zp2sLe49IM1tmk%wv`BxG%?cwuVYZ%Q?K5F9PSXbkKnK}KGGNeCVyQ8sT!*L<3^j=5
z*h8dHDD7Tv(i+lr0S4JcjozH1s_jui%~1&~&4GoIAqi4p&5%@^^i@E-o)l|ofcI|(
z4(MR^!Ed2%S5t*|NQTwKs_83&90v?)ekH)j+&bKYby%<>SqFdlAo1H#vV^uios=lq
z(>a-PXXsE<B<Z^3aMg8Frn7lYdVWr3()`>RvFHBx_|uLwY~QI}nSk6{Nm=L5@?-V{
zJ03xRSB=2ebbP{ODb^e9&;{9RfeYmt=Gv`-EQ0eqMa=*sLaEe!stj21Q!T)Utrx9_
z4JK4@RV6wGL85=J8X%lZ+O=A7RaR-(c$A5Hi@7x`4sQ|oqQ&5GXKM~sAt(l~RXRZK
zckDIZ?NFel2akF%B&*YVzgLb<7Q+SOmi4>){51AZPNb1^=kI6i?cmrW^<EpSsy@FB
zBC$P=rcZl#E2p%pJ(z}UnWz{&620+O)|YcUrC5ys2abOdoF%A9qtnWSoU$iaGPGX%
zl4rgupO0su2$tsdKt%&?8baVSP<85yH|Es~i8xxgi(5<+4<$3e5K^qTR7VGiUzoXc
z5Gj@fu7`D(48xGp5i(l%!uu*i*M4o=BVPLGhCRclfZjK6;836omMMClb?es8X{tkI
zi|1QArP6<jqpv5emI=SDs+lifx$io7#J735+UFE3la`ILV1yD+lDM-GX`JI=p-6)K
z&ytm=+#XF7qinU4aWZnp)9<@?mJoxP`XV$FnrP1WfY08CL-s`k^zkdoE9>PGRG8g^
zJAyON`#Q!4x=I^r?V*VQ;rQ4VqUTU$yq)5dg2aC*fT=)fNxL-@hFdXemCPEpYG%#e
zei(kpHlX)kUj3*SNZWb)XHji1WDBO@3yn?K^Ox6e-+*9A52P*f2aj5buB@RjTt_fu
zjlpDsWzGOOdwrqU4k(E_!bT<?#;6dn7PWpE3us?Dy{C{h>UWAB(LZT;SY$i^BWc)%
zkqCc|2LLpzrgs^KitZgQUDi`k4dPC_0dY-o9tJ;w0M-OtTZe`o$a=&ZErEswe<}7F
zVp8H62e>48PN&na?$3Pb=DvJS?B~yL@6fGnkAc!J%_fpZE~<QfR#<_7wV`e-7)Q!j
zTrrmjyVaLN{yDzJ2MQ>lm$yJR`T$hc(DQ#EJB2j@zTaYL*I-ANP1)64t?(Sd2oY-q
z`h=Sh<`Jt0gCqlbZgSYkNmIpNsY2!C{%o?myz#G)1#%vY4CWrDi^c1yuG(Dds<*^?
z;CI0Z&}LyNnOwEK7=EuO6y}kZQrsV?ky>dfVm5d+piZRnfz@ER+49|ghcW&3dLe&{
zSO+($p*BD$F%=7<#ute%$9KWf*SU=Lgl$tWK0+Xg&~5REcNifwboiW9!^g1TnJB2v
zaZFNdmAr%(gEp1$VG^E^NW!n+Z^8+$pl&!aivls!)GA%7DuaM79qw-<-gu^phRD$Z
zOE8ZpJFUx<C*4b7oJ5gQa+N;?VAOwz4c#uV+39b+Y~O2dJu+rK%>Ow@Jk5r|5t461
z<j#Jf?g6`@NIrag(q|Y$Cv;+Lw6$_xiFWD%YC(_IeD}yl)JIf=9bafufSy6G#6nPb
z^C}$TYA4(t7M!iF8)cu;$b2=!>H760vc}OVSM4LaF-etN)@0e7G<nJG9m;<_0soG2
zxN=RiSJ)1TETlaG!_Eu*ve7$qb0eshnLcVlsd&?7ij4sxY2~?O_i+0xyp(!^j+o|h
zXd4$;pR6^d+qETvw!$3mn|9Lb3#aXGx5Lv;hl$f}9~H(t>e?kN9ZHZwYWF<{)x$;h
zO#EFEaMmzt;=%&G4c#U5D3pI~DU??5f?EgvD-4rNyn}%&v(cj;_>LCw=&J+i{`1ug
zVa%xiC?n&0N$FYoZ0pGp;rVSwvq#X&3~p&D>mC{6phy3i;ZlXi7UF{fSZ1Vk1nH&(
z+dP%e&2ExR(BE72w`V4ou<S6CG>Ewe75wu%(4+6(xo3m3^R&q`fZ=~CzzdC?(_2<O
zyYDmMDs97j{d%!52GrP=g~XT3V5HPxvnL^_FYdYLVYPOLskFEltRsEDU(e4s-P?ZC
zX)DC|5dNe@oE;j_GSXgtdV6d<*|=zA#m;5-qHRopeJ>ke!rm!0ZE*yADSZV={vaI)
zc<y46;9B;=8jjsA2JuM%_J7Kr<BJgsw54V;Z}UL#BnXZmFt%X$OklDHujFzi%qfN|
zByS4&2i(yo*Xr)M7?sr{y=N{Mcw6x{0gs<30QJCL))u0!lM!At0T7c$UT^{Qleb<p
z0o0T1UOEA$lP6y;0iBapUo3z9=hfe+bPrPqTYdHN@>OqoxG!boP1vFN^EYo^z3GyD
zvmVe6J3qs^_Gyb*<9p@ZIF66aJIt}#Gq_L9^!BQQrHt4Yv!gh7Ko)#bHJtb{94QS5
z`OI)=7DHWPKnWJ+M?<wtqYj?sc3Dr;w;hf+-s2u33q!9Yzsz!s4~l<j_}<;xE{{ry
zjRT+uAvIdwv!Q9Y&gX7u3<eK%6UhQzRr&L6RN$$`6#w6Jm+8sCepRY@J`J}N8}&O2
z{7=!<5FT6#4?pecMVoz8;7sxkoxfGFvepf5{mq`VfB&}OS?4`Ed@rsY$&iKz;Nv(=
z$Qo@omV^QqXjw8I9Q=O~27t6aF42)Cf6bNL)D=F))FLE$&k-LyZ1kx738at>hpH#o
zpz$>~$UgRqfS_EkgmBf&A`Xd}QH!*F_jblDG<@6HpQa&QBWK<twYS|ph2Tz4za#4(
z)Exk1>Xe9B+vR7XgMSk63$UT}nBeb^3~eUxZ;K3_40llGPbhzI>!dr^C*BXn{o1Ap
zAmP)E*@;l)MoZ;RaQ8Yu06_l(jeY3ZI<PKw`bazs=J_H7{p0EBUkyQ%%K{X_N~+|R
znD3mh_gxbZWDDJz(2=--!C2FKdi2mQJTzNUY+s?(olp{k`2aG-{-IJI0R+|Ff}#Q_
zv!f?_rScC7?-_rLTpr=oX_f{J;vyT2gbWs$J?s+JCl??t`m%SF_=IZtqW}U5SH@RK
zp+qeR{}Bv?8B#4n1sNwvIxxoIWVL7sXYqyM5_OqIqF4!^mhtz=v%LVO6a~B(D?<rQ
ztZ=m>8uWFn7IpIECNc&PBq(2kvkkl#%d0*P{P{^RA>x0`FDe&luF^oVXi>IM4G4Ca
zQ(ytSc4yGDN&Abh4Z{IWyEKI@CcXe;uqOJ0J0efa&!jI)Y;Bh_8ow=zAZ|yZ8Fx&K
zvF9EyjEBJ8Oz1_Qmm=T7K@X7={Ld%#@ceS5^gY*5cr5RXL6~<#;r_^yfGXaDHvc5d
zj#YX~<ywEAWbuqxPOpiHnD5wacoBNf3v9R<FSWZ0ca7I(zl6~pvGh0euyu1?Hssb_
z>2%6`O^UPp7+VZ4?!d;#H9htXobI+GsE=<mFqEF=LkF_AB+(g$wg~pt`oL2NOaWG)
z7da4^3))_gUz}Jccy-cwgcc~ytRbQYAFy9$FxGzpJoeJA1Cz3RA;R=xU9q0WvmqWk
zg9Pj|FHeT{k(fochmqGH)MW366I(D+-1%TzzHJmhu9~Wzs0`9$@!G5{mg~bUOP_ge
z-R*(DT=+UU%m*BRaXQ}P;~v!LL#ZJBoVN>bxAdB<5Lq%c-+xcoWWVeg-c|1io(B>-
zBhi1OO)+_OJ5<<5>i-pl?q7QU%?bT;SAGqaf@5jE6Hp#ei^|!ybtp*n)yBW09-(B9
zmgmT~m#MsvA4S71y@k1l$z)My^IeM>h}1n#d;zVd0j+IZLW=_gE#N6VfQJ>hFfF6n
zp|_@xrl{Axe&A9T`R&&ns(<j-mv%xKg&u#_gGz$A9-5LdORH2hPplslR(z#!67$+`
zfCqH$C0VpVL`sYYm6>fK9XA%8dr>pdCO!i}pVC=6%}C?%VpJjIhN>rG##`_GRPY+j
z9X<CGKD;1`+J)dm*rhD-r6~jn%4B&5G3qUJ;x3lH(f7v+`UmI*Z-LRQ1c3_bJ40Z?
zx1^9i%%D}f{tTV$A$f*w0FR7Kg#p5D*tN>xG7vElTjyZ+w(j&}+trET<DqsMMCoG)
zeG*yKs{!7ux*GE&dfQNsD<6kDq#x%V0P|b<ywOsJ6eRB1GAa_&!T$r3&15?bcYzyQ
ziV6QR#~J_tgOe3yD1S;#Ny^DgGP_q-rIfDL6MJU8yUxSO++3A6B|#)4VN3!H0NT-}
z^53tz@gP7_c08Ngo7wY+NECqv(CBwJkkM##^ycC<d99?8xh%<fRhO({!cfDdI68W_
z<C;{mXi7$EC2x4aG%-6yR&Op|zbH8qW<~OnYCR*jJD%@IPLsT4ArCM`bd&E0)#T0D
z#g7?zx0CE;RDWMH6P}D}a&u$>?rRvwy_p;lLdYr4%?!W&m23Cyg4WQ`KD?@7Mzpj~
zKMF2|{hCu~KRvRrGp3YO9=6}eGx%`^7{qVB=?xP``?hEJ<X%-wYr19r-GzPhFJ-~X
zwjV&eWV)6@vl;n8F=hY_wr%Z6$+bQKocU?Z$Ymq6p)rQdh+-cauKW;<;bn@TwKPUn
z%N;E@cHQhF>>QV3shg^zYHxq86*FePG>YbIP4jD$@Ma%NleGrOrXAgp<VLENZEz2?
zC7fGDYap*P-yu6fNb{O1TG^&chweG~Q&UlKtQalmx@15#Kq&>&x#Da13e1LmvINS<
ze2zS#q%_BqT4yvUV@?+E`%yFjZIl~-N{;^%#%zaB%rr`n&FGWu9}n<(PCj|4ztf*b
zleT9h5lgm}hNCS0ipz|k{%Tfw_K}n8XB~g?^$W70ylj+T9_q3dGhBNxw>u_q6;2Z=
zmNS0CiW%X;Fa-hz9zc6oYOPCd^$PXs4769ywB9sjxqnX9jX{9^EjK{Ij?P&@C<1cR
zBOnkP?PmGCRD^w`poi!4qrf8cmU2TrRCQS9cwNf;`ncfA?2p&nRJ1;(MKPX*KK*~c
zPh%kqHlCC5M#}Mwj0LUmvDz;!i1Fu^nIDl4tZFhC^g~rIbBpG9r{+;x_<KQxkcMoy
zC`iF>c+M6$5)o1n82T6nw7(6|oN`LV5)SO3Rc`)wBG*5&+$40SnJ~6Je8Wy&m!UXD
z$}4a;0y#w8{a!m-rMJnjb-52MzEFRvwrfcBWPexe%rCqgZD-<sx?cvXpo9Td3nJRW
z2sJAQcXbFcsT5D9M-~X93)c_Wh0R65b81*I47#^evIwBML)28kG*d$8=o=)(6yKnL
z5*%4_`(v$S&6L@PO|-302p!N)qtVD)qjnb9*b_pezuL*Tf3e~6fm*N!1w?;sA}7y5
zDygP6ta!M*jHKj*8A8jcm4DQVOs=iav)~iSY3<BZYEqFvMAHu32a?OCEC>jLwcV{;
zf=C0H4j$O_sGZ1~S@mEgX0XMb(7dG@0T`w#E|93!Y$FvT)N$x7)h(AI(HPY*OW;Iq
z04_oZ3j^r4yez>{p*NJ5sM&vRI3=rsYd3|-bQP9wcR!f6g=8AnR?%xlxFNOFnga|Y
z$+eKTgl}5iaAY|+&lxLN0d5H}1!#ed34V_tXq-Q{NI*$zrzjYz8U*s8VQL?1DkDk3
zEMhGBa0A?$eIqHHzR9U($Lswue3CWQd_d>S$k~fmFJE4sy*+zJD!zZ+ftw?*Ny)Ea
z1g>MHWYm71$h842g>ybeT{WI2jUNpX^PHT9Er_In)!V~84oo55%Fe(q0+n4d&5RrP
z>P#si$+M?_cptR@p~DRDPR*F=87|8Uz5ONbL@P<qm8geYi5<NGDsP3W*MBWno^{+3
zuC3EM1oL*slqHh@5do6|YdwE~?hg(Sbn#=VwtDW+{QzI6(p%TLr-Bcu%Hi1u-4@S_
zEMR~Ks*};*#3=h&axu9|n4n#X1u{)7l+B2QVQ;AhC(NjcpVIX6-^8fx@V@o1-=fTU
z4NBX2EM5NSa0KAyt&0GG(2L34nGxubQKjKCHR*ZAGIHY7z>aD{l|p|cyEqOD!5|=G
zZh$>1uC*t$(fO&J#fGyI?fozy4MVzZ`Lz(W$Mko!xP*0o8Jf{41d(hdnpc~#+d`CV
z(9r>0fp%-ffLs0f#)a%cw5V0Q<dz%MV`E;7Ek4mhDoap@#!3$BKSjfEO(ewfZn<Wy
z1eu|xiF`G$tgY$cX5)YNJ(^5_017zWZWl8IgpMt1jJ@O<BZkN1n(Z})4Y}2y{`VVN
zHVnzj1A~^(uwvDkDNW?YnkYL)Zk2=$-`D>uUUeXucBK{xHuMx5Lc>l{;u|3a0Mp@9
ztz-pC^{YzK#EW>SB!$jj&6n{mjOU*aIG7|M$}0!?{mcbOTSk8h!Ex=_z}ac7&FZ#1
z367`n@J{GjDj%Efgu|_H1s0*@E!}HEa~ReJ-`lZM8UyFLMX?Vc6T$Oqc!fqmYGlvC
z#0tLIFvSD}hFa`#h}MdnX>0Dgh<7?F5b?U%luWc^TN3@qtEM9B1|XA?iLKe;9H57W
zcnY~y+%TI95Rre>esl3U+CVG8xghJkVLBtHY(tw8x%=7Ei|;a!OEJp54dkq5g5F1z
zD%dtm8$!28`H~n#g@ze!c*(SN16J7P7=A$}U9Fx3SjkIVz-k54%N3(evvBQ%Mn(51
zw`{E2ORn#z<~adYug$SG`x58_xDrflJ!ETl5)e*D1G;~Fh{iVwc<^1Z4>E5>xtL62
za~b+AB2Iuf&<$6nq2;oo`Hl;=#1K(zK(`jQH?Jq-XW#xu_B4B%JsZ!+vu}U@hiTj)
z%d+T6F-;Z^@f6bT2kiTJv0P@V5fi)IsaF;ix{SpQkaOYP9hd7dR2if~0}dTFGO!<V
zlc4-=f5?A|6^qFj$pdl|85P9kG<9B_xaw{TG@l;DNC<m+OHIC8+Qg1E1iKJ>PFf5C
zxDh)8d-<_lo@l5S!GsGJX0jQfc$||@*!pUWALI9*N7GI}=eqf=c>x<xmV?flrw~xj
z!|R2OwK$|jp_tZ-@e@<kPmm3425#I#fu)(SJWqc$4nlXnexkt7hAj=%*Vgb&a6RpH
zO9xRd(X|A>YRifYUQh1m)F<b+jFDBlA1mA~B;$e^%1iL<?dI*MC`xN48^e_Fj=fji
zkLKjBa=V3K0J=1-k65sEgEHM)2HG-PX`3&C$lo$!K}?n?5|&Fh%SYsr(ZWml!pY{6
zSKNO*NBL}drQ~anEWQDDUAC%&EXcE`<S}{n^xHqU4-fv`zGj4$oH|dLi2@BeA9ezX
zg6hDq%7%)TuWg%mWMLDhSHT_`3Lw0XT`I??iT%)eI+k(0X1&rzy(bIuQi>?R8|gy`
zIgv%YPHctF8g#nD9)gi(5v-PStXYyJA|-#44v*&KryWBp1B32X=^>p_vh^7*8+&5f
z&~?f6&U!qtArpaf`W6M`JJqn~<joH!PA`L%UZZ9OrDwwf-_ugFjvtE?So-1TWG!V$
z76|U1sn<+_rAu;J(={)-;ShX_QdmtbMGPWZ1=ce_`&ByrDjQx@4CZ0$^Alo+mPUVG
z(0s>=kYk*Immvju*LHin%^qMz=n53)yL+-rMsV4t^HT`F0+Dy%IhI-Sf-K0>xa+yE
zg@71v;#TSIq#Xavv4SCuDOg@ZF97EDJ`E|1y<mOQu!dc=iMznA@5gOOfh@?U&xeBc
z9x-j`qmQ5@fS(wqE8k|7Se&h;xgmd4?6Vd>F|Djtur#pv(0+A}DMSb@OGP}!btDs<
zmmH;GqfmFPB)H)jIXBj)MtH`DbXA*pksHD&l-abMLF8Ap$p+UOClvxcIkdJl68PLI
z*=3LmOU<<AQnZLp#a2Ajx!}kF5k9eRgvg&j3VAOCnzZsr6!&s^1Za+tfV6)G0ELH5
z<VIW2i)3^_bW`ke@FoC8EMmZ^)V;1}ufFdN>_YC1uu_8+7?%=TO`5vb|J7U99pdje
z#d41YOhz#uG0?-?n(n8U1%r$lF%tao&5%F_dE1d=X|%9AUSzl|x2y#;nf8Y|05b*=
zn#BDd*$_AK7;*uk7*rxtPtbqyQjwABz<E8|C*kS6WS=Y~P`oFyVXFUnl=x2~r>Gms
zhIv~0;ta$szhU5TN7J~260NnEp)olboxn~2tg!ej%SO@0_FR%mcWFwo{H9|qJ5lCm
zI)ONJXyBZTNv8nrFKq9=ffZ>3qYy=5s^fVuJB86HFXDXyvE&CF^*MjSts7B08GWPY
z<QqMjA+TA$UJYVg9k*csd*A5!-^4c>0_+Y@x5BOKdkj$E#7#k|snu~zsV=U*qnfqp
zqama`3N$(M)P?a;Eh}cwW^5@l8${>#Hq&XO*MPqfY~Cv>T7mTin(?($Y#6uuV&&G?
zOehzA{dy_gfv^myw2glk-w_H_Z5s@(I?EExNT`iPFU#43zOh#=%fx^fOSvA0NJe;<
zy}5Xu#Y1=WkRLeH%sBUSGG+n-^Wz!0dOyve=1vq7gx^w&rE!}Rqt`5$*OlH*##k{G
zjX9pR11-Y0MBAO4u~ib9Zm^WV*35Y~7qYjrq*x+Vg%KpGm>7Q>2$n!abe0z3lE*F_
zsUO?$7bSw-dTT9$63QHa%$#}K9~K7lBmM5Mh>wLk6oRMXo4C5V{rzfyYR!~slQ`Vv
zvj!Wv3=S-!*Y1GIIKi%->F(ARFFU-*ieI-_>lc*%O7z5U4UtYiL^bMN1+m^Oqln%C
zC2@#Lyb+rMY3F~j;Q|UZ(!N0^dJnx9S{+!wi^gK?FbS-2DRd{4i&@V_I!FdTb~qFq
zylb`xj)S)kokyFe9)F|9qZt6{#!Wju&eb*Bzt8H%EQQ24Yp*%Mdbokwxlvqh)Qepc
zlEmEyrw&>@1`@Pd*L*;_$V)~=Qy-#;u=T!OJ0mC4P(gphi@%>1rSOSW7dwx0wD;I9
zCq<DQmI(=DiMk*z1o{ARohKNJW~y(`bW1~=dE%=~e9fr##tK$`rbWBm$wh%HAFe`m
zgh~r*8f^LKlN(y{0&53Bi#|`39Fd~jlM-_~HDlKppK*13J9&%7E0iqYdfTX2Y>T+$
z2T(kq_Mm^_!)CzJHKY=TRA8F<ctpNqrM#VyTL#<TR15<CYHx*V$u<}+qQ16+J%P_-
zDc7M=Hkv2Av!OpYK{QE%*hmr(<DWqJ#x!-}eaB!}4GNfeot`aSPh`z6S?y2E*SvW$
zyM;PVv}JG$Kp?xqds?Pc#`V$`)h&YBx7!g~?5KaX>&Frm_cC5+00G7%(2(uXRdyJa
zfD|uOV?uJH+!Gp8ZB#Xs_B=OeRKa?<*kQW8gamOZEzMD}?~CCc5TOlZT;#?_%+Rzh
znLE26VO}%xbE8dLG}{FNxuwy|SMSb7fC#LNmGzSkReh{$me0rtS#(5nHy=QPEzv9W
z>d8#b$tbDp65VosC>fFszBxagkx>r-8M%`bkh(>_Cg-QeCDS_6QXXU3Ab=BKn~;?U
zZb{^e3Ib*{?bqnX)I(Bb@Q8S1WlEbF8NKk6_H{ZNyS#jLnGicGH4CqQI=Os#{_=+u
z6{8>Bu;-_fOLizscd5KWTHhh4S*K?14B`dH7Znpt99|$DCR|>HeOdI*b_bd>@V?h4
z@jOx~6M$T_)oq93hs5QIX{lwXjLMb;(@b$%^1rjyP}uK;lfQN=fA8I5zq#l_KG>LX
zc>gb`&2hW+=eMt35|60X0f|e<T(1y8d`PEXL{jmToz-X2ZC!3p3j;BK+aSY_3F7@O
z1`0uM$FBz}iCe>AGg>Y`RQ01z-F)Kw>Ax4ByXlE2-S}?{@~pe*4(z?5Hm-GZ&~UXW
zM@+!Oqz`2M&H;q0f3Isx2SLjh5>a{U6I1R?Nz5;I7wbxUtlgQzK2qN2Ht2eO>W^l$
z05n$d9pX7&3|AW@#%F8IbSyGUI$ZlIKA?pwX#O6Nm0DwVI9p?G8y$7_(_3}6B}^sB
z=uRs7QS-mE#j_Uhonau^n5_h*)3Xk(GoSV>SUEh8L*6+_f8@)TP#SK1OKFJxZch7t
zWpd(bEnHzP7^}5nH(WNl+!LhcBCetyq%|BnrI|jE0RS<%*$H$?;roi~-Xk?Z7g>Pu
z(Jnx+;_L<wWeuI;N&rErLn*;pa={^6{0-^vb;+t>HbAek8+HhLf0)u-RZu831Tg5~
zq-2klP+$<-e`&Z(s8?3z4Po19z_sbV84TMTrtAC2)rW@rqHzYi&$j}pD5^)fpLpMb
zFzX7+M7W{Yq%n<0^k?r2VP+=_^4NN;`Z3wd2F-WvTpQf_Yd9nf1(v~kZN*JEMu^7m
zW5@tM4jH8WsLA1m^V9ZhA0ok?1_%fN?YRPEtgzg|e_bsSc7tm$MBVWO^cj4#+y%>V
z1UzdX8ayrKM`>lUb<i@*pY)oM^HV)TlvLoMp;4#(={+b832??%*^a?2v}d=t<-_yS
zhv+AV-K^UqB9ZeAl7}Z7ml23@IwWJxItNOA>yjp>l3sq=s>C3NVyh|ATjkf&$3BJO
ziNNJBe+IO3YoEp1)tXI-T6Iz>HQ+(zIS7aREXM&8e|bxG`QuCQ5~n={N^euIGp(L<
zsGN6;GTps5R?FG~g!eOB?cb>(8y0J}&Hcj7n}Kl5zbD<ItBCHBa1pyDGdl^M+OSCo
zvc{bTh>o~H6<M0Oa=0K9F7mQLaS7?rDN-^_f7~8|ILyrHtQ<cJLe-iZMb$nPl{iV1
z+18XG=bHq;3X2_AGkYlLBiFW=rLe-woo0bGtue+kv5_;&Z7C(dG3Q=v3sE8~?&Ru}
zpb2i}AfD^MLtH5dxYsjy7euJ755hBOk=ShKg5xJ}hUbE}rMRdZY20xRYJRB%{4I#U
zf1Z2WpHHehJxK&rh{Zy-AkrSXgZYMSyu<tc(vkdqDt0)lGC@s0PXL^GYP7iF;b2Ef
z7T|c>xOoZT3&>L)H~L_P;aLV4z5!XhWr8W52OgEma$|dV%io*TcK;T-*ZUwHta#Hz
zOLqp31*MDV#AJjM9ni6zRxhwJ9Z~^Fe;hwnZMY`^yI3x9NKv`gk)rYey}>gp-VP2F
zG}(;KPswEb$#wfYo?1>t;pfO=)U(4~>&)&NY}&+c+QBhfVj9l*9h%7jnUVEF$Z0O5
zkHs&`j@&<6oM|*sdld&zCJvL^t99H_!p=OWJ&!lZIypZ9)7xS4i}39qRr_(>f6IX|
zjivfwt?Vuo@@Fbt5xW#!pTn(;8+W+w7umo)8J^(rU36ztd|Vm_wkP_!BHrFN0!LJ2
zsySUDWw`UHU_cn!FU$`U4&qv$J~ekBJ+j@AN@}$I;A+IM!EjeeZdP9`UR_ypikeX3
zg9(QCjI_g8I3(@=#y-X94tYbmf7T3|`r(qOUKyHQg6?O8eu2y2mKD{*12PM)Z4Y->
zOvgC~N75d7xyN(qxCel#NsAgQJ3$RZSg}7&?_vMZrS~dhOviAXMK(%uPA@=}#AyMq
zMx1sGPUd>w#s{Xc3H6ASGN)x^_<^(j+#b~TM-!}Ji02xZz{)8+cpm`?e+5C>-Vow4
zd6g{qW*<&CV2*aI$qsG<xL2FC`_nH3i)2_TkfU5L_Dk_Tc=$2B7>X>?o1-QOCRI+p
zRP2k(i5-$U=}xNSTT;61EsdntIt=QxQF|4YMOA$B!DwcGv`6Us+N>?}8X(Z*7QXHs
zW2inHa*`eJm3VQL-4OZTf3q}PR=Xf~)dCGW4D8{DvKKFm`s+Ki_P&P1mRBC4kGA0<
zUdt2sgFQFhDelZmJtKKl*c%T9_Xz)@lGYH!AM$nkX9#<LQm%u`i&pXd-tG%LzuLyv
z;sk48Z(-tF!C-YOQc{!pE>^qZf@2j8M-=;xXuIYU|CLPak*aA%e=eLE2pP2FkU?9Q
zeL%3{uB5kdyO~$#WCSutZpfO7VwBkfv)x*TRVN@}mBi>(V~*wKc%7*F(Qux0@O~D&
z*1=80-F_0I5w8y$+Ll!CeTp^51{Q7e@j)xh0q>s^OFp1DP6Q4FtU^1CTZN}CBOFY=
z_AESS-Q7p|UyJOAe`)@oV@m!%CH8+gYxrN$5WVZ%L0%QnL4z3()<VklT`}Oeduv`j
z&>I9P0a4H8rdaB{a*|?G;1T%YstlKEf@@UV#YeS~%EvwwtE91)GQNFrvAjJ0;ZN`0
z9>_EWFD5Ug3yO81_8ENR@~Z>G`{*s+MBf?FmRp8vL^g~Xf4IU|ZY(+4G))I4-gOzk
ztl>^mt^Fls?hYXw*0T_I_xRbw5#H4Rhr|O2&GJTZ1820UhT)AQT+1>XzJ;l{GsSk4
zHA5=1S5yW#vSwBYLTwk`eq+T-xMRqdw5?WKW;VrY6)r(0co7aQievma+)Tj*n4#Sh
zO(?7E?Hrl^e_*0OYh$RZ7YncVW*3DudGfy#U{9Y??4ewutS3LdJpXTZGygwWUnBGD
zko|FGHh6Oo+`5z5bdXVW39Fcl{Y@n4VOW`8XB%4aa!-D@@UuGkehHHtGMN3nt`p>y
z!DWWzWV+N*KWeq3ni{ynPv+M%GIsdyM%J(f*WEcve{LkVmulTorKm7_q7BclS+B13
z1aFb^yaV!yYog)4gQj#BjQ~I2&=Lf<-Bt4Y?Afy}pzpaT{28MXt9@<07#8JEL}9XE
ze@pjiXTL#!<Rw0kN2K1D94@JGY25`}iHSw}#}uc0l<co3%G5WxT93)NlZU#RI7GjF
z_1ERw9#=2^a`tZKao>uonfqTLvKr2jjQ_~TAQGq3wq#|q$$t<%1jdp=Ta(?q9RW47
z`@B6P1jdp=Ta!HC9RW3y`@9>ob>OHPBF2(JTO>B$3KtXr08lpo03iSX0C8n)bzgC9
za4j${FfKKdKl>Gb>v9{nlKzjUXe(7mQo<QtBws=wX^&z%UTrK}K1zJHDwP$%nI?${
z1B?L-DQ0V{&cA(zt=g@9xAO%12wUBl84l&EZ0ScLlR%^I{dEI6!yt1RWby``SJsSS
zlpczgvcNIC#iB~z7Nuyv8>EBuzPPB0Lb~-BUOzkg1<tI0QNcPmcQ)9}YS3!xq+5%B
zwgo1ooZ}c4A(VdH@6T1xa++qg=o9gzboO`5Lcd&xg*6F(MhBi$VPRc;z<FF$f0$q)
zm5$6MIWAHE#aJ^pRchT_oJnPZR0iD`{>3f~To}9ReA?{(-5b05i^F^&1F<9OLy}+q
z`eF>kIWgXUYZ+gyRw=8)x_N(l@l%^s1)3m(vSzb`S38t>SYU25pDNq0#q~Gk?W5^y
z8PJCtw4W@bnWMJzucli_Wzt0`^fy^UbNi@M?<@4Z+nTDM^cC;irgmW~m*K6ci6>fm
zuV#wu_mqrlOVTY|1Q{w1<1rjS0w=m$$gQ=fCZNfGF^7{XS7<V%r8NO`erxRum7(#x
z;?>0|oRp=rOU&ut&tLumFVK5*@Ei>~so}huYL&H{mLsqZNU09?UOYd4u^fH3e;e{2
zrn|?~wu!xzp`z6<q?uQ8&Qzb7xmJF$weD4kP6lP>aIy9Qi+DPmnjA|~j0s!I&z+q+
zSroK?p-cLl%@Ng16pUWqc3=AF);qqx^^W1-_FIOx-!i)WmV?`GIo#Uvvn^Fd-)s!|
ztf;gM_J$k(ye^6Cn191u{1x9;%Arpp@i819kujd4&zveLIVDdlg-J+g!x#cyhyHcq
zd=Rx4CO;~TV|a1;<k`!MXYc>?^7;9T5e55yQswBAXV<~WF>oT@u<%s*FotqXspt6U
zemY8rj{#{Z938<9uim-2JhMx5a*oGG57J>e`1FREmRI<Q=nvAvPcK*AI_74Wjy}D*
zSeNVLqhWgZIh>ShtxPyR8l}UB>A|Phdkj?>tkwSb=s_)QOof-rSZm?<2zJN?cR-qd
z{F4NOy(V)HAAJtDm0PKtnf)9^@alvrj#dVbj|S=RGbB7;&na<xI_CTLGIi61LZv>J
z(NL*dv-!R~A7PT0-(+!GDV--&b>A^)xlrbsoa<nl&SuNIPQZIEUC1xP0t+N>8*T^6
z$9A_abm_C=_~=nO`0&Y3*G3j98-uQYLwcq(3ef<(4<Wf1HIh5}ydNR{hB~ta-h;`c
zp*@*^@*rXI-Xp}H-;1z_>wY19K-X6$ptEKFZBZt^#EenQ@#IqM{p0Wd{rc?uAAkR!
z{RVTw%Bc|0fHHNC?Lp-!O-`g=n@riwfefBv(D9iE{hCO>UfLX#!f7BGDvvpT1PdNb
zE?%FVLxJAQIePlgRmSi+Eav;AI-XntHo2Ur3Rl~-DF_GYqwhH07t({VP)L*0>NPZV
zJX7dXA%0#c-wK^c15<4I%WP3pMrCDORag{iyPcs0N$D7R1{k_KB_*U8K)Q#LmT;s)
zN(Q7u7(g27ZV-m<kP-zXB_ssF^Xz@jKL7K(*Y&sFxLFr#J>M(HRB2%H_)8P+6%#lA
z8E2mHH_V)|2aLad+TwX*vYC04*kzBZsMlWI{2Y0y)6npl1!02TIWe%Y9^XxIOiCOG
zVN3q1X=Qu4NTqWX6w!x*g%c9^JjE_HG&zs9FH`qJzq4k>)C`V2L-!xlpVyo;WlQpX
zr3g$0IkaHxg&2CqZYOCRbY{6=Ni-T-`HpfM=)DHFn&CpXcq*S@Nnr}Va87$rfKi#c
zU^kIKGg330j7W;q5Qv=GW^Go>BV(#x-8vCHE1-8ozt4i)_}5RdTEtrI;W=+^?depC
zc$&CmG8Y|X`u0Qts;WuArdsJ}Uvt!b!G+6#Rh_{E-+DJHIckM-V^>7%i_6D~!np$m
zBZFnOsLn;l-P<t<0WC`&H*jehxa<mCqcfo+*PmmKfZur7m!*vDWQ=|AMcs*3KheX@
zo#JRL#If>ls|dO!e`-8bZPoa6xp3Oupv=en_?L908#dHA?ES0~SH$(FKC%lQ_;}?l
z<W2CVYRvGYrUaufUt&v~8MW7=L})n$=z{Fd)=P9Nn?;pF$a@CY*#4c@60dP@QC_G5
zTlI(*;;j=oEc0>9(o<~_{L5dNWBgf+GPexVYSyC{#xvo3jrj8GZjNIZrH>|`F=9ol
z(JZymGdPY#`V7m8Dt%mo(cuXtlV|cg$A>9E4DwpS$_NSkUfO3^n_<^@9fAmQZS|RX
z`Rl>oL-E^X<LYWb){V9D=5w^{%U!v!D>pU)#Ix`*@0+Pj4YhP019>CV_i4(sX}*Qt
zN<kXzG*oauj>)?PSG>KQVM#bKxfn5<!d9%zrpfV=zx^5oPn|(0#>BzPL<%XTk&Y>`
zrim9?v&3C!$^m(X=Q7u2SzFoH*$>I`mIALo2Vp3^?W^EM=Gjjjl*n#1#^FH7(&p5e
z5%rORctIW$mt{mFFUAOEhY!&N+Es-4(?lD0KX&5bq}xM@OPxcRfdZtT8?q+TGCfe}
zdDoIUjc9UhPI!^(FlzUCggKdUq%U5(cD0{^15IFc>vVT#MqzIzI(1G=+E5wS<P#Ug
zRwU~E{<X5Rv4NyCG`~Yh-*D?&63TlU!C$1tRzII<u9k*8pGQY=I;4n3U{}B6-960I
zt`>(#Rlp7dXzekX9fc0Oh}LY7oMY}S%5~qq5}=bL0!73!>aAufs}!ad>Xc4ZNw^$S
zpyuJtZCm?l`P~wa=suMss3?RrRXj#M;M`7INZwoCA(Elg*BT_&*uz-XlI)~Jlyfiy
z*1W}|f^N&?wAc_DtBjvS!B+IhlM)4VK00-jaB1%sZj(O^B$V=)AQKT=P=Dmd;rxAQ
z?fV?}bku;+au{)16^N=T`LXC~Nsf=)5%<gcv-xGA>t2hZ@2Mc);+fVD*k@>D-H#B9
zz95eE2$Iqv+%ge~kj0mkX?hOE2vj8PU|CD5=AJ8X_uzasW=q2`@uMPuezpd8yl*_!
zD8>>mF)r(x0G3&agV2;$B%Ssr=Fcr9{1DR^u#{{r7wGwOlf;P}H%)i(OS5ujVD_^M
z={EK@R(K$^tnW?kIM!)W32rJL2ca&7Dr2Cpu6nQuT(Ovq{-=!_<=JTfqKQl607H0~
zzHlHxdVH~Faw6Y}%cLQ4k*o<w>vVjSgndF>CAY%8_K9a<q)sg!T^-Nu>ECSui48K%
zby%Pt>sxnc-8$}iPo=9Nxz9ZaA1i+q)b0WGIT$$Su49Ku+V`!y?vB$8cnqC<C%UU?
zzi8XIw=FI1c$W#?@T}cH;Fq}WbJbHRe@em!le7+)mXgvh(7jXP>F$ueGiG|g|EdY=
z);n226P#d&R42~xwYcALUDR(^JY*Kb1fu~T3dO8A3ZSWP+I%J5b%_8q>@<=*ix>s0
zK5M`S%r6^w!H1G1Ma0)2CT169`_k$+Il3+8V_fBhKlQTY?`<3rg?>h8tIC|n`XM8b
zr+RV)Rqx}X@K}+#OQm&&Aw<XSb^N;*5lvybH+FirwqNp-tV7#Y;rFdN3DxgRzht>r
zPcx|_c`DI(d3anOwY4g0a|t=@#4H6|{P8@X%FoPajq>5CpwKq|3?pw#DvyO8Zwiay
zvQ1}cEJ%HYyGA?WNId^kV!ISvrm|SHWrh>us1WL&Xnrw;R4*}nm7|@cGSYCtfw~ga
z|G@l3dTy&JOYoNIvr_b?TWI_M%TU;hVlTGQn-3)Y4Z8Bro?GvRRan?K*4t??MC3H9
zFuku<{Ddm?3{|skNo#2ooWB?|nscZKBuRFB3ukSw6%;x^#0|^%r=kh8NzM0KUGN3U
z33B>p0tfGI(0Vdg9?(51u^;3LIaq?ZLs2^L7<ZWS9re?Nswn(Ts-_tUe>akL(Y|x_
zwPS0dAQ4GFj!u!=G7?<Q4$}{okEJ|u_T2QhaeBP5lbC{EtJ)OHW^h%pP$P|Lyha(4
zF+%ZM`z6v5!Elb2bYcI@Ud?B=AP=>ecpG_K3(A%dl`5faW=I1WH_6KaK}`K^0Rk_y
zRtVRCHT-%gGdZzHwByMKUT>fik8?u;zEvsM@0f{cIQnV2EJ$qkmOg6Y$c-Us!4^UB
zHA|s0`$^G#PmYtCw=%Cr#Qk@~?*q9=5*@Qz?)xgl9qE=KELtIfRGOOY#H^^_@7v=l
zo{p*(2+$d25j6ipmb{lyk=jjsWv9SztRbqk-f&qH{?SH|)9@$Jx9ha-dpAM#$PCT4
z9)w`4M7qmm6z{$+4!`kgsG_wud63rG`y5t?a=pO>ZCpdhwNchDvkWg?rjO;>bH|?X
zbUNXP!s9uyV0!|TUH}qmtc;uXq)fP8!5~D#4O!#k)Px*?JTh`Q?6&Eqf&e>;^~a?*
z_MT(bIIN!ISRYis9`cvlj8A!8Qxb{K9rks~L%LN}B=2T*VaQC6t=FPI{~jw>2j8B)
z>jL>e2~b^28LZB~Tv>Fx_)rf-<u;3<qwS2?*5q=}YdL2L>l*oh4t<sxlL{?szdT{2
zD~XDWYo5Df&$5?dXR{xiGcm2IcqL%G*jaN@9It#<82fTvw_C0S#^3yUmolSTT8ty|
z*IHY{)2QFs5(G#B?)3sV4$0#>iay&9hZy&w(hf%OQJO~mENRLath5)yP%kx5h(;SC
z^H@hfjboZKyMLYLbc#qC=k^O-B~G+pz)RCp)%y64F;beYrFYzsj;Xz{M86c1Gzt5D
z#cF1YL!o}iZfU{8UOt*RgBs_G*@L#_ZEMXg1CTdH5PO2rPD<aSldO_bu7>EL!U^8q
zd-};DX9Ia8{&S4tw^933ObqT=F;3P9eKC{Q!i!-HdK22jMh55G@yF9;{IO1Xra2Bc
zl4;BH8hSOW=&(1e@#XwHPo`_iOto4!TqkVJT%U^+hxKl=MGwCkY8_i}Ru5&Nr}VL8
zjHaj(l!?HDrmv<1PMAnZQ@DTq7Rvm-z>M+sm8}9Jcz1Gwm>8FC5&B^@E}v`^Vdohj
zzIjny=qI&RQ#W_;amw&5)hS(P3Vc$dV#gpnB(<P|bwd|mY??<EbLC94|5@|MFz*fg
z)(iE>?-v{1rAqkpot?NEPSKtAgHNe-B@0hZuqi^tkE@S*Xw;r9GWU+ifGdZ15Cf}{
zocCPD1Zk@03$Lb^OzvDnZ3$Z05G6#j!>iWOt)Eph-~pVuTqyl4NSN?5D9t7Eyt!Gp
zzaR8<;;otG>&t^^-HVHWzQv0{2f=2y@hk}#)3%JZ5X*X+Kf5hQhkzC&M{GuP(`6PE
z`)%}VLm9~f&0_|lZmb{twQa6U@ZX*R(yb_mAXZzbq!wJAh8<f=Rr%Q(g4{gCF3)zo
zR-}cX2}i_C2elw;fq6IW^H}07k*$=pau%!JUS4DFC4E2pRZ9BGidWLg&q%hDST(CG
zr2LrPcrKacp6j9M7f3jC`1)F2=;2nVvzdRUuFJ<GeUt0(ekqds&05NPM@m01REw0|
zJHrS|QXU$1Bh741nuu49DDdkao9pl*-6#1d$XaP95j(@t)%nHhGyosB*c=Y^MstGj
zMJ%yuVJd0utybg!Z=BCw6JNw#bi<KDk06BV?tStu9_kN>2dO@vX1fFoPX*Vw<q1&9
zW8NXnDNp7klMYml{6Bl+CQNF9eb<;y4_+VbSV)b>1t+Oskkb<(oN)%7yah+%e$9+f
zgp9zpkP(IZvTE^Nn+?4YF>PoF^3uJ0{ZVHo@(U#a-5Av)d}V-hlFchG;z5u4H#X(D
z*Sl+|dYYE(aeI;hf=r(NWZ1e>W`&|e%jV_sC;Ky(YR|nI19)k>8VonJE)bO#gj>{J
zAht3k)j|-6OBaZ^`X07^N7#w$ukgJ$YQ3nv;Oc#1w$QfFoGC5a;_}?Dk~drXYJezs
zb?-b*^`m3*r*h5v^L$Q$xy}4C-!y*)9?>O>M?r)=dz?0zXrwix!YAc11t*;cLhofZ
zQJKnL*R*>hl^yna^PSX|V5MB{D#|<|8+UD-St4CaR&lt9Y>$B2T0Lg7fo90=<OK4z
zKiY~q&z-sodi0IA1Mbt^w65>?Bp@9sxtGm1TBDnr-J!94xF)E=XCuzTsboJd26OiL
z-K(>YtJm-8S@P0~2}gLba7VzXHsGNC;1kP8o<lQ!zET-7%~$fRf28nk8QqP@rf*b+
z$2faNMd}~|6!M$GCaLRt%$mgm{=&^R21xZwZv{HY-72#NW_x4!u^tLtqE+0ZuxHq@
z!A2b4u2DwcuK5_8eCnnyXK@TbWD7!tW%t*87VRT~RM2P0prYwqgYVNZjElAT>&iw9
z40HRxbe}0{*8tu8l#!ptBMA(maG*ME3Y-Punr#Sk?($(}>6ZNY-!%zh^z<NlkUN`r
z2VJgk&v96I2hr)3azOBLv8IX+%hIxx;)9sl9EcqnSFxlko7)!f?li4mFKta&VoO*;
z11BkEI^0Y|OTc?KxB1qGa^rsY6M|D=b2k@-X8tI?YSIQwg~fPZjUVJ(7qUk_GEqZL
zav{1bKUdc+7OobbnI+7`SxYxEo`zD&6eR7PW=~U5G=l@P3;p^U2%Nb<i!+(h2YsBe
z3{PmfIn4?dy1`J$=*HG5are%3bVBJ|6{XzGS2@hh8Iyc1hpqz0qKBg)(pRtf6}^wc
z6PAwJ-Lmq9c`R$cceBQ+B6Jm865a82D-c>n;9ZSC*-h(XyC|RhmXgbHAHJ5KqTw|`
z`(t;?AtwDJ)&x#Y19i{si9!Q<PT*Wyo>QbE2K$(2ceGp=X2I772bpS14}Y>UhSzZF
zYDKnn_?WbfKH=c!c0HE56O_5H9o@Nmu%>G)yM!Tl6CSBT6J9gzS8>QSM_iOOdheV{
zTWA913=w0l;prA!F$WBdiMk4Y>bLs=1Z4DZcIKeu6A>tV$5!&oeOLjpu<H5zkE<Jy
zEGW>!@|WDxCzx2|0000F@R!@@vx7b~2LJ$O0002=FSjvWfdt5CM)Zu<vcLz*+)z)D
zFKmTk8dY-NxRRAC7@=GDpY78}|9I60&!9Svc^&~f2yWdKE)6ao<o>$6Ekf`1o05pJ
zpwT@t=^X5WUlvGq2mJ5$>Fp#y2B1j#G6~T4Z}&3+`{oxe006)S007{B&*Mo7<iz`H
z?wsckQ2IMkAPqu<*Us77hsW97!GljvSxHClaoU`EVvz|s_SB1NY?g1bn+k+IRz0zM
z0#OeNg=F$Y0X1C)>@WqQLexr6>s41TU7g=g`jva3hojXw{U=2mZ^~lu<bMzLBAd{H
z2H3h1(GO`&$~dZxFuzQQRG8}0u35F;x5h|>&0?Ly18fn?;(ez~(_u0S*`n$#pEI9#
zPcL=iA5~gCw@7+2Ov8DUN0A@Diq%W=Rz}SdCZ>bSl|jf$&6k1v8g-|5<=#quQOyG~
z{``qzR+;r`C7acQkCa&rhu%~K24cVM#Rzx?wy{)M?|(}0Rs$@d@~rX2`kH|FE{zV=
zfyW1Sb6OUJc;cJ6=6Q!%^6h0{c(KB7oW-G57mM@v-`6*{PK4)?R&U?iB`7w_Hzr>4
z(Xj~oV&D>#z`R%A=ZiYb4@a1mE;ZQ|H5#l+=~S=?6`rT%v<J-*76vpvefW+e?z-fL
zbb>#09B5WfV=H<HT9N+QmhWDXm6*{z=3H;0_gm8o;p1L8v}0=cnB!qTG3)J}?fNZu
zh(0-MjDzI|K@U}@ZyZ(8VFwasV`JrJ1#34IRBCqCd{k7uU77EW`ca(q?y_@PlZ4p=
zM0Xb)604s6Ln;Yi(f&6c@m;7?5hh?vT*S6_1b`g#{znN5JQ0e0squo|S7A0tu)M=&
z@R4T#V#)9wy>8En_Y_w3`i(k(G>nOQOUK;;jj8g&iAnvynUA-e$i1VSiaNj(e0;+<
zxR~OV&9~!%L!UzuRv>UgXSQ=@QUSqTsVR@|#;fD%XflRqY`apNB@z*DwY8U)dFBoH
zOiM)AIQ_EZpkDzE%GMH}ym>nt|90)sOHun2gs;A6NI4od=X#n;a!O9I=r=9H&uKWG
zdoWiqr4r#Wk+B*YhuBeuluxk-M*!Gv<yaI)p2qrXcdR_o76%hsodngF<BAs7`unJN
zt&6*2-&vK+tjL5Ql{|NRSYA%F#u7n1iLOejHs(zy8S>3`KvplrMZ>}sQ?^j#E@BFZ
z%$6mhh4IJ!M7^9jLoZ8YVC3`7bX^miWuFN{wuapNZTx~`r^F>f4(0ZH+m^C;0U^th
zj*8m9pQl%5f0Q}2{y7QBYW{gdoYruLZ^6A%1Rk~$%DGulO_uvkSYo2%>)dpGi8UMO
zO`2neAj+_26kRr6pueyQdre7J9IbrC8h_k0=Wji8VG1w+UuE86PL`qAaqx>tPU_ua
z9oNW6zAkPw8&Logf&YDJH2stks1IC9SEd3oG5vLe^#4De*a85Qfd2p$`oKOtk_xDZ
zf%PhVgbJvDfuEiJkQyk5!P1znMGXY~ous6DPy=bna6A9ZOVYEcfe`+`6oV0m_c)Ul
z08mo_0C@jX9)JpU*!+|AS0a;6NCPDOoB3xV^AAv=Pj3HYrfbsx8UJSesjUA175a`C
z_(!Ufo=XFy|KB}2X@In35+wh6u}AZNd;F=8{{R&_^>3CuEs*W+Y5vm7I7|Rw9~%H*
b{D%Sn02)H+NdMI<l@`d1vqcUB00933Zb0Pm

diff --git a/pythondeps.toml b/pythondeps.toml
index a071774242..0949a6e6cd 100644
--- a/pythondeps.toml
+++ b/pythondeps.toml
@@ -36,7 +36,7 @@ sphinx_rtd_theme = { accepted = ">=0.5", installed = "1.2.2" }
 # only include dependencies that can be guaranteed via configure from
 # system packages, or python packages we vendor.
 [tooling]
-"qemu.qmp" = { accepted = ">=0.0.5", installed = "0.0.5" }
+"qemu.qmp" = { accepted = ">=0.0.5", installed = "0.0.6" }
 "qemu" = { path = "python/" }
 # NB: The following dependencies should be a little bit more modern than
 # the versions listed here, but we are still using Debian 11 for several
-- 
2.53.0

[PATCH 02/16] gitlab: ensure all meson jobs capture build/meson-logs by default

[Daniel P. Berrangé]

The build-without-defaults and build-tci jobs do not capture any
artifacts, despite running 'make check'. This has proved a repeated
bug with CI jobs, so introduce a new '.meson_job_template' rule
which always captures 'build/meson-logs'. Jobs can still provide
their own 'artifacts:' config which will override this default
behaviour.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 .gitlab-ci.d/buildtest-template.yml | 30 ++++++++++++++++++-----------
 1 file changed, 19 insertions(+), 11 deletions(-)

diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml
index d866cb12bb..005058625e 100644
--- a/.gitlab-ci.d/buildtest-template.yml
+++ b/.gitlab-ci.d/buildtest-template.yml
@@ -1,5 +1,20 @@
-.native_build_job_template:
+
+# Any job running meson should capture meson logs
+# by default. Some jobs might override the artifacts
+# to capture further files
+.meson_job_template:
   extends: .base_job_template
+  artifacts:
+    name: "$CI_JOB_NAME-$CI_COMMIT_REF_SLUG"
+    when: always
+    expire_in: 7 days
+    paths:
+      - build/meson-logs
+    reports:
+      junit: build/meson-logs/*.junit.xml
+
+.native_build_job_template:
+  extends: .meson_job_template
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:$QEMU_CI_CONTAINER_TAG
   cache:
@@ -60,7 +75,7 @@
       - build/**/*.c.o.d
 
 .common_test_job_template:
-  extends: .base_job_template
+  extends: .meson_job_template
   stage: test
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:$QEMU_CI_CONTAINER_TAG
   script:
@@ -87,14 +102,7 @@
     # Prevent logs from the build job that run earlier
     # from being duplicated in the test job artifacts
     - rm -f build/meson-logs/*
-  artifacts:
-    name: "$CI_JOB_NAME-$CI_COMMIT_REF_SLUG"
-    when: always
-    expire_in: 7 days
-    paths:
-      - build/meson-logs
-    reports:
-      junit: build/meson-logs/*.junit.xml
+
 
 .functional_test_job_template:
   extends: .common_test_job_template
@@ -125,7 +133,7 @@
     QEMU_JOB_FUNCTIONAL: 1
 
 .wasm_build_job_template:
-  extends: .base_job_template
+  extends: .meson_job_template
   stage: build
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:$QEMU_CI_CONTAINER_TAG
   before_script:
-- 
2.53.0

[PATCH 03/16] tests: print reason when I/O test is skipped in TAP mode

[Daniel P. Berrangé]

The TAP output on a skipped test:

  ok raw 181 # SKIP

is not informative.

The test program included a reason, and that should be displayed
in TAP mode (it is already shown in non-TAP mode):

  ok raw 181 # SKIP Postcopy is not supported

Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 tests/qemu-iotests/testrunner.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/qemu-iotests/testrunner.py b/tests/qemu-iotests/testrunner.py
index e2a3658994..dbe2dddc32 100644
--- a/tests/qemu-iotests/testrunner.py
+++ b/tests/qemu-iotests/testrunner.py
@@ -174,7 +174,7 @@ def test_print_one_line(self, test: str,
             elif status == 'fail':
                 print(f'not ok {self.env.imgfmt} {test}')
             elif status == 'not run':
-                print(f'ok {self.env.imgfmt} {test} # SKIP')
+                print(f'ok {self.env.imgfmt} {test} # SKIP {description}')
             return
 
         if lasttime:
-- 
2.53.0

[PATCH 04/16] tests: remove redundant meson suite for iotests

[Daniel P. Berrangé]

If a test is in the 'block-slow' or 'block-thorough' suite, there is no
need to also add it to the 'slow' or 'thorough' suites.

Tested-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 tests/qemu-iotests/meson.build | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/qemu-iotests/meson.build b/tests/qemu-iotests/meson.build
index d7bae71ced..bf588cc2c9 100644
--- a/tests/qemu-iotests/meson.build
+++ b/tests/qemu-iotests/meson.build
@@ -33,7 +33,7 @@ foreach format, speed: qemu_iotests_formats
   if speed == 'quick'
     suites = 'block'
   else
-    suites = ['block-' + speed, speed]
+    suites = ['block-' + speed]
   endif
 
   args = ['-tap', '-' + format]
-- 
2.53.0

[PATCH 05/16] tests: ensure all qcow2 I/O tests are able to be run via make

[Daniel P. Berrangé]

For block formats marked as 'quick', only tests in the 'auto' group are
added to the meson test suite.

The result of this is that qcow2 tests not in the 'auto' group cannot be
run at all, even if passing SPEED=slow or SPEED=thorough.

To fix this we need todo two passes over the I/O test list. First add
all tests from 'auto' group into the 'block' suite, so they are run by
default. Then on the second pass add any tests which were not in 'auto'
into the 'block-slow' suite, so they get run when SPEED=slow or
SPEED=thorough.

Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 tests/qemu-iotests/meson.build | 48 ++++++++++++++++++++++++++++++----
 1 file changed, 43 insertions(+), 5 deletions(-)

diff --git a/tests/qemu-iotests/meson.build b/tests/qemu-iotests/meson.build
index bf588cc2c9..1a24d801a3 100644
--- a/tests/qemu-iotests/meson.build
+++ b/tests/qemu-iotests/meson.build
@@ -30,16 +30,48 @@ endforeach
 qemu_iotests_check_cmd = files('check')
 
 foreach format, speed: qemu_iotests_formats
+  # Formats tagged 'quick' get the subset of tests in the 'auto'
+  # group, run by default with 'make check' / 'make check-block'
+  seen = []
   if speed == 'quick'
-    suites = 'block'
+    args = ['-tap', '-' + format, '-g', 'auto']
+    suites = ['block']
+
+    rc = run_command(
+      [python, qemu_iotests_check_cmd] + args + ['-n'],
+      check: true,
+    )
+
+    foreach item: rc.stdout().strip().split()
+      seen += item
+      args = [qemu_iotests_check_cmd,
+              '-tap', '-' + format, item,
+              '--source-dir', meson.current_source_dir(),
+              '--build-dir', meson.current_build_dir()]
+      # Some individual tests take as long as 45 seconds
+      # Bump the timeout to 3 minutes for some headroom
+      # on slow machines to minimize spurious failures
+      test('io-' + format + '-' + item,
+           python,
+           args: args,
+           depends: qemu_iotests_binaries,
+           env: qemu_iotests_env,
+           protocol: 'tap',
+           timeout: 180,
+           suite: suites)
+    endforeach
+  endif
+
+  suites = []
+  # Any format tagged quick or slow also gets added to slow
+  # otherwise its tagged thorough
+  if speed != 'thorough'
+    suites += ['block-slow']
   else
-    suites = ['block-' + speed]
+    suites += ['block-thorough']
   endif
 
   args = ['-tap', '-' + format]
-  if speed == 'quick'
-      args += ['-g', 'auto']
-  endif
 
   rc = run_command(
       [python, qemu_iotests_check_cmd] + args + ['-n'],
@@ -47,6 +79,12 @@ foreach format, speed: qemu_iotests_formats
   )
 
   foreach item: rc.stdout().strip().split()
+      # Skip any tests already added from the 'auto' group
+      # as they're run in the 'quick' suite already
+      if item in seen
+          continue
+      endif
+
       args = [qemu_iotests_check_cmd,
               '-tap', '-' + format, item,
               '--source-dir', meson.current_source_dir(),
-- 
2.53.0

[PATCH 06/16] scripts/mtest2make: ensure output has stable sorting

[Daniel P. Berrangé]

When debugging mtest2make.py changes it is important to be able to
compare the old and new output. This requires that any lists in the
output have stable sort ordering.

Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 scripts/mtest2make.py | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/scripts/mtest2make.py b/scripts/mtest2make.py
index 4b252defc3..915f02d600 100644
--- a/scripts/mtest2make.py
+++ b/scripts/mtest2make.py
@@ -67,8 +67,10 @@ def process_tests(test, targets, suites):
             suites[s].speeds.add('thorough')
 
 def emit_prolog(suites, prefix):
-    all_targets = ' '.join((f'{prefix}-{k}' for k in suites.keys()))
-    all_xml = ' '.join((f'{prefix}-report-{k}.junit.xml' for k in suites.keys()))
+    all_targets = ' '.join((f'{prefix}-{k}'
+                            for k in sorted(suites.keys())))
+    all_xml = ' '.join((f'{prefix}-report-{k}.junit.xml'
+                        for k in sorted(suites.keys())))
     print()
     print(f'all-{prefix}-targets = {all_targets}')
     print(f'all-{prefix}-xml = {all_xml}')
@@ -81,12 +83,12 @@ def emit_prolog(suites, prefix):
     print(f'\t$(MAKE) {prefix}$* MTESTARGS="$(MTESTARGS) --logbase {prefix}-report$*" && ln -f meson-logs/$@ .')
 
 def emit_suite(name, suite, prefix):
-    deps = ' '.join(suite.deps)
+    deps = ' '.join(sorted(suite.deps))
     print()
     print(f'.{prefix}-{name}.deps = {deps}')
     print(f'.ninja-goals.check-build += $(.{prefix}-{name}.deps)')
 
-    names = ' '.join(suite.names(name))
+    names = ' '.join(sorted(suite.names(name)))
     targets = f'{prefix}-{name} {prefix}-report-{name}.junit.xml'
     if not name.endswith('-slow') and not name.endswith('-thorough'):
         targets += f' {prefix} {prefix}-report.junit.xml'
-- 
2.53.0

[PATCH 07/16] scripts/mtest2make: support optional tests grouping

[Daniel P. Berrangé]

Currently tests can be classified into three speed groups depending on
whether the meson suite name ends in '-slow' or '-thorough' or neither.

This gets turned into make targets that match the name of the meson
suite, with the speed suffix stripped. e.g.

 * suite=block ->  'make check-block'
 * suite=block-slow -> 'make check-block SPEED=slow'
 * suite=block-thorough -> 'make check-block SPEED=thorough'

The set of tests under the "thorough" speed, however, can get rather
large and it would be useful to have a way to expose further make
targets for directly running a particular subset of tests.

This needs a way to run a target without requiring the SPEED variable,
while also not having them enabled by default as if they were 'quick'
tests.

This modifies mtest2make.py to support this idea by allowing for a new
suffix '-optional' on a suite. When this is present, a correspondingly
named make target will be created without the '-optional' suffix which
will never be run automatically.

This is intended to be combined with use of other suites. For example,
a single NBD test might be added to two suites, 'block-thorough' and
'block-nbd-optional'.

This would allow running it as part of all the block tests with
'make check-block SPEED=thorough', and as part of a standalone target
'make check-block-nbd'.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 scripts/mtest2make.py | 22 +++++++++++++++-------
 1 file changed, 15 insertions(+), 7 deletions(-)

diff --git a/scripts/mtest2make.py b/scripts/mtest2make.py
index 915f02d600..383ea68b16 100644
--- a/scripts/mtest2make.py
+++ b/scripts/mtest2make.py
@@ -22,7 +22,7 @@ def names(self, base):
 print(r'''
 SPEED = quick
 
-.speed.quick = $(sort $(filter-out %-slow %-thorough, $1))
+.speed.quick = $(sort $(filter-out %-slow %-thorough %-optional, $1))
 .speed.slow = $(sort $(filter-out %-thorough, $1))
 .speed.thorough = $(sort $1)
 
@@ -66,10 +66,15 @@ def process_tests(test, targets, suites):
             s = s[:-9]
             suites[s].speeds.add('thorough')
 
+def target_name(suite):
+    if suite.endswith('-optional'):
+        return suite[0:-9]
+    return suite
+
 def emit_prolog(suites, prefix):
-    all_targets = ' '.join((f'{prefix}-{k}'
+    all_targets = ' '.join((f'{prefix}-{target_name(k)}'
                             for k in sorted(suites.keys())))
-    all_xml = ' '.join((f'{prefix}-report-{k}.junit.xml'
+    all_xml = ' '.join((f'{prefix}-report-{target_name(k)}.junit.xml'
                         for k in sorted(suites.keys())))
     print()
     print(f'all-{prefix}-targets = {all_targets}')
@@ -83,14 +88,17 @@ def emit_prolog(suites, prefix):
     print(f'\t$(MAKE) {prefix}$* MTESTARGS="$(MTESTARGS) --logbase {prefix}-report$*" && ln -f meson-logs/$@ .')
 
 def emit_suite(name, suite, prefix):
+    tgtname = target_name(name)
     deps = ' '.join(sorted(suite.deps))
     print()
-    print(f'.{prefix}-{name}.deps = {deps}')
-    print(f'.ninja-goals.check-build += $(.{prefix}-{name}.deps)')
+    print(f'.{prefix}-{tgtname}.deps = {deps}')
+    print(f'.ninja-goals.check-build += $(.{prefix}-{tgtname}.deps)')
 
     names = ' '.join(sorted(suite.names(name)))
-    targets = f'{prefix}-{name} {prefix}-report-{name}.junit.xml'
-    if not name.endswith('-slow') and not name.endswith('-thorough'):
+    targets = f'{prefix}-{tgtname} {prefix}-report-{tgtname}.junit.xml'
+    if not name.endswith('-slow') and \
+       not name.endswith('-thorough') and \
+       not name.endswith('-optional'):
         targets += f' {prefix} {prefix}-report.junit.xml'
     print(f'ifneq ($(filter {targets}, $(MAKECMDGOALS)),)')
     # for the "base" suite possibly add FOO-slow and FOO-thorough
-- 
2.53.0

[PATCH 08/16] tests: add a meson suite / make target per block I/O tests format

[Daniel P. Berrangé]

Currently each block format is classified as either 'quick', 'slow' or
'thorough' and this controls whether its I/O tests are added to the meson
suites 'block-quick', 'block-slow' or 'block-thorough'.

This suites are exposed the 'check-block' make target, accepting the
optional SPEED variable.

As we add more formats to the 'thorough' group, however, it becomes
increasingly large and time consuming to run. What is needed is a make
target that can exercise all tests for an individual format, regardless
of speed classification.

This makes use of the previous enhancement to mtest2make.py to introduce
new meson suites 'block-$FORMAT-optional', which translate to new top
level make targets 'check-block-$FORMAT'. These new targets always run
all tests and as such do not need the SPEED variable to be set, but are
not triggered by 'make check' or 'make check-block'.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 tests/Makefile.include         | 3 ++-
 tests/qemu-iotests/meson.build | 3 ++-
 2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/tests/Makefile.include b/tests/Makefile.include
index f257288194..a820980bbe 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -14,7 +14,8 @@ check-help:
 	@echo " $(MAKE) check-unit                    Run qobject tests"
 	@echo " $(MAKE) check-qapi-schema             Run QAPI schema tests"
 	@echo " $(MAKE) check-tracetool               Run tracetool generator tests"
-	@echo " $(MAKE) check-block                   Run block tests"
+	@echo " $(MAKE) check-block                   Run block tests (all formats)"
+	@echo " $(MAKE) check-block-FORMAT            Run block tests (only for FORMAT)"
 ifneq ($(filter $(all-check-targets), check-softfloat),)
 	@echo " $(MAKE) check-softfloat               Run FPU emulation tests"
 	@echo " $(MAKE) check-tcg                     Run TCG tests"
diff --git a/tests/qemu-iotests/meson.build b/tests/qemu-iotests/meson.build
index 1a24d801a3..16a5e39476 100644
--- a/tests/qemu-iotests/meson.build
+++ b/tests/qemu-iotests/meson.build
@@ -62,7 +62,8 @@ foreach format, speed: qemu_iotests_formats
     endforeach
   endif
 
-  suites = []
+  # Every format gets put in the format specific suite
+  suites = ['block-' + format + '-optional']
   # Any format tagged quick or slow also gets added to slow
   # otherwise its tagged thorough
   if speed != 'thorough'
-- 
2.53.0

[PATCH 09/16] docs/devel/testing: expand documentation for 'make check-block'

[Daniel P. Berrangé]

Explain in greater detail what 'check-block' will run for each format,
and also document the new format specific targets.

Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 docs/devel/testing/main.rst    | 25 ++++++++++++++++++++++---
 tests/qemu-iotests/meson.build |  2 ++
 2 files changed, 24 insertions(+), 3 deletions(-)

diff --git a/docs/devel/testing/main.rst b/docs/devel/testing/main.rst
index 0662766b5c..dc4f7202a5 100644
--- a/docs/devel/testing/main.rst
+++ b/docs/devel/testing/main.rst
@@ -221,9 +221,28 @@ same commit that alters the generator code.
 check-block
 ~~~~~~~~~~~
 
-``make check-block`` runs a subset of the block layer iotests (the tests that
-are in the "auto" group).
-See the "QEMU iotests" section below for more information.
+There are a variety of ways to exercise the block layer I/O tests
+via make targets.
+
+A default ``make check`` or ``make check-block`` command will exercise
+the ``qcow2`` format, using the tests tagged into the ``auto`` group
+only.
+
+These targets accept the ``SPEED`` variable to augment the set of tests
+to run. A slightly more comprehensive test plan can be run by defining
+``SPEED=slow``, which enables all tests for the ``qcow2`` and ``raw``
+formats. The most comprehensive test plan can be run by defining
+``SPEED=thorough``, which enables all available tests for the formats
+``parallels``, ``qcow2``, ``qed``, ``raw``, ``vdi``, ``vhdx``,
+``vmdk``, and ``vpc``.
+
+Each of formats also has its own dedicated make target, named
+``make check-block-$FORMAT`` which will run all available tests for
+the designated format and does not require the ``SPEED`` variable
+to be set.
+
+See the "QEMU iotests" section below for more information on the
+block I/O test framework that is leveraged by these ``make`` targets.
 
 .. _qemu-iotests:
 
diff --git a/tests/qemu-iotests/meson.build b/tests/qemu-iotests/meson.build
index 16a5e39476..66b09d6b97 100644
--- a/tests/qemu-iotests/meson.build
+++ b/tests/qemu-iotests/meson.build
@@ -10,6 +10,8 @@ endif
 
 qemu_iotests_binaries = [qemu_img, qemu_io, qemu_nbd, qsd]
 qemu_iotests_env = {'PYTHON': python.full_path()}
+# If altering this definition, also update docs/devel/testing/main.rst
+# section on 'check-block' targets to reflect the changes
 qemu_iotests_formats = {
   'qcow2': 'quick',
   'raw': 'slow',
-- 
2.53.0

[PATCH 10/16] tests: add nbd and luks to the I/O test suites

[Daniel P. Berrangé]

This introduces new suites for running I/O tests on NBD and LUKS
drivers, giving new make targets

 * make check-block-luks
 * make check-block-nbd

as well as adding their tests to 'make check-block SPEED=thorough'

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 docs/devel/testing/main.rst    | 4 ++--
 tests/qemu-iotests/meson.build | 4 +++-
 2 files changed, 5 insertions(+), 3 deletions(-)

diff --git a/docs/devel/testing/main.rst b/docs/devel/testing/main.rst
index dc4f7202a5..8067d0a3ed 100644
--- a/docs/devel/testing/main.rst
+++ b/docs/devel/testing/main.rst
@@ -233,8 +233,8 @@ to run. A slightly more comprehensive test plan can be run by defining
 ``SPEED=slow``, which enables all tests for the ``qcow2`` and ``raw``
 formats. The most comprehensive test plan can be run by defining
 ``SPEED=thorough``, which enables all available tests for the formats
-``parallels``, ``qcow2``, ``qed``, ``raw``, ``vdi``, ``vhdx``,
-``vmdk``, and ``vpc``.
+``luks``, ``nbd``, ``parallels``, ``qcow2``, ``qed``, ``raw``, ``vdi``,
+``vhdx``, ``vmdk``, and ``vpc``.
 
 Each of formats also has its own dedicated make target, named
 ``make check-block-$FORMAT`` which will run all available tests for
diff --git a/tests/qemu-iotests/meson.build b/tests/qemu-iotests/meson.build
index 66b09d6b97..744d0b6e88 100644
--- a/tests/qemu-iotests/meson.build
+++ b/tests/qemu-iotests/meson.build
@@ -15,12 +15,14 @@ qemu_iotests_env = {'PYTHON': python.full_path()}
 qemu_iotests_formats = {
   'qcow2': 'quick',
   'raw': 'slow',
+  'luks': 'thorough',
+  'nbd': 'thorough',
   'parallels': 'thorough',
   'qed': 'thorough',
   'vdi': 'thorough',
   'vhdx': 'thorough',
   'vmdk': 'thorough',
-  'vpc': 'thorough'
+  'vpc': 'thorough',
 }
 
 foreach k, v : emulators
-- 
2.53.0

[PATCH 11/16] tests: use 'driver' as collective term for either format or protocol

[Daniel P. Berrangé]

The I/O tests integration previously exclusively tested block formats
and now also covers the NBD protocol. Replace references to 'format'
with 'driver', as a generic term to collectively apply to either a
format or protocol.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 docs/devel/testing/main.rst    | 13 +++++++------
 tests/Makefile.include         |  4 ++--
 tests/qemu-iotests/meson.build |  6 +++---
 3 files changed, 12 insertions(+), 11 deletions(-)

diff --git a/docs/devel/testing/main.rst b/docs/devel/testing/main.rst
index 8067d0a3ed..797111009a 100644
--- a/docs/devel/testing/main.rst
+++ b/docs/devel/testing/main.rst
@@ -222,7 +222,8 @@ check-block
 ~~~~~~~~~~~
 
 There are a variety of ways to exercise the block layer I/O tests
-via make targets.
+via make targets for a selection of formats / protocols (collectively
+referred to as ``drivers`` below).
 
 A default ``make check`` or ``make check-block`` command will exercise
 the ``qcow2`` format, using the tests tagged into the ``auto`` group
@@ -231,14 +232,14 @@ only.
 These targets accept the ``SPEED`` variable to augment the set of tests
 to run. A slightly more comprehensive test plan can be run by defining
 ``SPEED=slow``, which enables all tests for the ``qcow2`` and ``raw``
-formats. The most comprehensive test plan can be run by defining
-``SPEED=thorough``, which enables all available tests for the formats
+drivers. The most comprehensive test plan can be run by defining
+``SPEED=thorough``, which enables all available tests for the drivers
 ``luks``, ``nbd``, ``parallels``, ``qcow2``, ``qed``, ``raw``, ``vdi``,
 ``vhdx``, ``vmdk``, and ``vpc``.
 
-Each of formats also has its own dedicated make target, named
-``make check-block-$FORMAT`` which will run all available tests for
-the designated format and does not require the ``SPEED`` variable
+Each of drivers also has its own dedicated make target, named
+``make check-block-$DRIVER`` which will run all available tests for
+the designated driver and does not require the ``SPEED`` variable
 to be set.
 
 See the "QEMU iotests" section below for more information on the
diff --git a/tests/Makefile.include b/tests/Makefile.include
index a820980bbe..a063a1d60f 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -14,8 +14,8 @@ check-help:
 	@echo " $(MAKE) check-unit                    Run qobject tests"
 	@echo " $(MAKE) check-qapi-schema             Run QAPI schema tests"
 	@echo " $(MAKE) check-tracetool               Run tracetool generator tests"
-	@echo " $(MAKE) check-block                   Run block tests (all formats)"
-	@echo " $(MAKE) check-block-FORMAT            Run block tests (only for FORMAT)"
+	@echo " $(MAKE) check-block                   Run block tests (all formats/protocols)"
+	@echo " $(MAKE) check-block-DRIVER            Run block tests (only for format/protocol 'DRIVER')"
 ifneq ($(filter $(all-check-targets), check-softfloat),)
 	@echo " $(MAKE) check-softfloat               Run FPU emulation tests"
 	@echo " $(MAKE) check-tcg                     Run TCG tests"
diff --git a/tests/qemu-iotests/meson.build b/tests/qemu-iotests/meson.build
index 744d0b6e88..5d6b0b5ed9 100644
--- a/tests/qemu-iotests/meson.build
+++ b/tests/qemu-iotests/meson.build
@@ -12,7 +12,7 @@ qemu_iotests_binaries = [qemu_img, qemu_io, qemu_nbd, qsd]
 qemu_iotests_env = {'PYTHON': python.full_path()}
 # If altering this definition, also update docs/devel/testing/main.rst
 # section on 'check-block' targets to reflect the changes
-qemu_iotests_formats = {
+qemu_iotests_drivers = {
   'qcow2': 'quick',
   'raw': 'slow',
   'luks': 'thorough',
@@ -33,8 +33,8 @@ endforeach
 
 qemu_iotests_check_cmd = files('check')
 
-foreach format, speed: qemu_iotests_formats
-  # Formats tagged 'quick' get the subset of tests in the 'auto'
+foreach format, speed: qemu_iotests_drivers
+  # Drivers tagged 'quick' get the subset of tests in the 'auto'
   # group, run by default with 'make check' / 'make check-block'
   seen = []
   if speed == 'quick'
-- 
2.53.0

[PATCH 12/16] tests: validate dmsetup result in test 128

[Daniel P. Berrangé]

The I/O test 128 uses 'dmsetup create' to create a device, optionally
using sudo to elevate privileges.

This dmsetup command works in GitLab CI, however, the test then fails
with a missing device name:

  1..1
  # running raw 128
  not ok raw 128
  ----------------------------------- stderr -----------------------------------
  --- /builds/berrange/qemu/tests/qemu-iotests/128.out
  +++ /builds/berrange/qemu/build/scratch/raw-file-128/128.out.bad
  @@ -1,5 +1,5 @@
   QA output created by 128

   == reading from error device ==
  -read failed: Input/output error
  +qemu-io: can't open device /dev/mapper/eiodev16546: Could not open '/dev/mapper/eiodev16546': No such file or directory
   *** done

  (test program exited with status code 1)

It is believed that this is due to the build env using a manually
populated /dev, such that the device mapper node won't ever appear.
It is not a race, since a test adding a sleep did not result in the
device appearing.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 tests/qemu-iotests/128 | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/tests/qemu-iotests/128 b/tests/qemu-iotests/128
index d0e00d24b1..d75b1a451a 100755
--- a/tests/qemu-iotests/128
+++ b/tests/qemu-iotests/128
@@ -42,6 +42,12 @@ _setup_eiodev()
 		echo "0 $((1024 * 1024 * 1024 / 512)) error" | \
 			$cmd dmsetup create "$devname" 2>/dev/null
 		if [ "$?" -eq 0 ]; then
+			DEV="/dev/mapper/$devname"
+			if ! -e $DEV
+			then
+				_notrun "Device $DEV not appearing"
+			fi
+
 			sudo="$cmd"
 			return
 		fi
-- 
2.53.0

[PATCH 13/16] tests: fix check for sudo access in LUKS I/O test

[Daniel P. Berrangé]

The test did not cope with the possibility that 'sudo' was not installed
at all, merely that it was not configured. This broke tests in any CI
env which lacks 'sudo'.

Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 tests/qemu-iotests/149 | 13 ++++++++-----
 1 file changed, 8 insertions(+), 5 deletions(-)

diff --git a/tests/qemu-iotests/149 b/tests/qemu-iotests/149
index c13343d7ef..6dff39a28a 100755
--- a/tests/qemu-iotests/149
+++ b/tests/qemu-iotests/149
@@ -95,11 +95,14 @@ def verify_passwordless_sudo():
 
     args = ["sudo", "-n", "/bin/true"]
 
-    proc = subprocess.Popen(args,
-                            stdin=subprocess.PIPE,
-                            stdout=subprocess.PIPE,
-                            stderr=subprocess.STDOUT,
-                            universal_newlines=True)
+    try:
+        proc = subprocess.Popen(args,
+                                stdin=subprocess.PIPE,
+                                stdout=subprocess.PIPE,
+                                stderr=subprocess.STDOUT,
+                                universal_newlines=True)
+    except FileNotFoundError as e:
+        iotests.notrun('requires sudo binary: %s' % e)
 
     msg = proc.communicate()[0]
 
-- 
2.53.0

[PATCH 14/16] tests: add QEMU_TEST_IO_SKIP for skipping I/O tests

[Daniel P. Berrangé]

The nature of block I/O tests is such that there can be unexpected false
positive failures in certain scenarios that have not been encountered
before, and sometimes non-deterministic failures that are hard to
reproduce.

Before enabling the I/O tests as gating jobs in CI, there needs to be a
mechanism to dynamically mark tests as skipped, without having to commit
code changes.

This introduces the QEMU_TEST_IO_SKIP environment variable that is set
to a list of FORMAT-OR-PROTOCOL:NAME pairs. The intent is that this
variable can be set as a GitLab CI pipeline variable to temporarily
disable a test while problems are being debugged.

Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 docs/devel/testing/main.rst      |  7 +++++++
 tests/qemu-iotests/testrunner.py | 16 ++++++++++++++++
 2 files changed, 23 insertions(+)

diff --git a/docs/devel/testing/main.rst b/docs/devel/testing/main.rst
index 797111009a..f779a64415 100644
--- a/docs/devel/testing/main.rst
+++ b/docs/devel/testing/main.rst
@@ -284,6 +284,13 @@ that are specific to certain cache mode.
 More options are supported by the ``./check`` script, run ``./check -h`` for
 help.
 
+If a test program is known to be broken, it can be disabled by setting
+the ``QEMU_TEST_IO_SKIP`` environment variable with a list of tests to
+be skipped. The values are of the form FORMAT-OR-PROTOCOL:NAME, the
+leading component can be omitted to skip the test for all formats and
+protocols. For example ``export QEMU_TEST_IO_SKIP="luks:149 185 iov-padding``
+will skip ``149`` for LUKS only, and ``185`` and ``iov-padding`` for all.
+
 Writing a new test case
 ~~~~~~~~~~~~~~~~~~~~~~~
 
diff --git a/tests/qemu-iotests/testrunner.py b/tests/qemu-iotests/testrunner.py
index dbe2dddc32..ecb5d4529f 100644
--- a/tests/qemu-iotests/testrunner.py
+++ b/tests/qemu-iotests/testrunner.py
@@ -145,6 +145,18 @@ def __init__(self, env: TestEnv, tap: bool = False,
 
         self._stack: contextlib.ExitStack
 
+        self.skip = {}
+        for rule in os.environ.get("QEMU_TEST_IO_SKIP", "").split(" "):
+            rule = rule.strip()
+            if rule == "":
+                continue
+            if ":" in rule:
+                fmt, name = rule.split(":")
+                if fmt in ("", env.imgfmt, env.imgproto):
+                    self.skip[name] = True
+            else:
+                self.skip[rule] = True
+
     def __enter__(self) -> 'TestRunner':
         self._stack = contextlib.ExitStack()
         self._stack.enter_context(self.env)
@@ -251,6 +263,10 @@ def do_run_test(self, test: str) -> TestResult:
                               description='No qualified output '
                                           f'(expected {f_reference})')
 
+        if f_test.name in self.skip:
+            return TestResult(status='not run',
+                              description='Listed in QEMU_TEST_IO_SKIP')
+
         args = [str(f_test.resolve())]
         env = self.env.prepare_subprocess(args)
 
-- 
2.53.0

[PATCH 15/16] gitlab: add jobs for thorough block tests

[Daniel P. Berrangé]

CI is only exercising the qcow2 'auto' tests currently. As a result we
get no exposure of changes which cause regressions in other block format
drivers.

This adds new CI jobs for each block format, that will run the target
'make check-block-$FORMAT'. The jobs are separate so that we have the
ability to make each format gating or not, depending on their level
of reliability.

The 'centos' image is used to run the I/O tests since several tests
have an implicit dependency on x86_64-softmmu, and thus break with
other architecture targets. The 'centos' build job is the only one
that creates the x86_64-softmmu target in CI. Ideally this target
portability in I/O tests would be fixed to avoid this limitation.

There is currently an undiagnosed failure of job 185 with the qcow2
when run in CI that is marked to be skipped, reported at:

  https://gitlab.com/qemu-project/qemu/-/issues/3270

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 .gitlab-ci.d/buildtest.yml | 98 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 98 insertions(+)

diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
index 4b1949a3a5..d4591d30e9 100644
--- a/.gitlab-ci.d/buildtest.yml
+++ b/.gitlab-ci.d/buildtest.yml
@@ -174,6 +174,104 @@ build-system-centos:
       x86_64-softmmu rx-softmmu sh4-softmmu
     MAKE_CHECK_ARGS: check-build
 
+
+# NB: block-XXX jobs use 'centos' since that is the build
+# job that provides the x86_64-softmmu.  Some I/O tests
+# are currently buggy and blindly assume characteristics
+# of x86 (such as PCIe) causing failures with other arches
+
+block-luks:
+  extends: .native_test_job_template
+  needs:
+    - job: build-system-centos
+      artifacts: true
+  variables:
+    IMAGE: centos9
+    MAKE_CHECK_ARGS: check-block-luks
+
+block-nbd:
+  extends: .native_test_job_template
+  needs:
+    - job: build-system-centos
+      artifacts: true
+  variables:
+    IMAGE: centos9
+    MAKE_CHECK_ARGS: check-block-nbd
+
+block-parallels:
+  extends: .native_test_job_template
+  needs:
+    - job: build-system-centos
+      artifacts: true
+  variables:
+    IMAGE: centos9
+    MAKE_CHECK_ARGS: check-block-parallels
+
+block-qcow2:
+  extends: .native_test_job_template
+  needs:
+    - job: build-system-centos
+      artifacts: true
+  variables:
+    IMAGE: centos9
+    MAKE_CHECK_ARGS: check-block-qcow2
+    # https://gitlab.com/qemu-project/qemu/-/issues/3270
+    QEMU_TEST_IO_SKIP: qcow2:185
+
+block-qed:
+  extends: .native_test_job_template
+  needs:
+    - job: build-system-centos
+      artifacts: true
+  variables:
+    IMAGE: centos9
+    MAKE_CHECK_ARGS: check-block-qed
+
+block-raw:
+  extends: .native_test_job_template
+  needs:
+    - job: build-system-centos
+      artifacts: true
+  variables:
+    IMAGE: centos9
+    MAKE_CHECK_ARGS: check-block-raw
+
+block-vdi:
+  extends: .native_test_job_template
+  needs:
+    - job: build-system-centos
+      artifacts: true
+  variables:
+    IMAGE: centos9
+    MAKE_CHECK_ARGS: check-block-vdi
+
+block-vhdx:
+  extends: .native_test_job_template
+  needs:
+    - job: build-system-centos
+      artifacts: true
+  variables:
+    IMAGE: centos9
+    MAKE_CHECK_ARGS: check-block-vhdx
+
+block-vmdk:
+  extends: .native_test_job_template
+  needs:
+    - job: build-system-centos
+      artifacts: true
+  variables:
+    IMAGE: centos9
+    MAKE_CHECK_ARGS: check-block-vmdk
+
+block-vpc:
+  extends: .native_test_job_template
+  needs:
+    - job: build-system-centos
+      artifacts: true
+  variables:
+    IMAGE: centos9
+    MAKE_CHECK_ARGS: check-block-vpc
+
 # Previous QEMU release. Used for cross-version migration tests.
 build-previous-qemu:
   extends: .native_build_job_template
-- 
2.53.0

[PATCH 16/16] gitlab: remove I/O tests from build-tcg-disabled job

[Daniel P. Berrangé]

Now that we have dedicated CI jobs for running I/O tests on each
supported format/protocol, we no longer need to special case a
run of a hand picked set of tests in the build-tcg-disabled job.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 .gitlab-ci.d/buildtest.yml | 9 ---------
 1 file changed, 9 deletions(-)

diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
index d4591d30e9..710dfbcc77 100644
--- a/.gitlab-ci.d/buildtest.yml
+++ b/.gitlab-ci.d/buildtest.yml
@@ -446,15 +446,6 @@ build-tcg-disabled:
     - make -j"$JOBS"
     - make check-unit
     - make check-qapi-schema
-    - ./run tests/qemu-iotests/check -raw 001 002 003 004 005 008 009
-            010 011 012 021 025 032 033 048 052 063 077 086 101 104 106
-            113 148 150 151 152 157 159 160 163 170 171 184 192 194 208
-            221 226 227 236 253 277 image-fleecing
-    - ./run tests/qemu-iotests/check -qcow2 028 051 056 057 058 065 068
-            082 085 091 095 096 102 122 124 132 139 142 144 145 151 152
-            155 157 165 194 196 200 202 208 209 216 218 227 234 246 247
-            248 250 254 255 257 258 260 261 262 263 264 270 272 273 277
-            279 image-fleecing
     - make distclean
 
 build-user:
-- 
2.53.0

Re: [PATCH 16/16] gitlab: remove I/O tests from build-tcg-disabled job

[Thomas Huth]

On 24/04/2026 17.42, Daniel P. Berrangé wrote:
> Now that we have dedicated CI jobs for running I/O tests on each
> supported format/protocol, we no longer need to special case a
> run of a hand picked set of tests in the build-tcg-disabled job.
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>   .gitlab-ci.d/buildtest.yml | 9 ---------
>   1 file changed, 9 deletions(-)
> 
> diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
> index d4591d30e9..710dfbcc77 100644
> --- a/.gitlab-ci.d/buildtest.yml
> +++ b/.gitlab-ci.d/buildtest.yml
> @@ -446,15 +446,6 @@ build-tcg-disabled:
>       - make -j"$JOBS"
>       - make check-unit
>       - make check-qapi-schema
> -    - ./run tests/qemu-iotests/check -raw 001 002 003 004 005 008 009
> -            010 011 012 021 025 032 033 048 052 063 077 086 101 104 106
> -            113 148 150 151 152 157 159 160 163 170 171 184 192 194 208
> -            221 226 227 236 253 277 image-fleecing
> -    - ./run tests/qemu-iotests/check -qcow2 028 051 056 057 058 065 068
> -            082 085 091 095 096 102 122 124 132 139 142 144 145 151 152
> -            155 157 165 194 196 200 202 208 209 216 218 227 234 246 247
> -            248 250 254 255 257 258 260 261 262 263 264 270 272 273 277
> -            279 image-fleecing
>       - make distclean
>   
>   build-user:

Reviewed-by: Thomas Huth <thuth@redhat.com>

Re: [PATCH 00/16] tests: do more testing of block drivers

[Daniel P. Berrangé]

ping: is there any more feedback on series ?

I'd really like to get the block driver CI coverage improved
to prevent further regressions in currently untested drivers.

On Fri, Apr 24, 2026 at 04:41:48PM +0100, Daniel P. Berrangé wrote:
>  v1: https://lists.nongnu.org/archive/html/qemu-devel/2025-10/msg01650.html
>  v2: https://lists.nongnu.org/archive/html/qemu-devel/2026-01/msg02083.html
>  v3: https://lists.nongnu.org/archive/html/qemu-devel/2026-01/msg02939.html
>  v4: https://lists.nongnu.org/archive/html/qemu-devel/2026-02/msg03245.html
>  v5: https://lists.nongnu.org/archive/html/qemu-devel/2026-03/msg01252.html
> 
> The recent (at the time of v1) set of regressions identified in the LUKS
> block driver re-inforced that despite having a hugely useful set of I/O
> tests, our CI coverage is still letting through too many bugs.
> 
> The core goals of this series were/are:
> 
>  * Add LUKS and NBD to the tested formats/protocols
>    integrated into "make check-block SPEED=thorough"
> 
>  * Ensure that all qcow2 tests can be run with "SPEED=slow",
>    not staying limited to only the 'auto' group used by the
>    'make check-block' target in its default 'quick' mode
> 
>  * Add  'make check-block-$FORMAT' to expose a standalone
>    target for running all tests for a given format (or
>    equivalently a protocol)
> 
>  * Add GitLab CI jobs for exercising tests for all formats
> 
> A sample pipeline for this is
> 
>   https://gitlab.com/berrange/qemu/-/pipelines/2258731804/
> 
> Showing the 'block-FORMAT' jobs which test the 10 block formats/protocols
> currently enabled.
> 
> This series fixes a couple of bugs
> 
>  * Test 128 check for device mapper usage doesn't work
>    sufficiently well to detect gitlab's containe env
>    can't do dynamic devfs
> 
>  * Test 185 is reliably failing on gitlab
> 
>  * Two tests fail when run on CentOS 9 because they trigger
>    python deprecation warnings. This will be fixed when the
>    python-qemu-qmp module is next released.
> 
> Testing of pipelines throughout development of v1-v5 postings does not
> exhibit any non-deterministic failures with the shared runners. Hopefully
> that will apply to QEMU's private runners used forthe staging tree too.
> 
> At this point it is a chicken & egg problem though.
> 
> If we don't bite the bullet and enable block I/O tests in GitLab CI for
> everyone, we'll never find out if they're truely stable enough to rely
> on.
> 
> As a mitigation against instability a patch in this series adds a new
> QEMU_TEST_IO_SKIP env variable that we are able to set as a GitLab CI
> env variable. This lets us skip broken tests dynamically without waiting
> for a git commit to disable them in the buildtest.yml config that we
> use for disabling the 3 known broken tests currently.
> 
> Changed in v6:
> 
>   - Drop redundant change from gitlab yml config
>   - Pull in rebase for qemu.qmp to 0.6.0 to fix two tests
>     by avoiding deprecation warnings
> 
> Changed in v5:
> 
>   - Adapt for python-qemu-qmp being spun off, by temp disabling
>     tests that are broken by it, that we formerly fixed in-tree.
> 
> Changed in v4:
> 
>   - Remove tests from build-tcg-disabled job
> 
> Changed in v3:
> 
>   - Test whether dmsetup create device appeared or not, instead
>     of unconditionally skipping test 128 in GitLab
>   - Use 'driver' term instead of 'format' or 'protocol'
>   - Fix misc typos & rephrasing
> 
> Daniel P. Berrangé (15):
>   gitlab: ensure all meson jobs capture build/meson-logs by default
>   tests: print reason when I/O test is skipped in TAP mode
>   tests: remove redundant meson suite for iotests
>   tests: ensure all qcow2 I/O tests are able to be run via make
>   scripts/mtest2make: ensure output has stable sorting
>   scripts/mtest2make: support optional tests grouping
>   tests: add a meson suite / make target per block I/O tests format
>   docs/devel/testing: expand documentation for 'make check-block'
>   tests: add nbd and luks to the I/O test suites
>   tests: use 'driver' as collective term for either format or protocol
>   tests: validate dmsetup result in test 128
>   tests: fix check for sudo access in LUKS I/O test
>   tests: add QEMU_TEST_IO_SKIP for skipping I/O tests
>   gitlab: add jobs for thorough block tests
>   gitlab: remove I/O tests from build-tcg-disabled job
> 
> John Snow (1):
>   python: bump qemu.qmp to v0.0.6
> 
>  .gitlab-ci.d/buildtest-template.yml           |  30 +++--
>  .gitlab-ci.d/buildtest.yml                    | 107 ++++++++++++++++--
>  docs/devel/testing/main.rst                   |  33 +++++-
>  python/scripts/vendor.py                      |   4 +-
>  ...ny.whl => qemu_qmp-0.0.6-py3-none-any.whl} | Bin 72263 -> 72379 bytes
>  pythondeps.toml                               |   2 +-
>  scripts/mtest2make.py                         |  30 +++--
>  tests/Makefile.include                        |   3 +-
>  tests/qemu-iotests/128                        |   6 +
>  tests/qemu-iotests/149                        |  13 ++-
>  tests/qemu-iotests/meson.build                |  59 ++++++++--
>  tests/qemu-iotests/testrunner.py              |  18 ++-
>  12 files changed, 254 insertions(+), 51 deletions(-)
>  rename python/wheels/{qemu_qmp-0.0.5-py3-none-any.whl => qemu_qmp-0.0.6-py3-none-any.whl} (64%)
> 
> -- 
> 2.53.0
> 

With regards,
Daniel
-- 
|: https://berrange.com       ~~        https://hachyderm.io/@berrange :|
|: https://libvirt.org          ~~          https://entangle-photo.org :|
|: https://pixelfed.art/berrange   ~~    https://fstop138.berrange.com :|

Re: [PATCH 01/16] python: bump qemu.qmp to v0.0.6

[Pierrick Bouvier]

On 4/24/2026 8:41 AM, Daniel P. Berrangé wrote:
> From: John Snow <jsnow@redhat.com>
> 
> This release removes some deprecated warnings for our use of the old
> sendmsg API on older python versions.
> 
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  python/scripts/vendor.py                      |   4 ++--
>  ...ny.whl => qemu_qmp-0.0.6-py3-none-any.whl} | Bin 72263 -> 72379 bytes
>  pythondeps.toml                               |   2 +-
>  3 files changed, 3 insertions(+), 3 deletions(-)
>  rename python/wheels/{qemu_qmp-0.0.5-py3-none-any.whl => qemu_qmp-0.0.6-py3-none-any.whl} (64%)
> 

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com>

Re: [PATCH 02/16] gitlab: ensure all meson jobs capture build/meson-logs by default

[Pierrick Bouvier]

On 4/24/2026 8:41 AM, Daniel P. Berrangé wrote:
> The build-without-defaults and build-tci jobs do not capture any
> artifacts, despite running 'make check'. This has proved a repeated
> bug with CI jobs, so introduce a new '.meson_job_template' rule
> which always captures 'build/meson-logs'. Jobs can still provide
> their own 'artifacts:' config which will override this default
> behaviour.
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>  .gitlab-ci.d/buildtest-template.yml | 30 ++++++++++++++++++-----------
>  1 file changed, 19 insertions(+), 11 deletions(-)
> 

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com>

Re: [PATCH 03/16] tests: print reason when I/O test is skipped in TAP mode

[Pierrick Bouvier]

On 4/24/2026 8:41 AM, Daniel P. Berrangé wrote:
> The TAP output on a skipped test:
> 
>   ok raw 181 # SKIP
> 
> is not informative.
> 
> The test program included a reason, and that should be displayed
> in TAP mode (it is already shown in non-TAP mode):
> 
>   ok raw 181 # SKIP Postcopy is not supported
> 
> Reviewed-by: Thomas Huth <thuth@redhat.com>
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>  tests/qemu-iotests/testrunner.py | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 

Good idea,
Reviewed-by: Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com>

Re: [PATCH 04/16] tests: remove redundant meson suite for iotests

[Pierrick Bouvier]

On 4/24/2026 8:41 AM, Daniel P. Berrangé wrote:
> If a test is in the 'block-slow' or 'block-thorough' suite, there is no
> need to also add it to the 'slow' or 'thorough' suites.
> 
> Tested-by: Thomas Huth <thuth@redhat.com>
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>  tests/qemu-iotests/meson.build | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com>

Re: [PATCH 07/16] scripts/mtest2make: support optional tests grouping

[Pierrick Bouvier]

On 4/24/2026 8:41 AM, Daniel P. Berrangé wrote:
> Currently tests can be classified into three speed groups depending on
> whether the meson suite name ends in '-slow' or '-thorough' or neither.
> 
> This gets turned into make targets that match the name of the meson
> suite, with the speed suffix stripped. e.g.
> 
>  * suite=block ->  'make check-block'
>  * suite=block-slow -> 'make check-block SPEED=slow'
>  * suite=block-thorough -> 'make check-block SPEED=thorough'
> 
> The set of tests under the "thorough" speed, however, can get rather
> large and it would be useful to have a way to expose further make
> targets for directly running a particular subset of tests.
> 
> This needs a way to run a target without requiring the SPEED variable,
> while also not having them enabled by default as if they were 'quick'
> tests.
> 
> This modifies mtest2make.py to support this idea by allowing for a new
> suffix '-optional' on a suite. When this is present, a correspondingly
> named make target will be created without the '-optional' suffix which
> will never be run automatically.
> 
> This is intended to be combined with use of other suites. For example,
> a single NBD test might be added to two suites, 'block-thorough' and
> 'block-nbd-optional'.
> 
> This would allow running it as part of all the block tests with
> 'make check-block SPEED=thorough', and as part of a standalone target
> 'make check-block-nbd'.
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>  scripts/mtest2make.py | 22 +++++++++++++++-------
>  1 file changed, 15 insertions(+), 7 deletions(-)
> 
> diff --git a/scripts/mtest2make.py b/scripts/mtest2make.py
> index 915f02d600..383ea68b16 100644
> --- a/scripts/mtest2make.py
> +++ b/scripts/mtest2make.py
> @@ -22,7 +22,7 @@ def names(self, base):
>  print(r'''
>  SPEED = quick
>  
> -.speed.quick = $(sort $(filter-out %-slow %-thorough, $1))
> +.speed.quick = $(sort $(filter-out %-slow %-thorough %-optional, $1))
>  .speed.slow = $(sort $(filter-out %-thorough, $1))
>  .speed.thorough = $(sort $1)
>  
> @@ -66,10 +66,15 @@ def process_tests(test, targets, suites):
>              s = s[:-9]
>              suites[s].speeds.add('thorough')
>  
> +def target_name(suite):
> +    if suite.endswith('-optional'):
> +        return suite[0:-9]
> +    return suite
> +
>  def emit_prolog(suites, prefix):
> -    all_targets = ' '.join((f'{prefix}-{k}'
> +    all_targets = ' '.join((f'{prefix}-{target_name(k)}'
>                              for k in sorted(suites.keys())))
> -    all_xml = ' '.join((f'{prefix}-report-{k}.junit.xml'
> +    all_xml = ' '.join((f'{prefix}-report-{target_name(k)}.junit.xml'
>                          for k in sorted(suites.keys())))
>      print()
>      print(f'all-{prefix}-targets = {all_targets}')
> @@ -83,14 +88,17 @@ def emit_prolog(suites, prefix):
>      print(f'\t$(MAKE) {prefix}$* MTESTARGS="$(MTESTARGS) --logbase {prefix}-report$*" && ln -f meson-logs/$@ .')
>  
>  def emit_suite(name, suite, prefix):
> +    tgtname = target_name(name)
>      deps = ' '.join(sorted(suite.deps))
>      print()
> -    print(f'.{prefix}-{name}.deps = {deps}')
> -    print(f'.ninja-goals.check-build += $(.{prefix}-{name}.deps)')
> +    print(f'.{prefix}-{tgtname}.deps = {deps}')
> +    print(f'.ninja-goals.check-build += $(.{prefix}-{tgtname}.deps)')
>  
>      names = ' '.join(sorted(suite.names(name)))
> -    targets = f'{prefix}-{name} {prefix}-report-{name}.junit.xml'
> -    if not name.endswith('-slow') and not name.endswith('-thorough'):
> +    targets = f'{prefix}-{tgtname} {prefix}-report-{tgtname}.junit.xml'
> +    if not name.endswith('-slow') and \
> +       not name.endswith('-thorough') and \
> +       not name.endswith('-optional'):
>          targets += f' {prefix} {prefix}-report.junit.xml'
>      print(f'ifneq ($(filter {targets}, $(MAKECMDGOALS)),)')
>      # for the "base" suite possibly add FOO-slow and FOO-thorough

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com>

Not blocking this patch, thus reviewed, but I think it would be a better
abstraction to use meson suites for this (a test can be added in
multiple suites), instead of adding more clunk to makefile layer.

Regards,
Pierrick

Re: [PATCH 08/16] tests: add a meson suite / make target per block I/O tests format

[Pierrick Bouvier]

On 4/24/2026 8:41 AM, Daniel P. Berrangé wrote:
> Currently each block format is classified as either 'quick', 'slow' or
> 'thorough' and this controls whether its I/O tests are added to the meson
> suites 'block-quick', 'block-slow' or 'block-thorough'.
> 
> This suites are exposed the 'check-block' make target, accepting the
> optional SPEED variable.
> 
> As we add more formats to the 'thorough' group, however, it becomes
> increasingly large and time consuming to run. What is needed is a make
> target that can exercise all tests for an individual format, regardless
> of speed classification.
> 
> This makes use of the previous enhancement to mtest2make.py to introduce
> new meson suites 'block-$FORMAT-optional', which translate to new top
> level make targets 'check-block-$FORMAT'. These new targets always run
> all tests and as such do not need the SPEED variable to be set, but are
> not triggered by 'make check' or 'make check-block'.
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>  tests/Makefile.include         | 3 ++-
>  tests/qemu-iotests/meson.build | 3 ++-
>  2 files changed, 4 insertions(+), 2 deletions(-)
> 

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com>

Re: [PATCH 09/16] docs/devel/testing: expand documentation for 'make check-block'

[Pierrick Bouvier]

On 4/24/2026 8:41 AM, Daniel P. Berrangé wrote:
> Explain in greater detail what 'check-block' will run for each format,
> and also document the new format specific targets.
> 
> Reviewed-by: Thomas Huth <thuth@redhat.com>
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>  docs/devel/testing/main.rst    | 25 ++++++++++++++++++++++---
>  tests/qemu-iotests/meson.build |  2 ++
>  2 files changed, 24 insertions(+), 3 deletions(-)
> 

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com>

Re: [PATCH 10/16] tests: add nbd and luks to the I/O test suites

[Pierrick Bouvier]

On 4/24/2026 8:41 AM, Daniel P. Berrangé wrote:
> This introduces new suites for running I/O tests on NBD and LUKS
> drivers, giving new make targets
> 
>  * make check-block-luks
>  * make check-block-nbd
> 
> as well as adding their tests to 'make check-block SPEED=thorough'
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>  docs/devel/testing/main.rst    | 4 ++--
>  tests/qemu-iotests/meson.build | 4 +++-
>  2 files changed, 5 insertions(+), 3 deletions(-)
> 

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com>

Re: [PATCH 16/16] gitlab: remove I/O tests from build-tcg-disabled job

[Pierrick Bouvier]

On 4/24/2026 8:42 AM, Daniel P. Berrangé wrote:
> Now that we have dedicated CI jobs for running I/O tests on each
> supported format/protocol, we no longer need to special case a
> run of a hand picked set of tests in the build-tcg-disabled job.
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>  .gitlab-ci.d/buildtest.yml | 9 ---------
>  1 file changed, 9 deletions(-)
> 

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com>

Re: [PATCH 11/16] tests: use 'driver' as collective term for either format or protocol

[Pierrick Bouvier]

On 4/24/2026 8:41 AM, Daniel P. Berrangé wrote:
> The I/O tests integration previously exclusively tested block formats
> and now also covers the NBD protocol. Replace references to 'format'
> with 'driver', as a generic term to collectively apply to either a
> format or protocol.
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>  docs/devel/testing/main.rst    | 13 +++++++------
>  tests/Makefile.include         |  4 ++--
>  tests/qemu-iotests/meson.build |  6 +++---
>  3 files changed, 12 insertions(+), 11 deletions(-)
> 
> diff --git a/docs/devel/testing/main.rst b/docs/devel/testing/main.rst
> index 8067d0a3ed..797111009a 100644
> --- a/docs/devel/testing/main.rst
> +++ b/docs/devel/testing/main.rst
> @@ -222,7 +222,8 @@ check-block
>  ~~~~~~~~~~~
>  
>  There are a variety of ways to exercise the block layer I/O tests
> -via make targets.
> +via make targets for a selection of formats / protocols (collectively
> +referred to as ``drivers`` below).
>  
>  A default ``make check`` or ``make check-block`` command will exercise
>  the ``qcow2`` format, using the tests tagged into the ``auto`` group
> @@ -231,14 +232,14 @@ only.
>  These targets accept the ``SPEED`` variable to augment the set of tests
>  to run. A slightly more comprehensive test plan can be run by defining
>  ``SPEED=slow``, which enables all tests for the ``qcow2`` and ``raw``
> -formats. The most comprehensive test plan can be run by defining
> -``SPEED=thorough``, which enables all available tests for the formats
> +drivers. The most comprehensive test plan can be run by defining
> +``SPEED=thorough``, which enables all available tests for the drivers
>  ``luks``, ``nbd``, ``parallels``, ``qcow2``, ``qed``, ``raw``, ``vdi``,
>  ``vhdx``, ``vmdk``, and ``vpc``.
>  
> -Each of formats also has its own dedicated make target, named
> -``make check-block-$FORMAT`` which will run all available tests for
> -the designated format and does not require the ``SPEED`` variable
> +Each of drivers also has its own dedicated make target, named
> +``make check-block-$DRIVER`` which will run all available tests for
> +the designated driver and does not require the ``SPEED`` variable
>  to be set.
>  
>  See the "QEMU iotests" section below for more information on the
> diff --git a/tests/Makefile.include b/tests/Makefile.include
> index a820980bbe..a063a1d60f 100644
> --- a/tests/Makefile.include
> +++ b/tests/Makefile.include
> @@ -14,8 +14,8 @@ check-help:
>  	@echo " $(MAKE) check-unit                    Run qobject tests"
>  	@echo " $(MAKE) check-qapi-schema             Run QAPI schema tests"
>  	@echo " $(MAKE) check-tracetool               Run tracetool generator tests"
> -	@echo " $(MAKE) check-block                   Run block tests (all formats)"
> -	@echo " $(MAKE) check-block-FORMAT            Run block tests (only for FORMAT)"
> +	@echo " $(MAKE) check-block                   Run block tests (all formats/protocols)"
> +	@echo " $(MAKE) check-block-DRIVER            Run block tests (only for format/protocol 'DRIVER')"

We had two words for this, now there is a third :)
I would suggest to remove format/protocol at all, and assume it's called
a driver.

>  ifneq ($(filter $(all-check-targets), check-softfloat),)
>  	@echo " $(MAKE) check-softfloat               Run FPU emulation tests"
>  	@echo " $(MAKE) check-tcg                     Run TCG tests"
> diff --git a/tests/qemu-iotests/meson.build b/tests/qemu-iotests/meson.build
> index 744d0b6e88..5d6b0b5ed9 100644
> --- a/tests/qemu-iotests/meson.build
> +++ b/tests/qemu-iotests/meson.build
> @@ -12,7 +12,7 @@ qemu_iotests_binaries = [qemu_img, qemu_io, qemu_nbd, qsd]
>  qemu_iotests_env = {'PYTHON': python.full_path()}
>  # If altering this definition, also update docs/devel/testing/main.rst
>  # section on 'check-block' targets to reflect the changes
> -qemu_iotests_formats = {
> +qemu_iotests_drivers = {
>    'qcow2': 'quick',
>    'raw': 'slow',
>    'luks': 'thorough',
> @@ -33,8 +33,8 @@ endforeach
>  
>  qemu_iotests_check_cmd = files('check')
>  
> -foreach format, speed: qemu_iotests_formats
> -  # Formats tagged 'quick' get the subset of tests in the 'auto'
> +foreach format, speed: qemu_iotests_drivers
> +  # Drivers tagged 'quick' get the subset of tests in the 'auto'
>    # group, run by default with 'make check' / 'make check-block'
>    seen = []
>    if speed == 'quick'

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com>

Re: [PATCH 12/16] tests: validate dmsetup result in test 128

[Pierrick Bouvier]

On 4/24/2026 8:42 AM, Daniel P. Berrangé wrote:
> The I/O test 128 uses 'dmsetup create' to create a device, optionally
> using sudo to elevate privileges.
> 
> This dmsetup command works in GitLab CI, however, the test then fails
> with a missing device name:
> 
>   1..1
>   # running raw 128
>   not ok raw 128
>   ----------------------------------- stderr -----------------------------------
>   --- /builds/berrange/qemu/tests/qemu-iotests/128.out
>   +++ /builds/berrange/qemu/build/scratch/raw-file-128/128.out.bad
>   @@ -1,5 +1,5 @@
>    QA output created by 128
> 
>    == reading from error device ==
>   -read failed: Input/output error
>   +qemu-io: can't open device /dev/mapper/eiodev16546: Could not open '/dev/mapper/eiodev16546': No such file or directory
>    *** done
> 
>   (test program exited with status code 1)
> 
> It is believed that this is due to the build env using a manually
> populated /dev, such that the device mapper node won't ever appear.
> It is not a race, since a test adding a sleep did not result in the
> device appearing.
>

Could that be related to the fact it's running inside a container (if it
is). Maybe it's just missing exposing all /dev instead of a limited set
of devices.

> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>  tests/qemu-iotests/128 | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/tests/qemu-iotests/128 b/tests/qemu-iotests/128
> index d0e00d24b1..d75b1a451a 100755
> --- a/tests/qemu-iotests/128
> +++ b/tests/qemu-iotests/128
> @@ -42,6 +42,12 @@ _setup_eiodev()
>  		echo "0 $((1024 * 1024 * 1024 / 512)) error" | \
>  			$cmd dmsetup create "$devname" 2>/dev/null
>  		if [ "$?" -eq 0 ]; then
> +			DEV="/dev/mapper/$devname"
> +			if ! -e $DEV
> +			then
> +				_notrun "Device $DEV not appearing"
> +			fi
> +
>  			sudo="$cmd"
>  			return
>  		fi

Re: [PATCH 13/16] tests: fix check for sudo access in LUKS I/O test

[Pierrick Bouvier]

On 4/24/2026 8:42 AM, Daniel P. Berrangé wrote:
> The test did not cope with the possibility that 'sudo' was not installed
> at all, merely that it was not configured. This broke tests in any CI
> env which lacks 'sudo'.
> 
> Reviewed-by: Thomas Huth <thuth@redhat.com>
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>  tests/qemu-iotests/149 | 13 ++++++++-----
>  1 file changed, 8 insertions(+), 5 deletions(-)
> 

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com>

Re: [PATCH 14/16] tests: add QEMU_TEST_IO_SKIP for skipping I/O tests

[Pierrick Bouvier]

On 4/24/2026 8:42 AM, Daniel P. Berrangé wrote:
> The nature of block I/O tests is such that there can be unexpected false
> positive failures in certain scenarios that have not been encountered
> before, and sometimes non-deterministic failures that are hard to
> reproduce.
> 
> Before enabling the I/O tests as gating jobs in CI, there needs to be a
> mechanism to dynamically mark tests as skipped, without having to commit
> code changes.
> 
> This introduces the QEMU_TEST_IO_SKIP environment variable that is set
> to a list of FORMAT-OR-PROTOCOL:NAME pairs. The intent is that this
> variable can be set as a GitLab CI pipeline variable to temporarily
> disable a test while problems are being debugged.
> 
> Reviewed-by: Thomas Huth <thuth@redhat.com>
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>  docs/devel/testing/main.rst      |  7 +++++++
>  tests/qemu-iotests/testrunner.py | 16 ++++++++++++++++
>  2 files changed, 23 insertions(+)
> 
> diff --git a/docs/devel/testing/main.rst b/docs/devel/testing/main.rst
> index 797111009a..f779a64415 100644
> --- a/docs/devel/testing/main.rst
> +++ b/docs/devel/testing/main.rst
> @@ -284,6 +284,13 @@ that are specific to certain cache mode.
>  More options are supported by the ``./check`` script, run ``./check -h`` for
>  help.
>  
> +If a test program is known to be broken, it can be disabled by setting
> +the ``QEMU_TEST_IO_SKIP`` environment variable with a list of tests to
> +be skipped. The values are of the form FORMAT-OR-PROTOCOL:NAME, the
> +leading component can be omitted to skip the test for all formats and
> +protocols. For example ``export QEMU_TEST_IO_SKIP="luks:149 185 iov-padding``
> +will skip ``149`` for LUKS only, and ``185`` and ``iov-padding`` for all.
> +
>  Writing a new test case
>  ~~~~~~~~~~~~~~~~~~~~~~~
>  
> diff --git a/tests/qemu-iotests/testrunner.py b/tests/qemu-iotests/testrunner.py
> index dbe2dddc32..ecb5d4529f 100644
> --- a/tests/qemu-iotests/testrunner.py
> +++ b/tests/qemu-iotests/testrunner.py
> @@ -145,6 +145,18 @@ def __init__(self, env: TestEnv, tap: bool = False,
>  
>          self._stack: contextlib.ExitStack
>  
> +        self.skip = {}
> +        for rule in os.environ.get("QEMU_TEST_IO_SKIP", "").split(" "):
> +            rule = rule.strip()
> +            if rule == "":
> +                continue
> +            if ":" in rule:
> +                fmt, name = rule.split(":")
> +                if fmt in ("", env.imgfmt, env.imgproto):
> +                    self.skip[name] = True
> +            else:
> +                self.skip[rule] = True
> +
>      def __enter__(self) -> 'TestRunner':
>          self._stack = contextlib.ExitStack()
>          self._stack.enter_context(self.env)
> @@ -251,6 +263,10 @@ def do_run_test(self, test: str) -> TestResult:
>                                description='No qualified output '
>                                            f'(expected {f_reference})')
>  
> +        if f_test.name in self.skip:
> +            return TestResult(status='not run',
> +                              description='Listed in QEMU_TEST_IO_SKIP')
> +
>          args = [str(f_test.resolve())]
>          env = self.env.prepare_subprocess(args)
>  

Why not simply remove the broken tests, and create issues to add them
again in the future?

Once it's green, in theory, code breaking existing tests should not be
merged, right? So what would be the usage of this variable?

Regards,
Pierrick

Re: [PATCH 15/16] gitlab: add jobs for thorough block tests

[Pierrick Bouvier]

On 4/24/2026 8:42 AM, Daniel P. Berrangé wrote:
> CI is only exercising the qcow2 'auto' tests currently. As a result we
> get no exposure of changes which cause regressions in other block format
> drivers.
> 
> This adds new CI jobs for each block format, that will run the target
> 'make check-block-$FORMAT'. The jobs are separate so that we have the
> ability to make each format gating or not, depending on their level
> of reliability.
> 
> The 'centos' image is used to run the I/O tests since several tests
> have an implicit dependency on x86_64-softmmu, and thus break with
> other architecture targets. The 'centos' build job is the only one
> that creates the x86_64-softmmu target in CI. Ideally this target
> portability in I/O tests would be fixed to avoid this limitation.
> 
> There is currently an undiagnosed failure of job 185 with the qcow2
> when run in CI that is marked to be skipped, reported at:
> 
>   https://gitlab.com/qemu-project/qemu/-/issues/3270
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>  .gitlab-ci.d/buildtest.yml | 98 ++++++++++++++++++++++++++++++++++++++
>  1 file changed, 98 insertions(+)
> 
> diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
> index 4b1949a3a5..d4591d30e9 100644
> --- a/.gitlab-ci.d/buildtest.yml
> +++ b/.gitlab-ci.d/buildtest.yml
> @@ -174,6 +174,104 @@ build-system-centos:
>        x86_64-softmmu rx-softmmu sh4-softmmu
>      MAKE_CHECK_ARGS: check-build
>  
> +
> +# NB: block-XXX jobs use 'centos' since that is the build
> +# job that provides the x86_64-softmmu.  Some I/O tests
> +# are currently buggy and blindly assume characteristics
> +# of x86 (such as PCIe) causing failures with other arches
> +
> +block-luks:
> +  extends: .native_test_job_template
> +  needs:
> +    - job: build-system-centos
> +      artifacts: true
> +  variables:
> +    IMAGE: centos9
> +    MAKE_CHECK_ARGS: check-block-luks
> +
> +block-nbd:
> +  extends: .native_test_job_template
> +  needs:
> +    - job: build-system-centos
> +      artifacts: true
> +  variables:
> +    IMAGE: centos9
> +    MAKE_CHECK_ARGS: check-block-nbd
> +
> +block-parallels:
> +  extends: .native_test_job_template
> +  needs:
> +    - job: build-system-centos
> +      artifacts: true
> +  variables:
> +    IMAGE: centos9
> +    MAKE_CHECK_ARGS: check-block-parallels
> +
> +block-qcow2:
> +  extends: .native_test_job_template
> +  needs:
> +    - job: build-system-centos
> +      artifacts: true
> +  variables:
> +    IMAGE: centos9
> +    MAKE_CHECK_ARGS: check-block-qcow2
> +    # https://gitlab.com/qemu-project/qemu/-/issues/3270
> +    QEMU_TEST_IO_SKIP: qcow2:185
> +
> +block-qed:
> +  extends: .native_test_job_template
> +  needs:
> +    - job: build-system-centos
> +      artifacts: true
> +  variables:
> +    IMAGE: centos9
> +    MAKE_CHECK_ARGS: check-block-qed
> +
> +block-raw:
> +  extends: .native_test_job_template
> +  needs:
> +    - job: build-system-centos
> +      artifacts: true
> +  variables:
> +    IMAGE: centos9
> +    MAKE_CHECK_ARGS: check-block-raw
> +
> +block-vdi:
> +  extends: .native_test_job_template
> +  needs:
> +    - job: build-system-centos
> +      artifacts: true
> +  variables:
> +    IMAGE: centos9
> +    MAKE_CHECK_ARGS: check-block-vdi
> +
> +block-vhdx:
> +  extends: .native_test_job_template
> +  needs:
> +    - job: build-system-centos
> +      artifacts: true
> +  variables:
> +    IMAGE: centos9
> +    MAKE_CHECK_ARGS: check-block-vhdx
> +
> +block-vmdk:
> +  extends: .native_test_job_template
> +  needs:
> +    - job: build-system-centos
> +      artifacts: true
> +  variables:
> +    IMAGE: centos9
> +    MAKE_CHECK_ARGS: check-block-vmdk
> +
> +block-vpc:
> +  extends: .native_test_job_template
> +  needs:
> +    - job: build-system-centos
> +      artifacts: true
> +  variables:
> +    IMAGE: centos9
> +    MAKE_CHECK_ARGS: check-block-vpc
> +
>  # Previous QEMU release. Used for cross-version migration tests.
>  build-previous-qemu:
>    extends: .native_build_job_template

You may want to use a gitlab parallel matrix here, since it's basically
a copy paste with driver name changed.
And either you can set QEMU_TEST_IO_SKIP for all of them, or simply
remove the broken test from tree.

Regards,
Pierrick

Re: [PATCH 14/16] tests: add QEMU_TEST_IO_SKIP for skipping I/O tests

[Daniel P. Berrangé]

On Tue, May 12, 2026 at 08:56:54AM -0700, Pierrick Bouvier wrote:
> On 4/24/2026 8:42 AM, Daniel P. Berrangé wrote:
> > The nature of block I/O tests is such that there can be unexpected false
> > positive failures in certain scenarios that have not been encountered
> > before, and sometimes non-deterministic failures that are hard to
> > reproduce.
> > 
> > Before enabling the I/O tests as gating jobs in CI, there needs to be a
> > mechanism to dynamically mark tests as skipped, without having to commit
> > code changes.
> > 
> > This introduces the QEMU_TEST_IO_SKIP environment variable that is set
> > to a list of FORMAT-OR-PROTOCOL:NAME pairs. The intent is that this
> > variable can be set as a GitLab CI pipeline variable to temporarily
> > disable a test while problems are being debugged.
> > 
> > Reviewed-by: Thomas Huth <thuth@redhat.com>
> > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> > ---
> >  docs/devel/testing/main.rst      |  7 +++++++
> >  tests/qemu-iotests/testrunner.py | 16 ++++++++++++++++
> >  2 files changed, 23 insertions(+)
> > 
> > diff --git a/docs/devel/testing/main.rst b/docs/devel/testing/main.rst
> > index 797111009a..f779a64415 100644
> > --- a/docs/devel/testing/main.rst
> > +++ b/docs/devel/testing/main.rst
> > @@ -284,6 +284,13 @@ that are specific to certain cache mode.
> >  More options are supported by the ``./check`` script, run ``./check -h`` for
> >  help.
> >  
> > +If a test program is known to be broken, it can be disabled by setting
> > +the ``QEMU_TEST_IO_SKIP`` environment variable with a list of tests to
> > +be skipped. The values are of the form FORMAT-OR-PROTOCOL:NAME, the
> > +leading component can be omitted to skip the test for all formats and
> > +protocols. For example ``export QEMU_TEST_IO_SKIP="luks:149 185 iov-padding``
> > +will skip ``149`` for LUKS only, and ``185`` and ``iov-padding`` for all.
> > +
> >  Writing a new test case
> >  ~~~~~~~~~~~~~~~~~~~~~~~
> >  
> > diff --git a/tests/qemu-iotests/testrunner.py b/tests/qemu-iotests/testrunner.py
> > index dbe2dddc32..ecb5d4529f 100644
> > --- a/tests/qemu-iotests/testrunner.py
> > +++ b/tests/qemu-iotests/testrunner.py
> > @@ -145,6 +145,18 @@ def __init__(self, env: TestEnv, tap: bool = False,
> >  
> >          self._stack: contextlib.ExitStack
> >  
> > +        self.skip = {}
> > +        for rule in os.environ.get("QEMU_TEST_IO_SKIP", "").split(" "):
> > +            rule = rule.strip()
> > +            if rule == "":
> > +                continue
> > +            if ":" in rule:
> > +                fmt, name = rule.split(":")
> > +                if fmt in ("", env.imgfmt, env.imgproto):
> > +                    self.skip[name] = True
> > +            else:
> > +                self.skip[rule] = True
> > +
> >      def __enter__(self) -> 'TestRunner':
> >          self._stack = contextlib.ExitStack()
> >          self._stack.enter_context(self.env)
> > @@ -251,6 +263,10 @@ def do_run_test(self, test: str) -> TestResult:
> >                                description='No qualified output '
> >                                            f'(expected {f_reference})')
> >  
> > +        if f_test.name in self.skip:
> > +            return TestResult(status='not run',
> > +                              description='Listed in QEMU_TEST_IO_SKIP')
> > +
> >          args = [str(f_test.resolve())]
> >          env = self.env.prepare_subprocess(args)
> >  
> 
> Why not simply remove the broken tests, and create issues to add them
> again in the future?

In theory that's what our policy today is, but in practice it is
too much of a burden on the release co-ordinator, to expect them
to create such a patch themselves, or wait on a subsys maintainer
todo it for them.

They end up just ignoring brokenness in CI which is a bad practice,
and will prevent us ever making CI truely gating or switching to
using MRs for pull requests. This gives us a super-fast way to skip
flaky tests, while the subsystem maintainers figure out the right
permanent answer.

> Once it's green, in theory, code breaking existing tests should not be
> merged, right? So what would be the usage of this variable?

We have had a pretty decent chunk of non-deterministic test failures,
so there is high likelihood we can merge stuff that passes once and
then subsequently fails some subset of the time. This non-determinism
is one of the key reasons that we currently only have a hand picked
selection of block I/O tests run by meson.

While I've tested this series and haven't see any non-determinstic
failures in what I'm proposing to enable thus far, I think there is
still a pretty high chance we'll uncover some more.

With regards,
Daniel
-- 
|: https://berrange.com       ~~        https://hachyderm.io/@berrange :|
|: https://libvirt.org          ~~          https://entangle-photo.org :|
|: https://pixelfed.art/berrange   ~~    https://fstop138.berrange.com :|

Re: [PATCH 14/16] tests: add QEMU_TEST_IO_SKIP for skipping I/O tests

[Pierrick Bouvier]

On 5/12/2026 9:06 AM, Daniel P. Berrangé wrote:
> On Tue, May 12, 2026 at 08:56:54AM -0700, Pierrick Bouvier wrote:
>> On 4/24/2026 8:42 AM, Daniel P. Berrangé wrote:
>>> The nature of block I/O tests is such that there can be unexpected false
>>> positive failures in certain scenarios that have not been encountered
>>> before, and sometimes non-deterministic failures that are hard to
>>> reproduce.
>>>
>>> Before enabling the I/O tests as gating jobs in CI, there needs to be a
>>> mechanism to dynamically mark tests as skipped, without having to commit
>>> code changes.
>>>
>>> This introduces the QEMU_TEST_IO_SKIP environment variable that is set
>>> to a list of FORMAT-OR-PROTOCOL:NAME pairs. The intent is that this
>>> variable can be set as a GitLab CI pipeline variable to temporarily
>>> disable a test while problems are being debugged.
>>>
>>> Reviewed-by: Thomas Huth <thuth@redhat.com>
>>> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
>>> ---
>>>  docs/devel/testing/main.rst      |  7 +++++++
>>>  tests/qemu-iotests/testrunner.py | 16 ++++++++++++++++
>>>  2 files changed, 23 insertions(+)
>>>
>>> diff --git a/docs/devel/testing/main.rst b/docs/devel/testing/main.rst
>>> index 797111009a..f779a64415 100644
>>> --- a/docs/devel/testing/main.rst
>>> +++ b/docs/devel/testing/main.rst
>>> @@ -284,6 +284,13 @@ that are specific to certain cache mode.
>>>  More options are supported by the ``./check`` script, run ``./check -h`` for
>>>  help.
>>>  
>>> +If a test program is known to be broken, it can be disabled by setting
>>> +the ``QEMU_TEST_IO_SKIP`` environment variable with a list of tests to
>>> +be skipped. The values are of the form FORMAT-OR-PROTOCOL:NAME, the
>>> +leading component can be omitted to skip the test for all formats and
>>> +protocols. For example ``export QEMU_TEST_IO_SKIP="luks:149 185 iov-padding``
>>> +will skip ``149`` for LUKS only, and ``185`` and ``iov-padding`` for all.
>>> +
>>>  Writing a new test case
>>>  ~~~~~~~~~~~~~~~~~~~~~~~
>>>  
>>> diff --git a/tests/qemu-iotests/testrunner.py b/tests/qemu-iotests/testrunner.py
>>> index dbe2dddc32..ecb5d4529f 100644
>>> --- a/tests/qemu-iotests/testrunner.py
>>> +++ b/tests/qemu-iotests/testrunner.py
>>> @@ -145,6 +145,18 @@ def __init__(self, env: TestEnv, tap: bool = False,
>>>  
>>>          self._stack: contextlib.ExitStack
>>>  
>>> +        self.skip = {}
>>> +        for rule in os.environ.get("QEMU_TEST_IO_SKIP", "").split(" "):
>>> +            rule = rule.strip()
>>> +            if rule == "":
>>> +                continue
>>> +            if ":" in rule:
>>> +                fmt, name = rule.split(":")
>>> +                if fmt in ("", env.imgfmt, env.imgproto):
>>> +                    self.skip[name] = True
>>> +            else:
>>> +                self.skip[rule] = True
>>> +
>>>      def __enter__(self) -> 'TestRunner':
>>>          self._stack = contextlib.ExitStack()
>>>          self._stack.enter_context(self.env)
>>> @@ -251,6 +263,10 @@ def do_run_test(self, test: str) -> TestResult:
>>>                                description='No qualified output '
>>>                                            f'(expected {f_reference})')
>>>  
>>> +        if f_test.name in self.skip:
>>> +            return TestResult(status='not run',
>>> +                              description='Listed in QEMU_TEST_IO_SKIP')
>>> +
>>>          args = [str(f_test.resolve())]
>>>          env = self.env.prepare_subprocess(args)
>>>  
>>
>> Why not simply remove the broken tests, and create issues to add them
>> again in the future?
> 
> In theory that's what our policy today is, but in practice it is
> too much of a burden on the release co-ordinator, to expect them
> to create such a patch themselves, or wait on a subsys maintainer
> todo it for them.
> 
> They end up just ignoring brokenness in CI which is a bad practice,
> and will prevent us ever making CI truely gating or switching to
> using MRs for pull requests. This gives us a super-fast way to skip
> flaky tests, while the subsystem maintainers figure out the right
> permanent answer.
>

I disagree on this one, merging a single patch doing a git rm, and a git
revert later is not more expensive than merging a variable modifying a
variable in a yaml file.

The issue with this approach is that people running tests locally will
not see which tests are skipped, and will see false positives. So you
just keep CI green, but not the test base itself.

The risk I see is that some tests will stay forever in this skip
variable, so it will be dead code for CI, but still alive and failing
for people running tests manually who hit the regression.

If you still want an alternative to removing test, implementing a
skip_list in tests/qemu-iotests/meson.build is better than an env var
IMHO, and achieves the exact same effect, for CI and for users.

What do you think?

>> Once it's green, in theory, code breaking existing tests should not be
>> merged, right? So what would be the usage of this variable?
> 
> We have had a pretty decent chunk of non-deterministic test failures,
> so there is high likelihood we can merge stuff that passes once and
> then subsequently fails some subset of the time. This non-determinism
> is one of the key reasons that we currently only have a hand picked
> selection of block I/O tests run by meson.
> 
> While I've tested this series and haven't see any non-determinstic
> failures in what I'm proposing to enable thus far, I think there is
> still a pretty high chance we'll uncover some more.
>

Fair point.

> With regards,
> Daniel

Re: [PATCH 14/16] tests: add QEMU_TEST_IO_SKIP for skipping I/O tests

[Daniel P. Berrangé]

On Tue, May 12, 2026 at 09:19:45AM -0700, Pierrick Bouvier wrote:
> On 5/12/2026 9:06 AM, Daniel P. Berrangé wrote:
> > On Tue, May 12, 2026 at 08:56:54AM -0700, Pierrick Bouvier wrote:
> >> On 4/24/2026 8:42 AM, Daniel P. Berrangé wrote:
> >>> The nature of block I/O tests is such that there can be unexpected false
> >>> positive failures in certain scenarios that have not been encountered
> >>> before, and sometimes non-deterministic failures that are hard to
> >>> reproduce.
> >>>
> >>> Before enabling the I/O tests as gating jobs in CI, there needs to be a
> >>> mechanism to dynamically mark tests as skipped, without having to commit
> >>> code changes.
> >>>
> >>> This introduces the QEMU_TEST_IO_SKIP environment variable that is set
> >>> to a list of FORMAT-OR-PROTOCOL:NAME pairs. The intent is that this
> >>> variable can be set as a GitLab CI pipeline variable to temporarily
> >>> disable a test while problems are being debugged.
> >>>
> >>> Reviewed-by: Thomas Huth <thuth@redhat.com>
> >>> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> >>> ---
> >>>  docs/devel/testing/main.rst      |  7 +++++++
> >>>  tests/qemu-iotests/testrunner.py | 16 ++++++++++++++++
> >>>  2 files changed, 23 insertions(+)
> >>>
> >>> diff --git a/docs/devel/testing/main.rst b/docs/devel/testing/main.rst
> >>> index 797111009a..f779a64415 100644
> >>> --- a/docs/devel/testing/main.rst
> >>> +++ b/docs/devel/testing/main.rst
> >>> @@ -284,6 +284,13 @@ that are specific to certain cache mode.
> >>>  More options are supported by the ``./check`` script, run ``./check -h`` for
> >>>  help.
> >>>  
> >>> +If a test program is known to be broken, it can be disabled by setting
> >>> +the ``QEMU_TEST_IO_SKIP`` environment variable with a list of tests to
> >>> +be skipped. The values are of the form FORMAT-OR-PROTOCOL:NAME, the
> >>> +leading component can be omitted to skip the test for all formats and
> >>> +protocols. For example ``export QEMU_TEST_IO_SKIP="luks:149 185 iov-padding``
> >>> +will skip ``149`` for LUKS only, and ``185`` and ``iov-padding`` for all.
> >>> +
> >>>  Writing a new test case
> >>>  ~~~~~~~~~~~~~~~~~~~~~~~
> >>>  
> >>> diff --git a/tests/qemu-iotests/testrunner.py b/tests/qemu-iotests/testrunner.py
> >>> index dbe2dddc32..ecb5d4529f 100644
> >>> --- a/tests/qemu-iotests/testrunner.py
> >>> +++ b/tests/qemu-iotests/testrunner.py
> >>> @@ -145,6 +145,18 @@ def __init__(self, env: TestEnv, tap: bool = False,
> >>>  
> >>>          self._stack: contextlib.ExitStack
> >>>  
> >>> +        self.skip = {}
> >>> +        for rule in os.environ.get("QEMU_TEST_IO_SKIP", "").split(" "):
> >>> +            rule = rule.strip()
> >>> +            if rule == "":
> >>> +                continue
> >>> +            if ":" in rule:
> >>> +                fmt, name = rule.split(":")
> >>> +                if fmt in ("", env.imgfmt, env.imgproto):
> >>> +                    self.skip[name] = True
> >>> +            else:
> >>> +                self.skip[rule] = True
> >>> +
> >>>      def __enter__(self) -> 'TestRunner':
> >>>          self._stack = contextlib.ExitStack()
> >>>          self._stack.enter_context(self.env)
> >>> @@ -251,6 +263,10 @@ def do_run_test(self, test: str) -> TestResult:
> >>>                                description='No qualified output '
> >>>                                            f'(expected {f_reference})')
> >>>  
> >>> +        if f_test.name in self.skip:
> >>> +            return TestResult(status='not run',
> >>> +                              description='Listed in QEMU_TEST_IO_SKIP')
> >>> +
> >>>          args = [str(f_test.resolve())]
> >>>          env = self.env.prepare_subprocess(args)
> >>>  
> >>
> >> Why not simply remove the broken tests, and create issues to add them
> >> again in the future?
> > 
> > In theory that's what our policy today is, but in practice it is
> > too much of a burden on the release co-ordinator, to expect them
> > to create such a patch themselves, or wait on a subsys maintainer
> > todo it for them.
> > 
> > They end up just ignoring brokenness in CI which is a bad practice,
> > and will prevent us ever making CI truely gating or switching to
> > using MRs for pull requests. This gives us a super-fast way to skip
> > flaky tests, while the subsystem maintainers figure out the right
> > permanent answer.
> >
> 
> I disagree on this one, merging a single patch doing a git rm, and a git
> revert later is not more expensive than merging a variable modifying a
> variable in a yaml file.

Any code changes like that need to be sent back to the subsystem
maintainer to be acked. IMHO the release manager should not be
unilaterally deleting tests without peer review.  So that's
got a non-negligible turn around time, during which CI is broken.

Setting an env variable to skip a problematic test is something
reasonable to do with zero oversight.

> The issue with this approach is that people running tests locally will
> not see which tests are skipped, and will see false positives. So you
> just keep CI green, but not the test base itself.

I would still expect the release manager to file a bug about any
flaky test they disable via the env var, and the subsystem maintainer
should still be fixing it or disabling it such that tests won't fail
more broadly, or deciding to remove it if terminally broken.

We're just decoupling the process so that there is an immediate
workaround possible. It can also be used by people working in
their forks - often I've been testing stuff in my fork, but
see spurious failures because git master has a non-deterministic
test failure merged. I would like to easily skip those in my fork
too, without adding extra commits to me working branches, as that
would require the same commit to be duped into several in-progress
branches, vs setting the env var once.

> The risk I see is that some tests will stay forever in this skip
> variable, so it will be dead code for CI, but still alive and failing
> for people running tests manually who hit the regression.

Again, there should be a bug filed for any flaky test. Anyone can
do this, if they see it locally or in their fork CI, or in staging
CI. If no one can see an obvious fix, then anyone can also propose
to disable the test.

> If you still want an alternative to removing test, implementing a
> skip_list in tests/qemu-iotests/meson.build is better than an env var
> IMHO, and achieves the exact same effect, for CI and for users.
> 
> What do you think?

IMHO there needs to be a way to skip flaky tests which does not
require code changes as the only available option. Code changes
are the permanent fix, env var is the immediate workaround.

> >> Once it's green, in theory, code breaking existing tests should not be
> >> merged, right? So what would be the usage of this variable?
> > 
> > We have had a pretty decent chunk of non-deterministic test failures,
> > so there is high likelihood we can merge stuff that passes once and
> > then subsequently fails some subset of the time. This non-determinism
> > is one of the key reasons that we currently only have a hand picked
> > selection of block I/O tests run by meson.
> > 
> > While I've tested this series and haven't see any non-determinstic
> > failures in what I'm proposing to enable thus far, I think there is
> > still a pretty high chance we'll uncover some more.
> >
> 
> Fair point.

With regards,
Daniel
-- 
|: https://berrange.com       ~~        https://hachyderm.io/@berrange :|
|: https://libvirt.org          ~~          https://entangle-photo.org :|
|: https://pixelfed.art/berrange   ~~    https://fstop138.berrange.com :|

Re: [PATCH 14/16] tests: add QEMU_TEST_IO_SKIP for skipping I/O tests

[Pierrick Bouvier]

On 5/12/2026 9:36 AM, Daniel P. Berrangé wrote:
> On Tue, May 12, 2026 at 09:19:45AM -0700, Pierrick Bouvier wrote:
>> On 5/12/2026 9:06 AM, Daniel P. Berrangé wrote:
>>> On Tue, May 12, 2026 at 08:56:54AM -0700, Pierrick Bouvier wrote:
>>>> On 4/24/2026 8:42 AM, Daniel P. Berrangé wrote:
>>>>> The nature of block I/O tests is such that there can be unexpected false
>>>>> positive failures in certain scenarios that have not been encountered
>>>>> before, and sometimes non-deterministic failures that are hard to
>>>>> reproduce.
>>>>>
>>>>> Before enabling the I/O tests as gating jobs in CI, there needs to be a
>>>>> mechanism to dynamically mark tests as skipped, without having to commit
>>>>> code changes.
>>>>>
>>>>> This introduces the QEMU_TEST_IO_SKIP environment variable that is set
>>>>> to a list of FORMAT-OR-PROTOCOL:NAME pairs. The intent is that this
>>>>> variable can be set as a GitLab CI pipeline variable to temporarily
>>>>> disable a test while problems are being debugged.
>>>>>
>>>>> Reviewed-by: Thomas Huth <thuth@redhat.com>
>>>>> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
>>>>> ---
>>>>>  docs/devel/testing/main.rst      |  7 +++++++
>>>>>  tests/qemu-iotests/testrunner.py | 16 ++++++++++++++++
>>>>>  2 files changed, 23 insertions(+)
>>>>>
>>>>> diff --git a/docs/devel/testing/main.rst b/docs/devel/testing/main.rst
>>>>> index 797111009a..f779a64415 100644
>>>>> --- a/docs/devel/testing/main.rst
>>>>> +++ b/docs/devel/testing/main.rst
>>>>> @@ -284,6 +284,13 @@ that are specific to certain cache mode.
>>>>>  More options are supported by the ``./check`` script, run ``./check -h`` for
>>>>>  help.
>>>>>  
>>>>> +If a test program is known to be broken, it can be disabled by setting
>>>>> +the ``QEMU_TEST_IO_SKIP`` environment variable with a list of tests to
>>>>> +be skipped. The values are of the form FORMAT-OR-PROTOCOL:NAME, the
>>>>> +leading component can be omitted to skip the test for all formats and
>>>>> +protocols. For example ``export QEMU_TEST_IO_SKIP="luks:149 185 iov-padding``
>>>>> +will skip ``149`` for LUKS only, and ``185`` and ``iov-padding`` for all.
>>>>> +
>>>>>  Writing a new test case
>>>>>  ~~~~~~~~~~~~~~~~~~~~~~~
>>>>>  
>>>>> diff --git a/tests/qemu-iotests/testrunner.py b/tests/qemu-iotests/testrunner.py
>>>>> index dbe2dddc32..ecb5d4529f 100644
>>>>> --- a/tests/qemu-iotests/testrunner.py
>>>>> +++ b/tests/qemu-iotests/testrunner.py
>>>>> @@ -145,6 +145,18 @@ def __init__(self, env: TestEnv, tap: bool = False,
>>>>>  
>>>>>          self._stack: contextlib.ExitStack
>>>>>  
>>>>> +        self.skip = {}
>>>>> +        for rule in os.environ.get("QEMU_TEST_IO_SKIP", "").split(" "):
>>>>> +            rule = rule.strip()
>>>>> +            if rule == "":
>>>>> +                continue
>>>>> +            if ":" in rule:
>>>>> +                fmt, name = rule.split(":")
>>>>> +                if fmt in ("", env.imgfmt, env.imgproto):
>>>>> +                    self.skip[name] = True
>>>>> +            else:
>>>>> +                self.skip[rule] = True
>>>>> +
>>>>>      def __enter__(self) -> 'TestRunner':
>>>>>          self._stack = contextlib.ExitStack()
>>>>>          self._stack.enter_context(self.env)
>>>>> @@ -251,6 +263,10 @@ def do_run_test(self, test: str) -> TestResult:
>>>>>                                description='No qualified output '
>>>>>                                            f'(expected {f_reference})')
>>>>>  
>>>>> +        if f_test.name in self.skip:
>>>>> +            return TestResult(status='not run',
>>>>> +                              description='Listed in QEMU_TEST_IO_SKIP')
>>>>> +
>>>>>          args = [str(f_test.resolve())]
>>>>>          env = self.env.prepare_subprocess(args)
>>>>>  
>>>>
>>>> Why not simply remove the broken tests, and create issues to add them
>>>> again in the future?
>>>
>>> In theory that's what our policy today is, but in practice it is
>>> too much of a burden on the release co-ordinator, to expect them
>>> to create such a patch themselves, or wait on a subsys maintainer
>>> todo it for them.
>>>
>>> They end up just ignoring brokenness in CI which is a bad practice,
>>> and will prevent us ever making CI truely gating or switching to
>>> using MRs for pull requests. This gives us a super-fast way to skip
>>> flaky tests, while the subsystem maintainers figure out the right
>>> permanent answer.
>>>
>>
>> I disagree on this one, merging a single patch doing a git rm, and a git
>> revert later is not more expensive than merging a variable modifying a
>> variable in a yaml file.
> 
> Any code changes like that need to be sent back to the subsystem
> maintainer to be acked. IMHO the release manager should not be
> unilaterally deleting tests without peer review.  So that's
> got a non-negligible turn around time, during which CI is broken.
>

I accept the argument, but it seems like a workaround for a human
process, more than a proper solution to the problem.

It would be better to have a proper policy for build/test fixes, instead
of implementing local overrides to this.

> Setting an env variable to skip a problematic test is something
> reasonable to do with zero oversight.
> 
>> The issue with this approach is that people running tests locally will
>> not see which tests are skipped, and will see false positives. So you
>> just keep CI green, but not the test base itself.
> 
> I would still expect the release manager to file a bug about any
> flaky test they disable via the env var, and the subsystem maintainer
> should still be fixing it or disabling it such that tests won't fail
> more broadly, or deciding to remove it if terminally broken.
> 
> We're just decoupling the process so that there is an immediate
> workaround possible. It can also be used by people working in
> their forks - often I've been testing stuff in my fork, but
> see spurious failures because git master has a non-deterministic
> test failure merged. I would like to easily skip those in my fork
> too, without adding extra commits to me working branches, as that
> would require the same commit to be duped into several in-progress
> branches, vs setting the env var once.
> 
>> The risk I see is that some tests will stay forever in this skip
>> variable, so it will be dead code for CI, but still alive and failing
>> for people running tests manually who hit the regression.
> 
> Again, there should be a bug filed for any flaky test. Anyone can
> do this, if they see it locally or in their fork CI, or in staging
> CI. If no one can see an obvious fix, then anyone can also propose
> to disable the test.
> 
>> If you still want an alternative to removing test, implementing a
>> skip_list in tests/qemu-iotests/meson.build is better than an env var
>> IMHO, and achieves the exact same effect, for CI and for users.
>>
>> What do you think?
> 
> IMHO there needs to be a way to skip flaky tests which does not
> require code changes as the only available option. Code changes
> are the permanent fix, env var is the immediate workaround.
>

I'm not sure all this answers to my question about How to ensure users
who run tests and the CI both see the same skip list.

I don't mind having an env var, a black list in meson or any other
solution, but having different results on a dev machine and in CI is not
a good design. So whatever the solution is, the CI yaml file is not the
proper place to store this information.

>>>> Once it's green, in theory, code breaking existing tests should not be
>>>> merged, right? So what would be the usage of this variable?
>>>
>>> We have had a pretty decent chunk of non-deterministic test failures,
>>> so there is high likelihood we can merge stuff that passes once and
>>> then subsequently fails some subset of the time. This non-determinism
>>> is one of the key reasons that we currently only have a hand picked
>>> selection of block I/O tests run by meson.
>>>
>>> While I've tested this series and haven't see any non-determinstic
>>> failures in what I'm proposing to enable thus far, I think there is
>>> still a pretty high chance we'll uncover some more.
>>>
>>
>> Fair point.
> 
> With regards,
> Daniel

Re: [PATCH 14/16] tests: add QEMU_TEST_IO_SKIP for skipping I/O tests

[Daniel P. Berrangé]

On Tue, May 12, 2026 at 09:47:12AM -0700, Pierrick Bouvier wrote:
> On 5/12/2026 9:36 AM, Daniel P. Berrangé wrote:
> > On Tue, May 12, 2026 at 09:19:45AM -0700, Pierrick Bouvier wrote:
> >> On 5/12/2026 9:06 AM, Daniel P. Berrangé wrote:
> >>> On Tue, May 12, 2026 at 08:56:54AM -0700, Pierrick Bouvier wrote:
> >>>> On 4/24/2026 8:42 AM, Daniel P. Berrangé wrote:
> >>>>> The nature of block I/O tests is such that there can be unexpected false
> >>>>> positive failures in certain scenarios that have not been encountered
> >>>>> before, and sometimes non-deterministic failures that are hard to
> >>>>> reproduce.
> >>>>>
> >>>>> Before enabling the I/O tests as gating jobs in CI, there needs to be a
> >>>>> mechanism to dynamically mark tests as skipped, without having to commit
> >>>>> code changes.
> >>>>>
> >>>>> This introduces the QEMU_TEST_IO_SKIP environment variable that is set
> >>>>> to a list of FORMAT-OR-PROTOCOL:NAME pairs. The intent is that this
> >>>>> variable can be set as a GitLab CI pipeline variable to temporarily
> >>>>> disable a test while problems are being debugged.
> >>>>>
> >>>>> Reviewed-by: Thomas Huth <thuth@redhat.com>
> >>>>> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> >>>>> ---
> >>>>>  docs/devel/testing/main.rst      |  7 +++++++
> >>>>>  tests/qemu-iotests/testrunner.py | 16 ++++++++++++++++
> >>>>>  2 files changed, 23 insertions(+)
> >>>>>
> >>>>> diff --git a/docs/devel/testing/main.rst b/docs/devel/testing/main.rst
> >>>>> index 797111009a..f779a64415 100644
> >>>>> --- a/docs/devel/testing/main.rst
> >>>>> +++ b/docs/devel/testing/main.rst
> >>>>> @@ -284,6 +284,13 @@ that are specific to certain cache mode.
> >>>>>  More options are supported by the ``./check`` script, run ``./check -h`` for
> >>>>>  help.
> >>>>>  
> >>>>> +If a test program is known to be broken, it can be disabled by setting
> >>>>> +the ``QEMU_TEST_IO_SKIP`` environment variable with a list of tests to
> >>>>> +be skipped. The values are of the form FORMAT-OR-PROTOCOL:NAME, the
> >>>>> +leading component can be omitted to skip the test for all formats and
> >>>>> +protocols. For example ``export QEMU_TEST_IO_SKIP="luks:149 185 iov-padding``
> >>>>> +will skip ``149`` for LUKS only, and ``185`` and ``iov-padding`` for all.
> >>>>> +
> >>>>>  Writing a new test case
> >>>>>  ~~~~~~~~~~~~~~~~~~~~~~~
> >>>>>  
> >>>>> diff --git a/tests/qemu-iotests/testrunner.py b/tests/qemu-iotests/testrunner.py
> >>>>> index dbe2dddc32..ecb5d4529f 100644
> >>>>> --- a/tests/qemu-iotests/testrunner.py
> >>>>> +++ b/tests/qemu-iotests/testrunner.py
> >>>>> @@ -145,6 +145,18 @@ def __init__(self, env: TestEnv, tap: bool = False,
> >>>>>  
> >>>>>          self._stack: contextlib.ExitStack
> >>>>>  
> >>>>> +        self.skip = {}
> >>>>> +        for rule in os.environ.get("QEMU_TEST_IO_SKIP", "").split(" "):
> >>>>> +            rule = rule.strip()
> >>>>> +            if rule == "":
> >>>>> +                continue
> >>>>> +            if ":" in rule:
> >>>>> +                fmt, name = rule.split(":")
> >>>>> +                if fmt in ("", env.imgfmt, env.imgproto):
> >>>>> +                    self.skip[name] = True
> >>>>> +            else:
> >>>>> +                self.skip[rule] = True
> >>>>> +
> >>>>>      def __enter__(self) -> 'TestRunner':
> >>>>>          self._stack = contextlib.ExitStack()
> >>>>>          self._stack.enter_context(self.env)
> >>>>> @@ -251,6 +263,10 @@ def do_run_test(self, test: str) -> TestResult:
> >>>>>                                description='No qualified output '
> >>>>>                                            f'(expected {f_reference})')
> >>>>>  
> >>>>> +        if f_test.name in self.skip:
> >>>>> +            return TestResult(status='not run',
> >>>>> +                              description='Listed in QEMU_TEST_IO_SKIP')
> >>>>> +
> >>>>>          args = [str(f_test.resolve())]
> >>>>>          env = self.env.prepare_subprocess(args)
> >>>>>  
> >>>>
> >>>> Why not simply remove the broken tests, and create issues to add them
> >>>> again in the future?
> >>>
> >>> In theory that's what our policy today is, but in practice it is
> >>> too much of a burden on the release co-ordinator, to expect them
> >>> to create such a patch themselves, or wait on a subsys maintainer
> >>> todo it for them.
> >>>
> >>> They end up just ignoring brokenness in CI which is a bad practice,
> >>> and will prevent us ever making CI truely gating or switching to
> >>> using MRs for pull requests. This gives us a super-fast way to skip
> >>> flaky tests, while the subsystem maintainers figure out the right
> >>> permanent answer.
> >>>
> >>
> >> I disagree on this one, merging a single patch doing a git rm, and a git
> >> revert later is not more expensive than merging a variable modifying a
> >> variable in a yaml file.
> > 
> > Any code changes like that need to be sent back to the subsystem
> > maintainer to be acked. IMHO the release manager should not be
> > unilaterally deleting tests without peer review.  So that's
> > got a non-negligible turn around time, during which CI is broken.
> >
> 
> I accept the argument, but it seems like a workaround for a human
> process, more than a proper solution to the problem.
> 
> It would be better to have a proper policy for build/test fixes, instead
> of implementing local overrides to this.
> 
> > Setting an env variable to skip a problematic test is something
> > reasonable to do with zero oversight.
> > 
> >> The issue with this approach is that people running tests locally will
> >> not see which tests are skipped, and will see false positives. So you
> >> just keep CI green, but not the test base itself.
> > 
> > I would still expect the release manager to file a bug about any
> > flaky test they disable via the env var, and the subsystem maintainer
> > should still be fixing it or disabling it such that tests won't fail
> > more broadly, or deciding to remove it if terminally broken.
> > 
> > We're just decoupling the process so that there is an immediate
> > workaround possible. It can also be used by people working in
> > their forks - often I've been testing stuff in my fork, but
> > see spurious failures because git master has a non-deterministic
> > test failure merged. I would like to easily skip those in my fork
> > too, without adding extra commits to me working branches, as that
> > would require the same commit to be duped into several in-progress
> > branches, vs setting the env var once.
> > 
> >> The risk I see is that some tests will stay forever in this skip
> >> variable, so it will be dead code for CI, but still alive and failing
> >> for people running tests manually who hit the regression.
> > 
> > Again, there should be a bug filed for any flaky test. Anyone can
> > do this, if they see it locally or in their fork CI, or in staging
> > CI. If no one can see an obvious fix, then anyone can also propose
> > to disable the test.
> > 
> >> If you still want an alternative to removing test, implementing a
> >> skip_list in tests/qemu-iotests/meson.build is better than an env var
> >> IMHO, and achieves the exact same effect, for CI and for users.
> >>
> >> What do you think?
> > 
> > IMHO there needs to be a way to skip flaky tests which does not
> > require code changes as the only available option. Code changes
> > are the permanent fix, env var is the immediate workaround.
> >
> 
> I'm not sure all this answers to my question about How to ensure users
> who run tests and the CI both see the same skip list.
> 
> I don't mind having an env var, a black list in meson or any other
> solution, but having different results on a dev machine and in CI is not
> a good design. So whatever the solution is, the CI yaml file is not the
> proper place to store this information.

AFAICT the test 185 that is being skipped in the CI yaml file only
fails when run under gitlab. I've never seen a failure running it
locally.

If it failed locally too, then I'd agree that it should not be
skipped in the CI yaml, but universally skipped in all scenarios.

With regards,
Daniel
-- 
|: https://berrange.com       ~~        https://hachyderm.io/@berrange :|
|: https://libvirt.org          ~~          https://entangle-photo.org :|
|: https://pixelfed.art/berrange   ~~    https://fstop138.berrange.com :|

Re: [PATCH 14/16] tests: add QEMU_TEST_IO_SKIP for skipping I/O tests

[Pierrick Bouvier]

On 5/12/2026 9:53 AM, Daniel P. Berrangé wrote:
> On Tue, May 12, 2026 at 09:47:12AM -0700, Pierrick Bouvier wrote:
>> On 5/12/2026 9:36 AM, Daniel P. Berrangé wrote:
>>> On Tue, May 12, 2026 at 09:19:45AM -0700, Pierrick Bouvier wrote:
>>>> On 5/12/2026 9:06 AM, Daniel P. Berrangé wrote:
>>>>> On Tue, May 12, 2026 at 08:56:54AM -0700, Pierrick Bouvier wrote:
>>>>>> On 4/24/2026 8:42 AM, Daniel P. Berrangé wrote:
>>>>>>> The nature of block I/O tests is such that there can be unexpected false
>>>>>>> positive failures in certain scenarios that have not been encountered
>>>>>>> before, and sometimes non-deterministic failures that are hard to
>>>>>>> reproduce.
>>>>>>>
>>>>>>> Before enabling the I/O tests as gating jobs in CI, there needs to be a
>>>>>>> mechanism to dynamically mark tests as skipped, without having to commit
>>>>>>> code changes.
>>>>>>>
>>>>>>> This introduces the QEMU_TEST_IO_SKIP environment variable that is set
>>>>>>> to a list of FORMAT-OR-PROTOCOL:NAME pairs. The intent is that this
>>>>>>> variable can be set as a GitLab CI pipeline variable to temporarily
>>>>>>> disable a test while problems are being debugged.
>>>>>>>
>>>>>>> Reviewed-by: Thomas Huth <thuth@redhat.com>
>>>>>>> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
>>>>>>> ---
>>>>>>>  docs/devel/testing/main.rst      |  7 +++++++
>>>>>>>  tests/qemu-iotests/testrunner.py | 16 ++++++++++++++++
>>>>>>>  2 files changed, 23 insertions(+)
>>>>>>>
>>>>>>> diff --git a/docs/devel/testing/main.rst b/docs/devel/testing/main.rst
>>>>>>> index 797111009a..f779a64415 100644
>>>>>>> --- a/docs/devel/testing/main.rst
>>>>>>> +++ b/docs/devel/testing/main.rst
>>>>>>> @@ -284,6 +284,13 @@ that are specific to certain cache mode.
>>>>>>>  More options are supported by the ``./check`` script, run ``./check -h`` for
>>>>>>>  help.
>>>>>>>  
>>>>>>> +If a test program is known to be broken, it can be disabled by setting
>>>>>>> +the ``QEMU_TEST_IO_SKIP`` environment variable with a list of tests to
>>>>>>> +be skipped. The values are of the form FORMAT-OR-PROTOCOL:NAME, the
>>>>>>> +leading component can be omitted to skip the test for all formats and
>>>>>>> +protocols. For example ``export QEMU_TEST_IO_SKIP="luks:149 185 iov-padding``
>>>>>>> +will skip ``149`` for LUKS only, and ``185`` and ``iov-padding`` for all.
>>>>>>> +
>>>>>>>  Writing a new test case
>>>>>>>  ~~~~~~~~~~~~~~~~~~~~~~~
>>>>>>>  
>>>>>>> diff --git a/tests/qemu-iotests/testrunner.py b/tests/qemu-iotests/testrunner.py
>>>>>>> index dbe2dddc32..ecb5d4529f 100644
>>>>>>> --- a/tests/qemu-iotests/testrunner.py
>>>>>>> +++ b/tests/qemu-iotests/testrunner.py
>>>>>>> @@ -145,6 +145,18 @@ def __init__(self, env: TestEnv, tap: bool = False,
>>>>>>>  
>>>>>>>          self._stack: contextlib.ExitStack
>>>>>>>  
>>>>>>> +        self.skip = {}
>>>>>>> +        for rule in os.environ.get("QEMU_TEST_IO_SKIP", "").split(" "):
>>>>>>> +            rule = rule.strip()
>>>>>>> +            if rule == "":
>>>>>>> +                continue
>>>>>>> +            if ":" in rule:
>>>>>>> +                fmt, name = rule.split(":")
>>>>>>> +                if fmt in ("", env.imgfmt, env.imgproto):
>>>>>>> +                    self.skip[name] = True
>>>>>>> +            else:
>>>>>>> +                self.skip[rule] = True
>>>>>>> +
>>>>>>>      def __enter__(self) -> 'TestRunner':
>>>>>>>          self._stack = contextlib.ExitStack()
>>>>>>>          self._stack.enter_context(self.env)
>>>>>>> @@ -251,6 +263,10 @@ def do_run_test(self, test: str) -> TestResult:
>>>>>>>                                description='No qualified output '
>>>>>>>                                            f'(expected {f_reference})')
>>>>>>>  
>>>>>>> +        if f_test.name in self.skip:
>>>>>>> +            return TestResult(status='not run',
>>>>>>> +                              description='Listed in QEMU_TEST_IO_SKIP')
>>>>>>> +
>>>>>>>          args = [str(f_test.resolve())]
>>>>>>>          env = self.env.prepare_subprocess(args)
>>>>>>>  
>>>>>>
>>>>>> Why not simply remove the broken tests, and create issues to add them
>>>>>> again in the future?
>>>>>
>>>>> In theory that's what our policy today is, but in practice it is
>>>>> too much of a burden on the release co-ordinator, to expect them
>>>>> to create such a patch themselves, or wait on a subsys maintainer
>>>>> todo it for them.
>>>>>
>>>>> They end up just ignoring brokenness in CI which is a bad practice,
>>>>> and will prevent us ever making CI truely gating or switching to
>>>>> using MRs for pull requests. This gives us a super-fast way to skip
>>>>> flaky tests, while the subsystem maintainers figure out the right
>>>>> permanent answer.
>>>>>
>>>>
>>>> I disagree on this one, merging a single patch doing a git rm, and a git
>>>> revert later is not more expensive than merging a variable modifying a
>>>> variable in a yaml file.
>>>
>>> Any code changes like that need to be sent back to the subsystem
>>> maintainer to be acked. IMHO the release manager should not be
>>> unilaterally deleting tests without peer review.  So that's
>>> got a non-negligible turn around time, during which CI is broken.
>>>
>>
>> I accept the argument, but it seems like a workaround for a human
>> process, more than a proper solution to the problem.
>>
>> It would be better to have a proper policy for build/test fixes, instead
>> of implementing local overrides to this.
>>
>>> Setting an env variable to skip a problematic test is something
>>> reasonable to do with zero oversight.
>>>
>>>> The issue with this approach is that people running tests locally will
>>>> not see which tests are skipped, and will see false positives. So you
>>>> just keep CI green, but not the test base itself.
>>>
>>> I would still expect the release manager to file a bug about any
>>> flaky test they disable via the env var, and the subsystem maintainer
>>> should still be fixing it or disabling it such that tests won't fail
>>> more broadly, or deciding to remove it if terminally broken.
>>>
>>> We're just decoupling the process so that there is an immediate
>>> workaround possible. It can also be used by people working in
>>> their forks - often I've been testing stuff in my fork, but
>>> see spurious failures because git master has a non-deterministic
>>> test failure merged. I would like to easily skip those in my fork
>>> too, without adding extra commits to me working branches, as that
>>> would require the same commit to be duped into several in-progress
>>> branches, vs setting the env var once.
>>>
>>>> The risk I see is that some tests will stay forever in this skip
>>>> variable, so it will be dead code for CI, but still alive and failing
>>>> for people running tests manually who hit the regression.
>>>
>>> Again, there should be a bug filed for any flaky test. Anyone can
>>> do this, if they see it locally or in their fork CI, or in staging
>>> CI. If no one can see an obvious fix, then anyone can also propose
>>> to disable the test.
>>>
>>>> If you still want an alternative to removing test, implementing a
>>>> skip_list in tests/qemu-iotests/meson.build is better than an env var
>>>> IMHO, and achieves the exact same effect, for CI and for users.
>>>>
>>>> What do you think?
>>>
>>> IMHO there needs to be a way to skip flaky tests which does not
>>> require code changes as the only available option. Code changes
>>> are the permanent fix, env var is the immediate workaround.
>>>
>>
>> I'm not sure all this answers to my question about How to ensure users
>> who run tests and the CI both see the same skip list.
>>
>> I don't mind having an env var, a black list in meson or any other
>> solution, but having different results on a dev machine and in CI is not
>> a good design. So whatever the solution is, the CI yaml file is not the
>> proper place to store this information.
> 
> AFAICT the test 185 that is being skipped in the CI yaml file only
> fails when run under gitlab. I've never seen a failure running it
> locally.
> 
> If it failed locally too, then I'd agree that it should not be
> skipped in the CI yaml, but universally skipped in all scenarios.
>

If I get all this correctly, we add a generic mechanic to be able to
gate CI with block tests just because there is a single test failing
with a single driver. Is that the right approach?

In the future, do we expect to merge code breaking tests?

It really seem there is just one failure, and we won't have more in the
future.
> With regards,
> Daniel

Re: [PATCH 14/16] tests: add QEMU_TEST_IO_SKIP for skipping I/O tests

[Daniel P. Berrangé]

On Tue, May 12, 2026 at 10:09:53AM -0700, Pierrick Bouvier wrote:
> On 5/12/2026 9:53 AM, Daniel P. Berrangé wrote:
> > On Tue, May 12, 2026 at 09:47:12AM -0700, Pierrick Bouvier wrote:
> >> On 5/12/2026 9:36 AM, Daniel P. Berrangé wrote:
> >>> On Tue, May 12, 2026 at 09:19:45AM -0700, Pierrick Bouvier wrote:
> >>>> On 5/12/2026 9:06 AM, Daniel P. Berrangé wrote:
> >>>>> On Tue, May 12, 2026 at 08:56:54AM -0700, Pierrick Bouvier wrote:
> >>>>>> On 4/24/2026 8:42 AM, Daniel P. Berrangé wrote:
> >>>>>>> The nature of block I/O tests is such that there can be unexpected false
> >>>>>>> positive failures in certain scenarios that have not been encountered
> >>>>>>> before, and sometimes non-deterministic failures that are hard to
> >>>>>>> reproduce.
> >>>>>>>
> >>>>>>> Before enabling the I/O tests as gating jobs in CI, there needs to be a
> >>>>>>> mechanism to dynamically mark tests as skipped, without having to commit
> >>>>>>> code changes.
> >>>>>>>
> >>>>>>> This introduces the QEMU_TEST_IO_SKIP environment variable that is set
> >>>>>>> to a list of FORMAT-OR-PROTOCOL:NAME pairs. The intent is that this
> >>>>>>> variable can be set as a GitLab CI pipeline variable to temporarily
> >>>>>>> disable a test while problems are being debugged.
> >>>>>>>
> >>>>>>> Reviewed-by: Thomas Huth <thuth@redhat.com>
> >>>>>>> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> >>>>>>> ---
> >>>>>>>  docs/devel/testing/main.rst      |  7 +++++++
> >>>>>>>  tests/qemu-iotests/testrunner.py | 16 ++++++++++++++++
> >>>>>>>  2 files changed, 23 insertions(+)
> >>>>>>>
> >>>>>>> diff --git a/docs/devel/testing/main.rst b/docs/devel/testing/main.rst
> >>>>>>> index 797111009a..f779a64415 100644
> >>>>>>> --- a/docs/devel/testing/main.rst
> >>>>>>> +++ b/docs/devel/testing/main.rst
> >>>>>>> @@ -284,6 +284,13 @@ that are specific to certain cache mode.
> >>>>>>>  More options are supported by the ``./check`` script, run ``./check -h`` for
> >>>>>>>  help.
> >>>>>>>  
> >>>>>>> +If a test program is known to be broken, it can be disabled by setting
> >>>>>>> +the ``QEMU_TEST_IO_SKIP`` environment variable with a list of tests to
> >>>>>>> +be skipped. The values are of the form FORMAT-OR-PROTOCOL:NAME, the
> >>>>>>> +leading component can be omitted to skip the test for all formats and
> >>>>>>> +protocols. For example ``export QEMU_TEST_IO_SKIP="luks:149 185 iov-padding``
> >>>>>>> +will skip ``149`` for LUKS only, and ``185`` and ``iov-padding`` for all.
> >>>>>>> +
> >>>>>>>  Writing a new test case
> >>>>>>>  ~~~~~~~~~~~~~~~~~~~~~~~
> >>>>>>>  
> >>>>>>> diff --git a/tests/qemu-iotests/testrunner.py b/tests/qemu-iotests/testrunner.py
> >>>>>>> index dbe2dddc32..ecb5d4529f 100644
> >>>>>>> --- a/tests/qemu-iotests/testrunner.py
> >>>>>>> +++ b/tests/qemu-iotests/testrunner.py
> >>>>>>> @@ -145,6 +145,18 @@ def __init__(self, env: TestEnv, tap: bool = False,
> >>>>>>>  
> >>>>>>>          self._stack: contextlib.ExitStack
> >>>>>>>  
> >>>>>>> +        self.skip = {}
> >>>>>>> +        for rule in os.environ.get("QEMU_TEST_IO_SKIP", "").split(" "):
> >>>>>>> +            rule = rule.strip()
> >>>>>>> +            if rule == "":
> >>>>>>> +                continue
> >>>>>>> +            if ":" in rule:
> >>>>>>> +                fmt, name = rule.split(":")
> >>>>>>> +                if fmt in ("", env.imgfmt, env.imgproto):
> >>>>>>> +                    self.skip[name] = True
> >>>>>>> +            else:
> >>>>>>> +                self.skip[rule] = True
> >>>>>>> +
> >>>>>>>      def __enter__(self) -> 'TestRunner':
> >>>>>>>          self._stack = contextlib.ExitStack()
> >>>>>>>          self._stack.enter_context(self.env)
> >>>>>>> @@ -251,6 +263,10 @@ def do_run_test(self, test: str) -> TestResult:
> >>>>>>>                                description='No qualified output '
> >>>>>>>                                            f'(expected {f_reference})')
> >>>>>>>  
> >>>>>>> +        if f_test.name in self.skip:
> >>>>>>> +            return TestResult(status='not run',
> >>>>>>> +                              description='Listed in QEMU_TEST_IO_SKIP')
> >>>>>>> +
> >>>>>>>          args = [str(f_test.resolve())]
> >>>>>>>          env = self.env.prepare_subprocess(args)
> >>>>>>>  
> >>>>>>
> >>>>>> Why not simply remove the broken tests, and create issues to add them
> >>>>>> again in the future?
> >>>>>
> >>>>> In theory that's what our policy today is, but in practice it is
> >>>>> too much of a burden on the release co-ordinator, to expect them
> >>>>> to create such a patch themselves, or wait on a subsys maintainer
> >>>>> todo it for them.
> >>>>>
> >>>>> They end up just ignoring brokenness in CI which is a bad practice,
> >>>>> and will prevent us ever making CI truely gating or switching to
> >>>>> using MRs for pull requests. This gives us a super-fast way to skip
> >>>>> flaky tests, while the subsystem maintainers figure out the right
> >>>>> permanent answer.
> >>>>>
> >>>>
> >>>> I disagree on this one, merging a single patch doing a git rm, and a git
> >>>> revert later is not more expensive than merging a variable modifying a
> >>>> variable in a yaml file.
> >>>
> >>> Any code changes like that need to be sent back to the subsystem
> >>> maintainer to be acked. IMHO the release manager should not be
> >>> unilaterally deleting tests without peer review.  So that's
> >>> got a non-negligible turn around time, during which CI is broken.
> >>>
> >>
> >> I accept the argument, but it seems like a workaround for a human
> >> process, more than a proper solution to the problem.
> >>
> >> It would be better to have a proper policy for build/test fixes, instead
> >> of implementing local overrides to this.
> >>
> >>> Setting an env variable to skip a problematic test is something
> >>> reasonable to do with zero oversight.
> >>>
> >>>> The issue with this approach is that people running tests locally will
> >>>> not see which tests are skipped, and will see false positives. So you
> >>>> just keep CI green, but not the test base itself.
> >>>
> >>> I would still expect the release manager to file a bug about any
> >>> flaky test they disable via the env var, and the subsystem maintainer
> >>> should still be fixing it or disabling it such that tests won't fail
> >>> more broadly, or deciding to remove it if terminally broken.
> >>>
> >>> We're just decoupling the process so that there is an immediate
> >>> workaround possible. It can also be used by people working in
> >>> their forks - often I've been testing stuff in my fork, but
> >>> see spurious failures because git master has a non-deterministic
> >>> test failure merged. I would like to easily skip those in my fork
> >>> too, without adding extra commits to me working branches, as that
> >>> would require the same commit to be duped into several in-progress
> >>> branches, vs setting the env var once.
> >>>
> >>>> The risk I see is that some tests will stay forever in this skip
> >>>> variable, so it will be dead code for CI, but still alive and failing
> >>>> for people running tests manually who hit the regression.
> >>>
> >>> Again, there should be a bug filed for any flaky test. Anyone can
> >>> do this, if they see it locally or in their fork CI, or in staging
> >>> CI. If no one can see an obvious fix, then anyone can also propose
> >>> to disable the test.
> >>>
> >>>> If you still want an alternative to removing test, implementing a
> >>>> skip_list in tests/qemu-iotests/meson.build is better than an env var
> >>>> IMHO, and achieves the exact same effect, for CI and for users.
> >>>>
> >>>> What do you think?
> >>>
> >>> IMHO there needs to be a way to skip flaky tests which does not
> >>> require code changes as the only available option. Code changes
> >>> are the permanent fix, env var is the immediate workaround.
> >>>
> >>
> >> I'm not sure all this answers to my question about How to ensure users
> >> who run tests and the CI both see the same skip list.
> >>
> >> I don't mind having an env var, a black list in meson or any other
> >> solution, but having different results on a dev machine and in CI is not
> >> a good design. So whatever the solution is, the CI yaml file is not the
> >> proper place to store this information.
> > 
> > AFAICT the test 185 that is being skipped in the CI yaml file only
> > fails when run under gitlab. I've never seen a failure running it
> > locally.
> > 
> > If it failed locally too, then I'd agree that it should not be
> > skipped in the CI yaml, but universally skipped in all scenarios.
> >
> 
> If I get all this correctly, we add a generic mechanic to be able to
> gate CI with block tests just because there is a single test failing
> with a single driver. Is that the right approach?

The env variable is the generic mechanism.

The yaml file exclusion for 185 is the special case, but we get
that basically for free with the former.

> In the future, do we expect to merge code breaking tests?

Yes. We will certainly merge more non-deterministic tests. We've seen
this over & over again. Something passes CI initially but after a
number of CI pipelines turns out to be flaky

With regards,
Daniel
-- 
|: https://berrange.com       ~~        https://hachyderm.io/@berrange :|
|: https://libvirt.org          ~~          https://entangle-photo.org :|
|: https://pixelfed.art/berrange   ~~    https://fstop138.berrange.com :|

Re: [PATCH 14/16] tests: add QEMU_TEST_IO_SKIP for skipping I/O tests

[Pierrick Bouvier]

On 5/12/2026 10:24 AM, Daniel P. Berrangé wrote:
> On Tue, May 12, 2026 at 10:09:53AM -0700, Pierrick Bouvier wrote:
>> On 5/12/2026 9:53 AM, Daniel P. Berrangé wrote:
>>> On Tue, May 12, 2026 at 09:47:12AM -0700, Pierrick Bouvier wrote:
>>>> On 5/12/2026 9:36 AM, Daniel P. Berrangé wrote:
>>>>> On Tue, May 12, 2026 at 09:19:45AM -0700, Pierrick Bouvier wrote:
>>>>>> On 5/12/2026 9:06 AM, Daniel P. Berrangé wrote:
>>>>>>> On Tue, May 12, 2026 at 08:56:54AM -0700, Pierrick Bouvier wrote:
>>>>>>>> On 4/24/2026 8:42 AM, Daniel P. Berrangé wrote:
>>>>>>>>> The nature of block I/O tests is such that there can be unexpected false
>>>>>>>>> positive failures in certain scenarios that have not been encountered
>>>>>>>>> before, and sometimes non-deterministic failures that are hard to
>>>>>>>>> reproduce.
>>>>>>>>>
>>>>>>>>> Before enabling the I/O tests as gating jobs in CI, there needs to be a
>>>>>>>>> mechanism to dynamically mark tests as skipped, without having to commit
>>>>>>>>> code changes.
>>>>>>>>>
>>>>>>>>> This introduces the QEMU_TEST_IO_SKIP environment variable that is set
>>>>>>>>> to a list of FORMAT-OR-PROTOCOL:NAME pairs. The intent is that this
>>>>>>>>> variable can be set as a GitLab CI pipeline variable to temporarily
>>>>>>>>> disable a test while problems are being debugged.
>>>>>>>>>
>>>>>>>>> Reviewed-by: Thomas Huth <thuth@redhat.com>
>>>>>>>>> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
>>>>>>>>> ---
>>>>>>>>>  docs/devel/testing/main.rst      |  7 +++++++
>>>>>>>>>  tests/qemu-iotests/testrunner.py | 16 ++++++++++++++++
>>>>>>>>>  2 files changed, 23 insertions(+)
>>>>>>>>>
>>>>>>>>> diff --git a/docs/devel/testing/main.rst b/docs/devel/testing/main.rst
>>>>>>>>> index 797111009a..f779a64415 100644
>>>>>>>>> --- a/docs/devel/testing/main.rst
>>>>>>>>> +++ b/docs/devel/testing/main.rst
>>>>>>>>> @@ -284,6 +284,13 @@ that are specific to certain cache mode.
>>>>>>>>>  More options are supported by the ``./check`` script, run ``./check -h`` for
>>>>>>>>>  help.
>>>>>>>>>  
>>>>>>>>> +If a test program is known to be broken, it can be disabled by setting
>>>>>>>>> +the ``QEMU_TEST_IO_SKIP`` environment variable with a list of tests to
>>>>>>>>> +be skipped. The values are of the form FORMAT-OR-PROTOCOL:NAME, the
>>>>>>>>> +leading component can be omitted to skip the test for all formats and
>>>>>>>>> +protocols. For example ``export QEMU_TEST_IO_SKIP="luks:149 185 iov-padding``
>>>>>>>>> +will skip ``149`` for LUKS only, and ``185`` and ``iov-padding`` for all.
>>>>>>>>> +
>>>>>>>>>  Writing a new test case
>>>>>>>>>  ~~~~~~~~~~~~~~~~~~~~~~~
>>>>>>>>>  
>>>>>>>>> diff --git a/tests/qemu-iotests/testrunner.py b/tests/qemu-iotests/testrunner.py
>>>>>>>>> index dbe2dddc32..ecb5d4529f 100644
>>>>>>>>> --- a/tests/qemu-iotests/testrunner.py
>>>>>>>>> +++ b/tests/qemu-iotests/testrunner.py
>>>>>>>>> @@ -145,6 +145,18 @@ def __init__(self, env: TestEnv, tap: bool = False,
>>>>>>>>>  
>>>>>>>>>          self._stack: contextlib.ExitStack
>>>>>>>>>  
>>>>>>>>> +        self.skip = {}
>>>>>>>>> +        for rule in os.environ.get("QEMU_TEST_IO_SKIP", "").split(" "):
>>>>>>>>> +            rule = rule.strip()
>>>>>>>>> +            if rule == "":
>>>>>>>>> +                continue
>>>>>>>>> +            if ":" in rule:
>>>>>>>>> +                fmt, name = rule.split(":")
>>>>>>>>> +                if fmt in ("", env.imgfmt, env.imgproto):
>>>>>>>>> +                    self.skip[name] = True
>>>>>>>>> +            else:
>>>>>>>>> +                self.skip[rule] = True
>>>>>>>>> +
>>>>>>>>>      def __enter__(self) -> 'TestRunner':
>>>>>>>>>          self._stack = contextlib.ExitStack()
>>>>>>>>>          self._stack.enter_context(self.env)
>>>>>>>>> @@ -251,6 +263,10 @@ def do_run_test(self, test: str) -> TestResult:
>>>>>>>>>                                description='No qualified output '
>>>>>>>>>                                            f'(expected {f_reference})')
>>>>>>>>>  
>>>>>>>>> +        if f_test.name in self.skip:
>>>>>>>>> +            return TestResult(status='not run',
>>>>>>>>> +                              description='Listed in QEMU_TEST_IO_SKIP')
>>>>>>>>> +
>>>>>>>>>          args = [str(f_test.resolve())]
>>>>>>>>>          env = self.env.prepare_subprocess(args)
>>>>>>>>>  
>>>>>>>>
>>>>>>>> Why not simply remove the broken tests, and create issues to add them
>>>>>>>> again in the future?
>>>>>>>
>>>>>>> In theory that's what our policy today is, but in practice it is
>>>>>>> too much of a burden on the release co-ordinator, to expect them
>>>>>>> to create such a patch themselves, or wait on a subsys maintainer
>>>>>>> todo it for them.
>>>>>>>
>>>>>>> They end up just ignoring brokenness in CI which is a bad practice,
>>>>>>> and will prevent us ever making CI truely gating or switching to
>>>>>>> using MRs for pull requests. This gives us a super-fast way to skip
>>>>>>> flaky tests, while the subsystem maintainers figure out the right
>>>>>>> permanent answer.
>>>>>>>
>>>>>>
>>>>>> I disagree on this one, merging a single patch doing a git rm, and a git
>>>>>> revert later is not more expensive than merging a variable modifying a
>>>>>> variable in a yaml file.
>>>>>
>>>>> Any code changes like that need to be sent back to the subsystem
>>>>> maintainer to be acked. IMHO the release manager should not be
>>>>> unilaterally deleting tests without peer review.  So that's
>>>>> got a non-negligible turn around time, during which CI is broken.
>>>>>
>>>>
>>>> I accept the argument, but it seems like a workaround for a human
>>>> process, more than a proper solution to the problem.
>>>>
>>>> It would be better to have a proper policy for build/test fixes, instead
>>>> of implementing local overrides to this.
>>>>
>>>>> Setting an env variable to skip a problematic test is something
>>>>> reasonable to do with zero oversight.
>>>>>
>>>>>> The issue with this approach is that people running tests locally will
>>>>>> not see which tests are skipped, and will see false positives. So you
>>>>>> just keep CI green, but not the test base itself.
>>>>>
>>>>> I would still expect the release manager to file a bug about any
>>>>> flaky test they disable via the env var, and the subsystem maintainer
>>>>> should still be fixing it or disabling it such that tests won't fail
>>>>> more broadly, or deciding to remove it if terminally broken.
>>>>>
>>>>> We're just decoupling the process so that there is an immediate
>>>>> workaround possible. It can also be used by people working in
>>>>> their forks - often I've been testing stuff in my fork, but
>>>>> see spurious failures because git master has a non-deterministic
>>>>> test failure merged. I would like to easily skip those in my fork
>>>>> too, without adding extra commits to me working branches, as that
>>>>> would require the same commit to be duped into several in-progress
>>>>> branches, vs setting the env var once.
>>>>>
>>>>>> The risk I see is that some tests will stay forever in this skip
>>>>>> variable, so it will be dead code for CI, but still alive and failing
>>>>>> for people running tests manually who hit the regression.
>>>>>
>>>>> Again, there should be a bug filed for any flaky test. Anyone can
>>>>> do this, if they see it locally or in their fork CI, or in staging
>>>>> CI. If no one can see an obvious fix, then anyone can also propose
>>>>> to disable the test.
>>>>>
>>>>>> If you still want an alternative to removing test, implementing a
>>>>>> skip_list in tests/qemu-iotests/meson.build is better than an env var
>>>>>> IMHO, and achieves the exact same effect, for CI and for users.
>>>>>>
>>>>>> What do you think?
>>>>>
>>>>> IMHO there needs to be a way to skip flaky tests which does not
>>>>> require code changes as the only available option. Code changes
>>>>> are the permanent fix, env var is the immediate workaround.
>>>>>
>>>>
>>>> I'm not sure all this answers to my question about How to ensure users
>>>> who run tests and the CI both see the same skip list.
>>>>
>>>> I don't mind having an env var, a black list in meson or any other
>>>> solution, but having different results on a dev machine and in CI is not
>>>> a good design. So whatever the solution is, the CI yaml file is not the
>>>> proper place to store this information.
>>>
>>> AFAICT the test 185 that is being skipped in the CI yaml file only
>>> fails when run under gitlab. I've never seen a failure running it
>>> locally.
>>>
>>> If it failed locally too, then I'd agree that it should not be
>>> skipped in the CI yaml, but universally skipped in all scenarios.
>>>
>>
>> If I get all this correctly, we add a generic mechanic to be able to
>> gate CI with block tests just because there is a single test failing
>> with a single driver. Is that the right approach?
> 
> The env variable is the generic mechanism.
> 
> The yaml file exclusion for 185 is the special case, but we get
> that basically for free with the former.
> 
>> In the future, do we expect to merge code breaking tests?
> 
> Yes. We will certainly merge more non-deterministic tests. We've seen
> this over & over again. Something passes CI initially but after a
> number of CI pipelines turns out to be flaky
>

Then we can mark them as flaky in tests/qemu-iotests/meson.build.

It seems like you ignore the point that there is a problem between
setting something in CI only vs making something that works for all
users. I'm not against an env var, I just don't see how it answers this
need.

> With regards,
> Daniel

Re: [PATCH 14/16] tests: add QEMU_TEST_IO_SKIP for skipping I/O tests

[Daniel P. Berrangé]

On Tue, May 12, 2026 at 10:53:05AM -0700, Pierrick Bouvier wrote:
> On 5/12/2026 10:24 AM, Daniel P. Berrangé wrote:
> > On Tue, May 12, 2026 at 10:09:53AM -0700, Pierrick Bouvier wrote:
> >> On 5/12/2026 9:53 AM, Daniel P. Berrangé wrote:
> >>> On Tue, May 12, 2026 at 09:47:12AM -0700, Pierrick Bouvier wrote:
> >>>> On 5/12/2026 9:36 AM, Daniel P. Berrangé wrote:
> >>>>> On Tue, May 12, 2026 at 09:19:45AM -0700, Pierrick Bouvier wrote:
> >>>>>> On 5/12/2026 9:06 AM, Daniel P. Berrangé wrote:
> >>>>>>> On Tue, May 12, 2026 at 08:56:54AM -0700, Pierrick Bouvier wrote:
> >>>>>>>> On 4/24/2026 8:42 AM, Daniel P. Berrangé wrote:
> >>>>>>>>> The nature of block I/O tests is such that there can be unexpected false
> >>>>>>>>> positive failures in certain scenarios that have not been encountered
> >>>>>>>>> before, and sometimes non-deterministic failures that are hard to
> >>>>>>>>> reproduce.
> >>>>>>>>>
> >>>>>>>>> Before enabling the I/O tests as gating jobs in CI, there needs to be a
> >>>>>>>>> mechanism to dynamically mark tests as skipped, without having to commit
> >>>>>>>>> code changes.
> >>>>>>>>>
> >>>>>>>>> This introduces the QEMU_TEST_IO_SKIP environment variable that is set
> >>>>>>>>> to a list of FORMAT-OR-PROTOCOL:NAME pairs. The intent is that this
> >>>>>>>>> variable can be set as a GitLab CI pipeline variable to temporarily
> >>>>>>>>> disable a test while problems are being debugged.
> >>>>>>>>>
> >>>>>>>>> Reviewed-by: Thomas Huth <thuth@redhat.com>
> >>>>>>>>> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> >>>>>>>>> ---
> >>>>>>>>>  docs/devel/testing/main.rst      |  7 +++++++
> >>>>>>>>>  tests/qemu-iotests/testrunner.py | 16 ++++++++++++++++
> >>>>>>>>>  2 files changed, 23 insertions(+)
> >>>>>>>>>
> >>>>>>>>> diff --git a/docs/devel/testing/main.rst b/docs/devel/testing/main.rst
> >>>>>>>>> index 797111009a..f779a64415 100644
> >>>>>>>>> --- a/docs/devel/testing/main.rst
> >>>>>>>>> +++ b/docs/devel/testing/main.rst
> >>>>>>>>> @@ -284,6 +284,13 @@ that are specific to certain cache mode.
> >>>>>>>>>  More options are supported by the ``./check`` script, run ``./check -h`` for
> >>>>>>>>>  help.
> >>>>>>>>>  
> >>>>>>>>> +If a test program is known to be broken, it can be disabled by setting
> >>>>>>>>> +the ``QEMU_TEST_IO_SKIP`` environment variable with a list of tests to
> >>>>>>>>> +be skipped. The values are of the form FORMAT-OR-PROTOCOL:NAME, the
> >>>>>>>>> +leading component can be omitted to skip the test for all formats and
> >>>>>>>>> +protocols. For example ``export QEMU_TEST_IO_SKIP="luks:149 185 iov-padding``
> >>>>>>>>> +will skip ``149`` for LUKS only, and ``185`` and ``iov-padding`` for all.
> >>>>>>>>> +
> >>>>>>>>>  Writing a new test case
> >>>>>>>>>  ~~~~~~~~~~~~~~~~~~~~~~~
> >>>>>>>>>  
> >>>>>>>>> diff --git a/tests/qemu-iotests/testrunner.py b/tests/qemu-iotests/testrunner.py
> >>>>>>>>> index dbe2dddc32..ecb5d4529f 100644
> >>>>>>>>> --- a/tests/qemu-iotests/testrunner.py
> >>>>>>>>> +++ b/tests/qemu-iotests/testrunner.py
> >>>>>>>>> @@ -145,6 +145,18 @@ def __init__(self, env: TestEnv, tap: bool = False,
> >>>>>>>>>  
> >>>>>>>>>          self._stack: contextlib.ExitStack
> >>>>>>>>>  
> >>>>>>>>> +        self.skip = {}
> >>>>>>>>> +        for rule in os.environ.get("QEMU_TEST_IO_SKIP", "").split(" "):
> >>>>>>>>> +            rule = rule.strip()
> >>>>>>>>> +            if rule == "":
> >>>>>>>>> +                continue
> >>>>>>>>> +            if ":" in rule:
> >>>>>>>>> +                fmt, name = rule.split(":")
> >>>>>>>>> +                if fmt in ("", env.imgfmt, env.imgproto):
> >>>>>>>>> +                    self.skip[name] = True
> >>>>>>>>> +            else:
> >>>>>>>>> +                self.skip[rule] = True
> >>>>>>>>> +
> >>>>>>>>>      def __enter__(self) -> 'TestRunner':
> >>>>>>>>>          self._stack = contextlib.ExitStack()
> >>>>>>>>>          self._stack.enter_context(self.env)
> >>>>>>>>> @@ -251,6 +263,10 @@ def do_run_test(self, test: str) -> TestResult:
> >>>>>>>>>                                description='No qualified output '
> >>>>>>>>>                                            f'(expected {f_reference})')
> >>>>>>>>>  
> >>>>>>>>> +        if f_test.name in self.skip:
> >>>>>>>>> +            return TestResult(status='not run',
> >>>>>>>>> +                              description='Listed in QEMU_TEST_IO_SKIP')
> >>>>>>>>> +
> >>>>>>>>>          args = [str(f_test.resolve())]
> >>>>>>>>>          env = self.env.prepare_subprocess(args)
> >>>>>>>>>  
> >>>>>>>>
> >>>>>>>> Why not simply remove the broken tests, and create issues to add them
> >>>>>>>> again in the future?
> >>>>>>>
> >>>>>>> In theory that's what our policy today is, but in practice it is
> >>>>>>> too much of a burden on the release co-ordinator, to expect them
> >>>>>>> to create such a patch themselves, or wait on a subsys maintainer
> >>>>>>> todo it for them.
> >>>>>>>
> >>>>>>> They end up just ignoring brokenness in CI which is a bad practice,
> >>>>>>> and will prevent us ever making CI truely gating or switching to
> >>>>>>> using MRs for pull requests. This gives us a super-fast way to skip
> >>>>>>> flaky tests, while the subsystem maintainers figure out the right
> >>>>>>> permanent answer.
> >>>>>>>
> >>>>>>
> >>>>>> I disagree on this one, merging a single patch doing a git rm, and a git
> >>>>>> revert later is not more expensive than merging a variable modifying a
> >>>>>> variable in a yaml file.
> >>>>>
> >>>>> Any code changes like that need to be sent back to the subsystem
> >>>>> maintainer to be acked. IMHO the release manager should not be
> >>>>> unilaterally deleting tests without peer review.  So that's
> >>>>> got a non-negligible turn around time, during which CI is broken.
> >>>>>
> >>>>
> >>>> I accept the argument, but it seems like a workaround for a human
> >>>> process, more than a proper solution to the problem.
> >>>>
> >>>> It would be better to have a proper policy for build/test fixes, instead
> >>>> of implementing local overrides to this.
> >>>>
> >>>>> Setting an env variable to skip a problematic test is something
> >>>>> reasonable to do with zero oversight.
> >>>>>
> >>>>>> The issue with this approach is that people running tests locally will
> >>>>>> not see which tests are skipped, and will see false positives. So you
> >>>>>> just keep CI green, but not the test base itself.
> >>>>>
> >>>>> I would still expect the release manager to file a bug about any
> >>>>> flaky test they disable via the env var, and the subsystem maintainer
> >>>>> should still be fixing it or disabling it such that tests won't fail
> >>>>> more broadly, or deciding to remove it if terminally broken.
> >>>>>
> >>>>> We're just decoupling the process so that there is an immediate
> >>>>> workaround possible. It can also be used by people working in
> >>>>> their forks - often I've been testing stuff in my fork, but
> >>>>> see spurious failures because git master has a non-deterministic
> >>>>> test failure merged. I would like to easily skip those in my fork
> >>>>> too, without adding extra commits to me working branches, as that
> >>>>> would require the same commit to be duped into several in-progress
> >>>>> branches, vs setting the env var once.
> >>>>>
> >>>>>> The risk I see is that some tests will stay forever in this skip
> >>>>>> variable, so it will be dead code for CI, but still alive and failing
> >>>>>> for people running tests manually who hit the regression.
> >>>>>
> >>>>> Again, there should be a bug filed for any flaky test. Anyone can
> >>>>> do this, if they see it locally or in their fork CI, or in staging
> >>>>> CI. If no one can see an obvious fix, then anyone can also propose
> >>>>> to disable the test.
> >>>>>
> >>>>>> If you still want an alternative to removing test, implementing a
> >>>>>> skip_list in tests/qemu-iotests/meson.build is better than an env var
> >>>>>> IMHO, and achieves the exact same effect, for CI and for users.
> >>>>>>
> >>>>>> What do you think?
> >>>>>
> >>>>> IMHO there needs to be a way to skip flaky tests which does not
> >>>>> require code changes as the only available option. Code changes
> >>>>> are the permanent fix, env var is the immediate workaround.
> >>>>>
> >>>>
> >>>> I'm not sure all this answers to my question about How to ensure users
> >>>> who run tests and the CI both see the same skip list.
> >>>>
> >>>> I don't mind having an env var, a black list in meson or any other
> >>>> solution, but having different results on a dev machine and in CI is not
> >>>> a good design. So whatever the solution is, the CI yaml file is not the
> >>>> proper place to store this information.
> >>>
> >>> AFAICT the test 185 that is being skipped in the CI yaml file only
> >>> fails when run under gitlab. I've never seen a failure running it
> >>> locally.
> >>>
> >>> If it failed locally too, then I'd agree that it should not be
> >>> skipped in the CI yaml, but universally skipped in all scenarios.
> >>>
> >>
> >> If I get all this correctly, we add a generic mechanic to be able to
> >> gate CI with block tests just because there is a single test failing
> >> with a single driver. Is that the right approach?
> > 
> > The env variable is the generic mechanism.
> > 
> > The yaml file exclusion for 185 is the special case, but we get
> > that basically for free with the former.
> > 
> >> In the future, do we expect to merge code breaking tests?
> > 
> > Yes. We will certainly merge more non-deterministic tests. We've seen
> > this over & over again. Something passes CI initially but after a
> > number of CI pipelines turns out to be flaky
> >
> 
> Then we can mark them as flaky in tests/qemu-iotests/meson.build.

That is a long term solution. It does not address the immediate
time critical goal to have the ability to fix a broken CI pipeline
immediately by skipping the test without waiting for code changes.

> It seems like you ignore the point that there is a problem between
> setting something in CI only vs making something that works for all
> users. I'm not against an env var, I just don't see how it answers this
> need.

Again, I'm not saying that we fix this only for CI. The env var is
to allow broken jobs to be immediately skipped, while waiting for
code changes to permanently skipped/fix the tests. The latter
addresses it for every scenario.

With regards,
Daniel
-- 
|: https://berrange.com       ~~        https://hachyderm.io/@berrange :|
|: https://libvirt.org          ~~          https://entangle-photo.org :|
|: https://pixelfed.art/berrange   ~~    https://fstop138.berrange.com :|

Re: [PATCH 14/16] tests: add QEMU_TEST_IO_SKIP for skipping I/O tests

[Pierrick Bouvier]

On 5/12/2026 11:46 AM, Daniel P. Berrangé wrote:
> On Tue, May 12, 2026 at 10:53:05AM -0700, Pierrick Bouvier wrote:
>> On 5/12/2026 10:24 AM, Daniel P. Berrangé wrote:
>>> On Tue, May 12, 2026 at 10:09:53AM -0700, Pierrick Bouvier wrote:
>>>> On 5/12/2026 9:53 AM, Daniel P. Berrangé wrote:
>>>>> On Tue, May 12, 2026 at 09:47:12AM -0700, Pierrick Bouvier wrote:
>>>>>> On 5/12/2026 9:36 AM, Daniel P. Berrangé wrote:
>>>>>>> On Tue, May 12, 2026 at 09:19:45AM -0700, Pierrick Bouvier wrote:
>>>>>>>> On 5/12/2026 9:06 AM, Daniel P. Berrangé wrote:
>>>>>>>>> On Tue, May 12, 2026 at 08:56:54AM -0700, Pierrick Bouvier wrote:
>>>>>>>>>> On 4/24/2026 8:42 AM, Daniel P. Berrangé wrote:
>>>>>>>>>>> The nature of block I/O tests is such that there can be unexpected false
>>>>>>>>>>> positive failures in certain scenarios that have not been encountered
>>>>>>>>>>> before, and sometimes non-deterministic failures that are hard to
>>>>>>>>>>> reproduce.
>>>>>>>>>>>
>>>>>>>>>>> Before enabling the I/O tests as gating jobs in CI, there needs to be a
>>>>>>>>>>> mechanism to dynamically mark tests as skipped, without having to commit
>>>>>>>>>>> code changes.
>>>>>>>>>>>
>>>>>>>>>>> This introduces the QEMU_TEST_IO_SKIP environment variable that is set
>>>>>>>>>>> to a list of FORMAT-OR-PROTOCOL:NAME pairs. The intent is that this
>>>>>>>>>>> variable can be set as a GitLab CI pipeline variable to temporarily
>>>>>>>>>>> disable a test while problems are being debugged.
>>>>>>>>>>>
>>>>>>>>>>> Reviewed-by: Thomas Huth <thuth@redhat.com>
>>>>>>>>>>> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
>>>>>>>>>>> ---
>>>>>>>>>>>  docs/devel/testing/main.rst      |  7 +++++++
>>>>>>>>>>>  tests/qemu-iotests/testrunner.py | 16 ++++++++++++++++
>>>>>>>>>>>  2 files changed, 23 insertions(+)
>>>>>>>>>>>
>>>>>>>>>>> diff --git a/docs/devel/testing/main.rst b/docs/devel/testing/main.rst
>>>>>>>>>>> index 797111009a..f779a64415 100644
>>>>>>>>>>> --- a/docs/devel/testing/main.rst
>>>>>>>>>>> +++ b/docs/devel/testing/main.rst
>>>>>>>>>>> @@ -284,6 +284,13 @@ that are specific to certain cache mode.
>>>>>>>>>>>  More options are supported by the ``./check`` script, run ``./check -h`` for
>>>>>>>>>>>  help.
>>>>>>>>>>>  
>>>>>>>>>>> +If a test program is known to be broken, it can be disabled by setting
>>>>>>>>>>> +the ``QEMU_TEST_IO_SKIP`` environment variable with a list of tests to
>>>>>>>>>>> +be skipped. The values are of the form FORMAT-OR-PROTOCOL:NAME, the
>>>>>>>>>>> +leading component can be omitted to skip the test for all formats and
>>>>>>>>>>> +protocols. For example ``export QEMU_TEST_IO_SKIP="luks:149 185 iov-padding``
>>>>>>>>>>> +will skip ``149`` for LUKS only, and ``185`` and ``iov-padding`` for all.
>>>>>>>>>>> +
>>>>>>>>>>>  Writing a new test case
>>>>>>>>>>>  ~~~~~~~~~~~~~~~~~~~~~~~
>>>>>>>>>>>  
>>>>>>>>>>> diff --git a/tests/qemu-iotests/testrunner.py b/tests/qemu-iotests/testrunner.py
>>>>>>>>>>> index dbe2dddc32..ecb5d4529f 100644
>>>>>>>>>>> --- a/tests/qemu-iotests/testrunner.py
>>>>>>>>>>> +++ b/tests/qemu-iotests/testrunner.py
>>>>>>>>>>> @@ -145,6 +145,18 @@ def __init__(self, env: TestEnv, tap: bool = False,
>>>>>>>>>>>  
>>>>>>>>>>>          self._stack: contextlib.ExitStack
>>>>>>>>>>>  
>>>>>>>>>>> +        self.skip = {}
>>>>>>>>>>> +        for rule in os.environ.get("QEMU_TEST_IO_SKIP", "").split(" "):
>>>>>>>>>>> +            rule = rule.strip()
>>>>>>>>>>> +            if rule == "":
>>>>>>>>>>> +                continue
>>>>>>>>>>> +            if ":" in rule:
>>>>>>>>>>> +                fmt, name = rule.split(":")
>>>>>>>>>>> +                if fmt in ("", env.imgfmt, env.imgproto):
>>>>>>>>>>> +                    self.skip[name] = True
>>>>>>>>>>> +            else:
>>>>>>>>>>> +                self.skip[rule] = True
>>>>>>>>>>> +
>>>>>>>>>>>      def __enter__(self) -> 'TestRunner':
>>>>>>>>>>>          self._stack = contextlib.ExitStack()
>>>>>>>>>>>          self._stack.enter_context(self.env)
>>>>>>>>>>> @@ -251,6 +263,10 @@ def do_run_test(self, test: str) -> TestResult:
>>>>>>>>>>>                                description='No qualified output '
>>>>>>>>>>>                                            f'(expected {f_reference})')
>>>>>>>>>>>  
>>>>>>>>>>> +        if f_test.name in self.skip:
>>>>>>>>>>> +            return TestResult(status='not run',
>>>>>>>>>>> +                              description='Listed in QEMU_TEST_IO_SKIP')
>>>>>>>>>>> +
>>>>>>>>>>>          args = [str(f_test.resolve())]
>>>>>>>>>>>          env = self.env.prepare_subprocess(args)
>>>>>>>>>>>  
>>>>>>>>>>
>>>>>>>>>> Why not simply remove the broken tests, and create issues to add them
>>>>>>>>>> again in the future?
>>>>>>>>>
>>>>>>>>> In theory that's what our policy today is, but in practice it is
>>>>>>>>> too much of a burden on the release co-ordinator, to expect them
>>>>>>>>> to create such a patch themselves, or wait on a subsys maintainer
>>>>>>>>> todo it for them.
>>>>>>>>>
>>>>>>>>> They end up just ignoring brokenness in CI which is a bad practice,
>>>>>>>>> and will prevent us ever making CI truely gating or switching to
>>>>>>>>> using MRs for pull requests. This gives us a super-fast way to skip
>>>>>>>>> flaky tests, while the subsystem maintainers figure out the right
>>>>>>>>> permanent answer.
>>>>>>>>>
>>>>>>>>
>>>>>>>> I disagree on this one, merging a single patch doing a git rm, and a git
>>>>>>>> revert later is not more expensive than merging a variable modifying a
>>>>>>>> variable in a yaml file.
>>>>>>>
>>>>>>> Any code changes like that need to be sent back to the subsystem
>>>>>>> maintainer to be acked. IMHO the release manager should not be
>>>>>>> unilaterally deleting tests without peer review.  So that's
>>>>>>> got a non-negligible turn around time, during which CI is broken.
>>>>>>>
>>>>>>
>>>>>> I accept the argument, but it seems like a workaround for a human
>>>>>> process, more than a proper solution to the problem.
>>>>>>
>>>>>> It would be better to have a proper policy for build/test fixes, instead
>>>>>> of implementing local overrides to this.
>>>>>>
>>>>>>> Setting an env variable to skip a problematic test is something
>>>>>>> reasonable to do with zero oversight.
>>>>>>>
>>>>>>>> The issue with this approach is that people running tests locally will
>>>>>>>> not see which tests are skipped, and will see false positives. So you
>>>>>>>> just keep CI green, but not the test base itself.
>>>>>>>
>>>>>>> I would still expect the release manager to file a bug about any
>>>>>>> flaky test they disable via the env var, and the subsystem maintainer
>>>>>>> should still be fixing it or disabling it such that tests won't fail
>>>>>>> more broadly, or deciding to remove it if terminally broken.
>>>>>>>
>>>>>>> We're just decoupling the process so that there is an immediate
>>>>>>> workaround possible. It can also be used by people working in
>>>>>>> their forks - often I've been testing stuff in my fork, but
>>>>>>> see spurious failures because git master has a non-deterministic
>>>>>>> test failure merged. I would like to easily skip those in my fork
>>>>>>> too, without adding extra commits to me working branches, as that
>>>>>>> would require the same commit to be duped into several in-progress
>>>>>>> branches, vs setting the env var once.
>>>>>>>
>>>>>>>> The risk I see is that some tests will stay forever in this skip
>>>>>>>> variable, so it will be dead code for CI, but still alive and failing
>>>>>>>> for people running tests manually who hit the regression.
>>>>>>>
>>>>>>> Again, there should be a bug filed for any flaky test. Anyone can
>>>>>>> do this, if they see it locally or in their fork CI, or in staging
>>>>>>> CI. If no one can see an obvious fix, then anyone can also propose
>>>>>>> to disable the test.
>>>>>>>
>>>>>>>> If you still want an alternative to removing test, implementing a
>>>>>>>> skip_list in tests/qemu-iotests/meson.build is better than an env var
>>>>>>>> IMHO, and achieves the exact same effect, for CI and for users.
>>>>>>>>
>>>>>>>> What do you think?
>>>>>>>
>>>>>>> IMHO there needs to be a way to skip flaky tests which does not
>>>>>>> require code changes as the only available option. Code changes
>>>>>>> are the permanent fix, env var is the immediate workaround.
>>>>>>>
>>>>>>
>>>>>> I'm not sure all this answers to my question about How to ensure users
>>>>>> who run tests and the CI both see the same skip list.
>>>>>>
>>>>>> I don't mind having an env var, a black list in meson or any other
>>>>>> solution, but having different results on a dev machine and in CI is not
>>>>>> a good design. So whatever the solution is, the CI yaml file is not the
>>>>>> proper place to store this information.
>>>>>
>>>>> AFAICT the test 185 that is being skipped in the CI yaml file only
>>>>> fails when run under gitlab. I've never seen a failure running it
>>>>> locally.
>>>>>
>>>>> If it failed locally too, then I'd agree that it should not be
>>>>> skipped in the CI yaml, but universally skipped in all scenarios.
>>>>>
>>>>
>>>> If I get all this correctly, we add a generic mechanic to be able to
>>>> gate CI with block tests just because there is a single test failing
>>>> with a single driver. Is that the right approach?
>>>
>>> The env variable is the generic mechanism.
>>>
>>> The yaml file exclusion for 185 is the special case, but we get
>>> that basically for free with the former.
>>>
>>>> In the future, do we expect to merge code breaking tests?
>>>
>>> Yes. We will certainly merge more non-deterministic tests. We've seen
>>> this over & over again. Something passes CI initially but after a
>>> number of CI pipelines turns out to be flaky
>>>
>>
>> Then we can mark them as flaky in tests/qemu-iotests/meson.build.
> 
> That is a long term solution. It does not address the immediate
> time critical goal to have the ability to fix a broken CI pipeline
> immediately by skipping the test without waiting for code changes.
> 
>> It seems like you ignore the point that there is a problem between
>> setting something in CI only vs making something that works for all
>> users. I'm not against an env var, I just don't see how it answers this
>> need.
> 
> Again, I'm not saying that we fix this only for CI. The env var is
> to allow broken jobs to be immediately skipped, while waiting for
> code changes to permanently skipped/fix the tests. The latter
> addresses it for every scenario.
>

I might have missed where we have a default value for this env var, out
of yaml file, that makes it apply the exact same set of skip tests for
CI, and for users running tests manually.

Where is this default applied for both CI and users?

I understand it's not needed for test 185 which fails only in GitLab,
but as you mentioned, we'll probably have non deterministic tests in the
future, so we need to consider this.

> With regards,
> Daniel

Re: [PATCH 14/16] tests: add QEMU_TEST_IO_SKIP for skipping I/O tests

[Daniel P. Berrangé]

On Tue, May 12, 2026 at 11:52:45AM -0700, Pierrick Bouvier wrote:
> On 5/12/2026 11:46 AM, Daniel P. Berrangé wrote:
> > On Tue, May 12, 2026 at 10:53:05AM -0700, Pierrick Bouvier wrote:
> >> On 5/12/2026 10:24 AM, Daniel P. Berrangé wrote:
> >>> On Tue, May 12, 2026 at 10:09:53AM -0700, Pierrick Bouvier wrote:
> >>>> On 5/12/2026 9:53 AM, Daniel P. Berrangé wrote:
> >>>>> On Tue, May 12, 2026 at 09:47:12AM -0700, Pierrick Bouvier wrote:
> >>>>>> On 5/12/2026 9:36 AM, Daniel P. Berrangé wrote:
> >>>>>>> On Tue, May 12, 2026 at 09:19:45AM -0700, Pierrick Bouvier wrote:
> >>>>>>>> On 5/12/2026 9:06 AM, Daniel P. Berrangé wrote:
> >>>>>>>>> On Tue, May 12, 2026 at 08:56:54AM -0700, Pierrick Bouvier wrote:
> >>>>>>>>>> On 4/24/2026 8:42 AM, Daniel P. Berrangé wrote:
> >>>>>>>>>>> The nature of block I/O tests is such that there can be unexpected false
> >>>>>>>>>>> positive failures in certain scenarios that have not been encountered
> >>>>>>>>>>> before, and sometimes non-deterministic failures that are hard to
> >>>>>>>>>>> reproduce.
> >>>>>>>>>>>
> >>>>>>>>>>> Before enabling the I/O tests as gating jobs in CI, there needs to be a
> >>>>>>>>>>> mechanism to dynamically mark tests as skipped, without having to commit
> >>>>>>>>>>> code changes.
> >>>>>>>>>>>
> >>>>>>>>>>> This introduces the QEMU_TEST_IO_SKIP environment variable that is set
> >>>>>>>>>>> to a list of FORMAT-OR-PROTOCOL:NAME pairs. The intent is that this
> >>>>>>>>>>> variable can be set as a GitLab CI pipeline variable to temporarily
> >>>>>>>>>>> disable a test while problems are being debugged.
> >>>>>>>>>>>
> >>>>>>>>>>> Reviewed-by: Thomas Huth <thuth@redhat.com>
> >>>>>>>>>>> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> >>>>>>>>>>> ---
> >>>>>>>>>>>  docs/devel/testing/main.rst      |  7 +++++++
> >>>>>>>>>>>  tests/qemu-iotests/testrunner.py | 16 ++++++++++++++++
> >>>>>>>>>>>  2 files changed, 23 insertions(+)
> >>>>>>>>>>>
> >>>>>>>>>>> diff --git a/docs/devel/testing/main.rst b/docs/devel/testing/main.rst
> >>>>>>>>>>> index 797111009a..f779a64415 100644
> >>>>>>>>>>> --- a/docs/devel/testing/main.rst
> >>>>>>>>>>> +++ b/docs/devel/testing/main.rst
> >>>>>>>>>>> @@ -284,6 +284,13 @@ that are specific to certain cache mode.
> >>>>>>>>>>>  More options are supported by the ``./check`` script, run ``./check -h`` for
> >>>>>>>>>>>  help.
> >>>>>>>>>>>  
> >>>>>>>>>>> +If a test program is known to be broken, it can be disabled by setting
> >>>>>>>>>>> +the ``QEMU_TEST_IO_SKIP`` environment variable with a list of tests to
> >>>>>>>>>>> +be skipped. The values are of the form FORMAT-OR-PROTOCOL:NAME, the
> >>>>>>>>>>> +leading component can be omitted to skip the test for all formats and
> >>>>>>>>>>> +protocols. For example ``export QEMU_TEST_IO_SKIP="luks:149 185 iov-padding``
> >>>>>>>>>>> +will skip ``149`` for LUKS only, and ``185`` and ``iov-padding`` for all.
> >>>>>>>>>>> +
> >>>>>>>>>>>  Writing a new test case
> >>>>>>>>>>>  ~~~~~~~~~~~~~~~~~~~~~~~
> >>>>>>>>>>>  
> >>>>>>>>>>> diff --git a/tests/qemu-iotests/testrunner.py b/tests/qemu-iotests/testrunner.py
> >>>>>>>>>>> index dbe2dddc32..ecb5d4529f 100644
> >>>>>>>>>>> --- a/tests/qemu-iotests/testrunner.py
> >>>>>>>>>>> +++ b/tests/qemu-iotests/testrunner.py
> >>>>>>>>>>> @@ -145,6 +145,18 @@ def __init__(self, env: TestEnv, tap: bool = False,
> >>>>>>>>>>>  
> >>>>>>>>>>>          self._stack: contextlib.ExitStack
> >>>>>>>>>>>  
> >>>>>>>>>>> +        self.skip = {}
> >>>>>>>>>>> +        for rule in os.environ.get("QEMU_TEST_IO_SKIP", "").split(" "):
> >>>>>>>>>>> +            rule = rule.strip()
> >>>>>>>>>>> +            if rule == "":
> >>>>>>>>>>> +                continue
> >>>>>>>>>>> +            if ":" in rule:
> >>>>>>>>>>> +                fmt, name = rule.split(":")
> >>>>>>>>>>> +                if fmt in ("", env.imgfmt, env.imgproto):
> >>>>>>>>>>> +                    self.skip[name] = True
> >>>>>>>>>>> +            else:
> >>>>>>>>>>> +                self.skip[rule] = True
> >>>>>>>>>>> +
> >>>>>>>>>>>      def __enter__(self) -> 'TestRunner':
> >>>>>>>>>>>          self._stack = contextlib.ExitStack()
> >>>>>>>>>>>          self._stack.enter_context(self.env)
> >>>>>>>>>>> @@ -251,6 +263,10 @@ def do_run_test(self, test: str) -> TestResult:
> >>>>>>>>>>>                                description='No qualified output '
> >>>>>>>>>>>                                            f'(expected {f_reference})')
> >>>>>>>>>>>  
> >>>>>>>>>>> +        if f_test.name in self.skip:
> >>>>>>>>>>> +            return TestResult(status='not run',
> >>>>>>>>>>> +                              description='Listed in QEMU_TEST_IO_SKIP')
> >>>>>>>>>>> +
> >>>>>>>>>>>          args = [str(f_test.resolve())]
> >>>>>>>>>>>          env = self.env.prepare_subprocess(args)
> >>>>>>>>>>>  
> >>>>>>>>>>
> >>>>>>>>>> Why not simply remove the broken tests, and create issues to add them
> >>>>>>>>>> again in the future?
> >>>>>>>>>
> >>>>>>>>> In theory that's what our policy today is, but in practice it is
> >>>>>>>>> too much of a burden on the release co-ordinator, to expect them
> >>>>>>>>> to create such a patch themselves, or wait on a subsys maintainer
> >>>>>>>>> todo it for them.
> >>>>>>>>>
> >>>>>>>>> They end up just ignoring brokenness in CI which is a bad practice,
> >>>>>>>>> and will prevent us ever making CI truely gating or switching to
> >>>>>>>>> using MRs for pull requests. This gives us a super-fast way to skip
> >>>>>>>>> flaky tests, while the subsystem maintainers figure out the right
> >>>>>>>>> permanent answer.
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>> I disagree on this one, merging a single patch doing a git rm, and a git
> >>>>>>>> revert later is not more expensive than merging a variable modifying a
> >>>>>>>> variable in a yaml file.
> >>>>>>>
> >>>>>>> Any code changes like that need to be sent back to the subsystem
> >>>>>>> maintainer to be acked. IMHO the release manager should not be
> >>>>>>> unilaterally deleting tests without peer review.  So that's
> >>>>>>> got a non-negligible turn around time, during which CI is broken.
> >>>>>>>
> >>>>>>
> >>>>>> I accept the argument, but it seems like a workaround for a human
> >>>>>> process, more than a proper solution to the problem.
> >>>>>>
> >>>>>> It would be better to have a proper policy for build/test fixes, instead
> >>>>>> of implementing local overrides to this.
> >>>>>>
> >>>>>>> Setting an env variable to skip a problematic test is something
> >>>>>>> reasonable to do with zero oversight.
> >>>>>>>
> >>>>>>>> The issue with this approach is that people running tests locally will
> >>>>>>>> not see which tests are skipped, and will see false positives. So you
> >>>>>>>> just keep CI green, but not the test base itself.
> >>>>>>>
> >>>>>>> I would still expect the release manager to file a bug about any
> >>>>>>> flaky test they disable via the env var, and the subsystem maintainer
> >>>>>>> should still be fixing it or disabling it such that tests won't fail
> >>>>>>> more broadly, or deciding to remove it if terminally broken.
> >>>>>>>
> >>>>>>> We're just decoupling the process so that there is an immediate
> >>>>>>> workaround possible. It can also be used by people working in
> >>>>>>> their forks - often I've been testing stuff in my fork, but
> >>>>>>> see spurious failures because git master has a non-deterministic
> >>>>>>> test failure merged. I would like to easily skip those in my fork
> >>>>>>> too, without adding extra commits to me working branches, as that
> >>>>>>> would require the same commit to be duped into several in-progress
> >>>>>>> branches, vs setting the env var once.
> >>>>>>>
> >>>>>>>> The risk I see is that some tests will stay forever in this skip
> >>>>>>>> variable, so it will be dead code for CI, but still alive and failing
> >>>>>>>> for people running tests manually who hit the regression.
> >>>>>>>
> >>>>>>> Again, there should be a bug filed for any flaky test. Anyone can
> >>>>>>> do this, if they see it locally or in their fork CI, or in staging
> >>>>>>> CI. If no one can see an obvious fix, then anyone can also propose
> >>>>>>> to disable the test.
> >>>>>>>
> >>>>>>>> If you still want an alternative to removing test, implementing a
> >>>>>>>> skip_list in tests/qemu-iotests/meson.build is better than an env var
> >>>>>>>> IMHO, and achieves the exact same effect, for CI and for users.
> >>>>>>>>
> >>>>>>>> What do you think?
> >>>>>>>
> >>>>>>> IMHO there needs to be a way to skip flaky tests which does not
> >>>>>>> require code changes as the only available option. Code changes
> >>>>>>> are the permanent fix, env var is the immediate workaround.
> >>>>>>>
> >>>>>>
> >>>>>> I'm not sure all this answers to my question about How to ensure users
> >>>>>> who run tests and the CI both see the same skip list.
> >>>>>>
> >>>>>> I don't mind having an env var, a black list in meson or any other
> >>>>>> solution, but having different results on a dev machine and in CI is not
> >>>>>> a good design. So whatever the solution is, the CI yaml file is not the
> >>>>>> proper place to store this information.
> >>>>>
> >>>>> AFAICT the test 185 that is being skipped in the CI yaml file only
> >>>>> fails when run under gitlab. I've never seen a failure running it
> >>>>> locally.
> >>>>>
> >>>>> If it failed locally too, then I'd agree that it should not be
> >>>>> skipped in the CI yaml, but universally skipped in all scenarios.
> >>>>>
> >>>>
> >>>> If I get all this correctly, we add a generic mechanic to be able to
> >>>> gate CI with block tests just because there is a single test failing
> >>>> with a single driver. Is that the right approach?
> >>>
> >>> The env variable is the generic mechanism.
> >>>
> >>> The yaml file exclusion for 185 is the special case, but we get
> >>> that basically for free with the former.
> >>>
> >>>> In the future, do we expect to merge code breaking tests?
> >>>
> >>> Yes. We will certainly merge more non-deterministic tests. We've seen
> >>> this over & over again. Something passes CI initially but after a
> >>> number of CI pipelines turns out to be flaky
> >>>
> >>
> >> Then we can mark them as flaky in tests/qemu-iotests/meson.build.
> > 
> > That is a long term solution. It does not address the immediate
> > time critical goal to have the ability to fix a broken CI pipeline
> > immediately by skipping the test without waiting for code changes.
> > 
> >> It seems like you ignore the point that there is a problem between
> >> setting something in CI only vs making something that works for all
> >> users. I'm not against an env var, I just don't see how it answers this
> >> need.
> > 
> > Again, I'm not saying that we fix this only for CI. The env var is
> > to allow broken jobs to be immediately skipped, while waiting for
> > code changes to permanently skipped/fix the tests. The latter
> > addresses it for every scenario.
> >
> 
> I might have missed where we have a default value for this env var, out
> of yaml file, that makes it apply the exact same set of skip tests for
> CI, and for users running tests manually.
> 
> Where is this default applied for both CI and users?
>
> I understand it's not needed for test 185 which fails only in GitLab,
> but as you mentioned, we'll probably have non deterministic tests in the
> future, so we need to consider this.

I was considering any change in meson.build to permanently skip a
test would be independent of the env var handling, and outside the
scope of this series since there's no need for it here.


With regards,
Daniel
-- 
|: https://berrange.com       ~~        https://hachyderm.io/@berrange :|
|: https://libvirt.org          ~~          https://entangle-photo.org :|
|: https://pixelfed.art/berrange   ~~    https://fstop138.berrange.com :|

Re: [PATCH 14/16] tests: add QEMU_TEST_IO_SKIP for skipping I/O tests

[Pierrick Bouvier]

On 5/12/2026 12:00 PM, Daniel P. Berrangé wrote:
> On Tue, May 12, 2026 at 11:52:45AM -0700, Pierrick Bouvier wrote:
>> On 5/12/2026 11:46 AM, Daniel P. Berrangé wrote:
>>> On Tue, May 12, 2026 at 10:53:05AM -0700, Pierrick Bouvier wrote:
>>>> On 5/12/2026 10:24 AM, Daniel P. Berrangé wrote:
>>>>> On Tue, May 12, 2026 at 10:09:53AM -0700, Pierrick Bouvier wrote:
>>>>>> On 5/12/2026 9:53 AM, Daniel P. Berrangé wrote:
>>>>>>> On Tue, May 12, 2026 at 09:47:12AM -0700, Pierrick Bouvier wrote:
>>>>>>>> On 5/12/2026 9:36 AM, Daniel P. Berrangé wrote:
>>>>>>>>> On Tue, May 12, 2026 at 09:19:45AM -0700, Pierrick Bouvier wrote:
>>>>>>>>>> On 5/12/2026 9:06 AM, Daniel P. Berrangé wrote:
>>>>>>>>>>> On Tue, May 12, 2026 at 08:56:54AM -0700, Pierrick Bouvier wrote:
>>>>>>>>>>>> On 4/24/2026 8:42 AM, Daniel P. Berrangé wrote:
>>>>>>>>>>>>> The nature of block I/O tests is such that there can be unexpected false
>>>>>>>>>>>>> positive failures in certain scenarios that have not been encountered
>>>>>>>>>>>>> before, and sometimes non-deterministic failures that are hard to
>>>>>>>>>>>>> reproduce.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Before enabling the I/O tests as gating jobs in CI, there needs to be a
>>>>>>>>>>>>> mechanism to dynamically mark tests as skipped, without having to commit
>>>>>>>>>>>>> code changes.
>>>>>>>>>>>>>
>>>>>>>>>>>>> This introduces the QEMU_TEST_IO_SKIP environment variable that is set
>>>>>>>>>>>>> to a list of FORMAT-OR-PROTOCOL:NAME pairs. The intent is that this
>>>>>>>>>>>>> variable can be set as a GitLab CI pipeline variable to temporarily
>>>>>>>>>>>>> disable a test while problems are being debugged.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Reviewed-by: Thomas Huth <thuth@redhat.com>
>>>>>>>>>>>>> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
>>>>>>>>>>>>> ---
>>>>>>>>>>>>>  docs/devel/testing/main.rst      |  7 +++++++
>>>>>>>>>>>>>  tests/qemu-iotests/testrunner.py | 16 ++++++++++++++++
>>>>>>>>>>>>>  2 files changed, 23 insertions(+)
>>>>>>>>>>>>>
>>>>>>>>>>>>> diff --git a/docs/devel/testing/main.rst b/docs/devel/testing/main.rst
>>>>>>>>>>>>> index 797111009a..f779a64415 100644
>>>>>>>>>>>>> --- a/docs/devel/testing/main.rst
>>>>>>>>>>>>> +++ b/docs/devel/testing/main.rst
>>>>>>>>>>>>> @@ -284,6 +284,13 @@ that are specific to certain cache mode.
>>>>>>>>>>>>>  More options are supported by the ``./check`` script, run ``./check -h`` for
>>>>>>>>>>>>>  help.
>>>>>>>>>>>>>  
>>>>>>>>>>>>> +If a test program is known to be broken, it can be disabled by setting
>>>>>>>>>>>>> +the ``QEMU_TEST_IO_SKIP`` environment variable with a list of tests to
>>>>>>>>>>>>> +be skipped. The values are of the form FORMAT-OR-PROTOCOL:NAME, the
>>>>>>>>>>>>> +leading component can be omitted to skip the test for all formats and
>>>>>>>>>>>>> +protocols. For example ``export QEMU_TEST_IO_SKIP="luks:149 185 iov-padding``
>>>>>>>>>>>>> +will skip ``149`` for LUKS only, and ``185`` and ``iov-padding`` for all.
>>>>>>>>>>>>> +
>>>>>>>>>>>>>  Writing a new test case
>>>>>>>>>>>>>  ~~~~~~~~~~~~~~~~~~~~~~~
>>>>>>>>>>>>>  
>>>>>>>>>>>>> diff --git a/tests/qemu-iotests/testrunner.py b/tests/qemu-iotests/testrunner.py
>>>>>>>>>>>>> index dbe2dddc32..ecb5d4529f 100644
>>>>>>>>>>>>> --- a/tests/qemu-iotests/testrunner.py
>>>>>>>>>>>>> +++ b/tests/qemu-iotests/testrunner.py
>>>>>>>>>>>>> @@ -145,6 +145,18 @@ def __init__(self, env: TestEnv, tap: bool = False,
>>>>>>>>>>>>>  
>>>>>>>>>>>>>          self._stack: contextlib.ExitStack
>>>>>>>>>>>>>  
>>>>>>>>>>>>> +        self.skip = {}
>>>>>>>>>>>>> +        for rule in os.environ.get("QEMU_TEST_IO_SKIP", "").split(" "):
>>>>>>>>>>>>> +            rule = rule.strip()
>>>>>>>>>>>>> +            if rule == "":
>>>>>>>>>>>>> +                continue
>>>>>>>>>>>>> +            if ":" in rule:
>>>>>>>>>>>>> +                fmt, name = rule.split(":")
>>>>>>>>>>>>> +                if fmt in ("", env.imgfmt, env.imgproto):
>>>>>>>>>>>>> +                    self.skip[name] = True
>>>>>>>>>>>>> +            else:
>>>>>>>>>>>>> +                self.skip[rule] = True
>>>>>>>>>>>>> +
>>>>>>>>>>>>>      def __enter__(self) -> 'TestRunner':
>>>>>>>>>>>>>          self._stack = contextlib.ExitStack()
>>>>>>>>>>>>>          self._stack.enter_context(self.env)
>>>>>>>>>>>>> @@ -251,6 +263,10 @@ def do_run_test(self, test: str) -> TestResult:
>>>>>>>>>>>>>                                description='No qualified output '
>>>>>>>>>>>>>                                            f'(expected {f_reference})')
>>>>>>>>>>>>>  
>>>>>>>>>>>>> +        if f_test.name in self.skip:
>>>>>>>>>>>>> +            return TestResult(status='not run',
>>>>>>>>>>>>> +                              description='Listed in QEMU_TEST_IO_SKIP')
>>>>>>>>>>>>> +
>>>>>>>>>>>>>          args = [str(f_test.resolve())]
>>>>>>>>>>>>>          env = self.env.prepare_subprocess(args)
>>>>>>>>>>>>>  
>>>>>>>>>>>>
>>>>>>>>>>>> Why not simply remove the broken tests, and create issues to add them
>>>>>>>>>>>> again in the future?
>>>>>>>>>>>
>>>>>>>>>>> In theory that's what our policy today is, but in practice it is
>>>>>>>>>>> too much of a burden on the release co-ordinator, to expect them
>>>>>>>>>>> to create such a patch themselves, or wait on a subsys maintainer
>>>>>>>>>>> todo it for them.
>>>>>>>>>>>
>>>>>>>>>>> They end up just ignoring brokenness in CI which is a bad practice,
>>>>>>>>>>> and will prevent us ever making CI truely gating or switching to
>>>>>>>>>>> using MRs for pull requests. This gives us a super-fast way to skip
>>>>>>>>>>> flaky tests, while the subsystem maintainers figure out the right
>>>>>>>>>>> permanent answer.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I disagree on this one, merging a single patch doing a git rm, and a git
>>>>>>>>>> revert later is not more expensive than merging a variable modifying a
>>>>>>>>>> variable in a yaml file.
>>>>>>>>>
>>>>>>>>> Any code changes like that need to be sent back to the subsystem
>>>>>>>>> maintainer to be acked. IMHO the release manager should not be
>>>>>>>>> unilaterally deleting tests without peer review.  So that's
>>>>>>>>> got a non-negligible turn around time, during which CI is broken.
>>>>>>>>>
>>>>>>>>
>>>>>>>> I accept the argument, but it seems like a workaround for a human
>>>>>>>> process, more than a proper solution to the problem.
>>>>>>>>
>>>>>>>> It would be better to have a proper policy for build/test fixes, instead
>>>>>>>> of implementing local overrides to this.
>>>>>>>>
>>>>>>>>> Setting an env variable to skip a problematic test is something
>>>>>>>>> reasonable to do with zero oversight.
>>>>>>>>>
>>>>>>>>>> The issue with this approach is that people running tests locally will
>>>>>>>>>> not see which tests are skipped, and will see false positives. So you
>>>>>>>>>> just keep CI green, but not the test base itself.
>>>>>>>>>
>>>>>>>>> I would still expect the release manager to file a bug about any
>>>>>>>>> flaky test they disable via the env var, and the subsystem maintainer
>>>>>>>>> should still be fixing it or disabling it such that tests won't fail
>>>>>>>>> more broadly, or deciding to remove it if terminally broken.
>>>>>>>>>
>>>>>>>>> We're just decoupling the process so that there is an immediate
>>>>>>>>> workaround possible. It can also be used by people working in
>>>>>>>>> their forks - often I've been testing stuff in my fork, but
>>>>>>>>> see spurious failures because git master has a non-deterministic
>>>>>>>>> test failure merged. I would like to easily skip those in my fork
>>>>>>>>> too, without adding extra commits to me working branches, as that
>>>>>>>>> would require the same commit to be duped into several in-progress
>>>>>>>>> branches, vs setting the env var once.
>>>>>>>>>
>>>>>>>>>> The risk I see is that some tests will stay forever in this skip
>>>>>>>>>> variable, so it will be dead code for CI, but still alive and failing
>>>>>>>>>> for people running tests manually who hit the regression.
>>>>>>>>>
>>>>>>>>> Again, there should be a bug filed for any flaky test. Anyone can
>>>>>>>>> do this, if they see it locally or in their fork CI, or in staging
>>>>>>>>> CI. If no one can see an obvious fix, then anyone can also propose
>>>>>>>>> to disable the test.
>>>>>>>>>
>>>>>>>>>> If you still want an alternative to removing test, implementing a
>>>>>>>>>> skip_list in tests/qemu-iotests/meson.build is better than an env var
>>>>>>>>>> IMHO, and achieves the exact same effect, for CI and for users.
>>>>>>>>>>
>>>>>>>>>> What do you think?
>>>>>>>>>
>>>>>>>>> IMHO there needs to be a way to skip flaky tests which does not
>>>>>>>>> require code changes as the only available option. Code changes
>>>>>>>>> are the permanent fix, env var is the immediate workaround.
>>>>>>>>>
>>>>>>>>
>>>>>>>> I'm not sure all this answers to my question about How to ensure users
>>>>>>>> who run tests and the CI both see the same skip list.
>>>>>>>>
>>>>>>>> I don't mind having an env var, a black list in meson or any other
>>>>>>>> solution, but having different results on a dev machine and in CI is not
>>>>>>>> a good design. So whatever the solution is, the CI yaml file is not the
>>>>>>>> proper place to store this information.
>>>>>>>
>>>>>>> AFAICT the test 185 that is being skipped in the CI yaml file only
>>>>>>> fails when run under gitlab. I've never seen a failure running it
>>>>>>> locally.
>>>>>>>
>>>>>>> If it failed locally too, then I'd agree that it should not be
>>>>>>> skipped in the CI yaml, but universally skipped in all scenarios.
>>>>>>>
>>>>>>
>>>>>> If I get all this correctly, we add a generic mechanic to be able to
>>>>>> gate CI with block tests just because there is a single test failing
>>>>>> with a single driver. Is that the right approach?
>>>>>
>>>>> The env variable is the generic mechanism.
>>>>>
>>>>> The yaml file exclusion for 185 is the special case, but we get
>>>>> that basically for free with the former.
>>>>>
>>>>>> In the future, do we expect to merge code breaking tests?
>>>>>
>>>>> Yes. We will certainly merge more non-deterministic tests. We've seen
>>>>> this over & over again. Something passes CI initially but after a
>>>>> number of CI pipelines turns out to be flaky
>>>>>
>>>>
>>>> Then we can mark them as flaky in tests/qemu-iotests/meson.build.
>>>
>>> That is a long term solution. It does not address the immediate
>>> time critical goal to have the ability to fix a broken CI pipeline
>>> immediately by skipping the test without waiting for code changes.
>>>
>>>> It seems like you ignore the point that there is a problem between
>>>> setting something in CI only vs making something that works for all
>>>> users. I'm not against an env var, I just don't see how it answers this
>>>> need.
>>>
>>> Again, I'm not saying that we fix this only for CI. The env var is
>>> to allow broken jobs to be immediately skipped, while waiting for
>>> code changes to permanently skipped/fix the tests. The latter
>>> addresses it for every scenario.
>>>
>>
>> I might have missed where we have a default value for this env var, out
>> of yaml file, that makes it apply the exact same set of skip tests for
>> CI, and for users running tests manually.
>>
>> Where is this default applied for both CI and users?
>>
>> I understand it's not needed for test 185 which fails only in GitLab,
>> but as you mentioned, we'll probably have non deterministic tests in the
>> future, so we need to consider this.
> 
> I was considering any change in meson.build to permanently skip a
> test would be independent of the env var handling, and outside the
> scope of this series since there's no need for it here.
>

Where is the default value for this env var applied for both CI and
users? yaml is for CI only.

> 
> With regards,
> Daniel

Re: [PATCH 14/16] tests: add QEMU_TEST_IO_SKIP for skipping I/O tests

[Thomas Huth]

On 12/05/2026 18.36, Daniel P. Berrangé wrote:
...
> IMHO there needs to be a way to skip flaky tests which does not
> require code changes as the only available option. Code changes
> are the permanent fix, env var is the immediate workaround.

The functional tests and the qtests both already use the 
QEMU_TEST_FLAKY_TESTS environment variable to decide whether to run a flaky 
test or not. Could we please use the same mechanism in the iotests, too? ... 
I'd really like to avoid that we have to remember specific ways to disable 
flaky tests for each test subsystem...

  Thanks,
   Thomas

Re: [PATCH 07/16] scripts/mtest2make: support optional tests grouping

[Daniel P. Berrangé]

On Tue, May 12, 2026 at 08:45:42AM -0700, Pierrick Bouvier wrote:
> On 4/24/2026 8:41 AM, Daniel P. Berrangé wrote:
> > Currently tests can be classified into three speed groups depending on
> > whether the meson suite name ends in '-slow' or '-thorough' or neither.
> > 
> > This gets turned into make targets that match the name of the meson
> > suite, with the speed suffix stripped. e.g.
> > 
> >  * suite=block ->  'make check-block'
> >  * suite=block-slow -> 'make check-block SPEED=slow'
> >  * suite=block-thorough -> 'make check-block SPEED=thorough'
> > 
> > The set of tests under the "thorough" speed, however, can get rather
> > large and it would be useful to have a way to expose further make
> > targets for directly running a particular subset of tests.
> > 
> > This needs a way to run a target without requiring the SPEED variable,
> > while also not having them enabled by default as if they were 'quick'
> > tests.
> > 
> > This modifies mtest2make.py to support this idea by allowing for a new
> > suffix '-optional' on a suite. When this is present, a correspondingly
> > named make target will be created without the '-optional' suffix which
> > will never be run automatically.
> > 
> > This is intended to be combined with use of other suites. For example,
> > a single NBD test might be added to two suites, 'block-thorough' and
> > 'block-nbd-optional'.
> > 
> > This would allow running it as part of all the block tests with
> > 'make check-block SPEED=thorough', and as part of a standalone target
> > 'make check-block-nbd'.
> > 
> > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> > ---
> >  scripts/mtest2make.py | 22 +++++++++++++++-------
> >  1 file changed, 15 insertions(+), 7 deletions(-)
> > 
> > diff --git a/scripts/mtest2make.py b/scripts/mtest2make.py
> > index 915f02d600..383ea68b16 100644
> > --- a/scripts/mtest2make.py
> > +++ b/scripts/mtest2make.py
> > @@ -22,7 +22,7 @@ def names(self, base):
> >  print(r'''
> >  SPEED = quick
> >  
> > -.speed.quick = $(sort $(filter-out %-slow %-thorough, $1))
> > +.speed.quick = $(sort $(filter-out %-slow %-thorough %-optional, $1))
> >  .speed.slow = $(sort $(filter-out %-thorough, $1))
> >  .speed.thorough = $(sort $1)
> >  
> > @@ -66,10 +66,15 @@ def process_tests(test, targets, suites):
> >              s = s[:-9]
> >              suites[s].speeds.add('thorough')
> >  
> > +def target_name(suite):
> > +    if suite.endswith('-optional'):
> > +        return suite[0:-9]
> > +    return suite
> > +
> >  def emit_prolog(suites, prefix):
> > -    all_targets = ' '.join((f'{prefix}-{k}'
> > +    all_targets = ' '.join((f'{prefix}-{target_name(k)}'
> >                              for k in sorted(suites.keys())))
> > -    all_xml = ' '.join((f'{prefix}-report-{k}.junit.xml'
> > +    all_xml = ' '.join((f'{prefix}-report-{target_name(k)}.junit.xml'
> >                          for k in sorted(suites.keys())))
> >      print()
> >      print(f'all-{prefix}-targets = {all_targets}')
> > @@ -83,14 +88,17 @@ def emit_prolog(suites, prefix):
> >      print(f'\t$(MAKE) {prefix}$* MTESTARGS="$(MTESTARGS) --logbase {prefix}-report$*" && ln -f meson-logs/$@ .')
> >  
> >  def emit_suite(name, suite, prefix):
> > +    tgtname = target_name(name)
> >      deps = ' '.join(sorted(suite.deps))
> >      print()
> > -    print(f'.{prefix}-{name}.deps = {deps}')
> > -    print(f'.ninja-goals.check-build += $(.{prefix}-{name}.deps)')
> > +    print(f'.{prefix}-{tgtname}.deps = {deps}')
> > +    print(f'.ninja-goals.check-build += $(.{prefix}-{tgtname}.deps)')
> >  
> >      names = ' '.join(sorted(suite.names(name)))
> > -    targets = f'{prefix}-{name} {prefix}-report-{name}.junit.xml'
> > -    if not name.endswith('-slow') and not name.endswith('-thorough'):
> > +    targets = f'{prefix}-{tgtname} {prefix}-report-{tgtname}.junit.xml'
> > +    if not name.endswith('-slow') and \
> > +       not name.endswith('-thorough') and \
> > +       not name.endswith('-optional'):
> >          targets += f' {prefix} {prefix}-report.junit.xml'
> >      print(f'ifneq ($(filter {targets}, $(MAKECMDGOALS)),)')
> >      # for the "base" suite possibly add FOO-slow and FOO-thorough
> 
> Reviewed-by: Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com>
> 
> Not blocking this patch, thus reviewed, but I think it would be a better
> abstraction to use meson suites for this (a test can be added in
> multiple suites), instead of adding more clunk to makefile layer.

This *is* using meson suites. The mtest2make script takes the list
of meson suites and generates makefile targets which can be used to
select execution of the meson suites. This change is exposing more
of the suites as makefile targets, since make is our primary entry
point.


With regards,
Daniel
-- 
|: https://berrange.com       ~~        https://hachyderm.io/@berrange :|
|: https://libvirt.org          ~~          https://entangle-photo.org :|
|: https://pixelfed.art/berrange   ~~    https://fstop138.berrange.com :|

Re: [PATCH 12/16] tests: validate dmsetup result in test 128

[Daniel P. Berrangé]

On Tue, May 12, 2026 at 08:53:47AM -0700, Pierrick Bouvier wrote:
> On 4/24/2026 8:42 AM, Daniel P. Berrangé wrote:
> > The I/O test 128 uses 'dmsetup create' to create a device, optionally
> > using sudo to elevate privileges.
> > 
> > This dmsetup command works in GitLab CI, however, the test then fails
> > with a missing device name:
> > 
> >   1..1
> >   # running raw 128
> >   not ok raw 128
> >   ----------------------------------- stderr -----------------------------------
> >   --- /builds/berrange/qemu/tests/qemu-iotests/128.out
> >   +++ /builds/berrange/qemu/build/scratch/raw-file-128/128.out.bad
> >   @@ -1,5 +1,5 @@
> >    QA output created by 128
> > 
> >    == reading from error device ==
> >   -read failed: Input/output error
> >   +qemu-io: can't open device /dev/mapper/eiodev16546: Could not open '/dev/mapper/eiodev16546': No such file or directory
> >    *** done
> > 
> >   (test program exited with status code 1)
> > 
> > It is believed that this is due to the build env using a manually
> > populated /dev, such that the device mapper node won't ever appear.
> > It is not a race, since a test adding a sleep did not result in the
> > device appearing.
> >
> 
> Could that be related to the fact it's running inside a container (if it
> is). Maybe it's just missing exposing all /dev instead of a limited set
> of devices.

We don't have any control over what's present in /dev in test environments,
so we shouldn't assume /dev automatically exposes devices that we can
create via device mapper, as it could be locked down.

> 
> > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> > ---
> >  tests/qemu-iotests/128 | 6 ++++++
> >  1 file changed, 6 insertions(+)
> > 
> > diff --git a/tests/qemu-iotests/128 b/tests/qemu-iotests/128
> > index d0e00d24b1..d75b1a451a 100755
> > --- a/tests/qemu-iotests/128
> > +++ b/tests/qemu-iotests/128
> > @@ -42,6 +42,12 @@ _setup_eiodev()
> >  		echo "0 $((1024 * 1024 * 1024 / 512)) error" | \
> >  			$cmd dmsetup create "$devname" 2>/dev/null
> >  		if [ "$?" -eq 0 ]; then
> > +			DEV="/dev/mapper/$devname"
> > +			if ! -e $DEV
> > +			then
> > +				_notrun "Device $DEV not appearing"
> > +			fi
> > +
> >  			sudo="$cmd"
> >  			return
> >  		fi
> 

With regards,
Daniel
-- 
|: https://berrange.com       ~~        https://hachyderm.io/@berrange :|
|: https://libvirt.org          ~~          https://entangle-photo.org :|
|: https://pixelfed.art/berrange   ~~    https://fstop138.berrange.com :|

Re: [PATCH 14/16] tests: add QEMU_TEST_IO_SKIP for skipping I/O tests

[Fabiano Rosas]

Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com> writes:

> On 5/12/2026 12:00 PM, Daniel P. Berrangé wrote:
>> On Tue, May 12, 2026 at 11:52:45AM -0700, Pierrick Bouvier wrote:
>>> On 5/12/2026 11:46 AM, Daniel P. Berrangé wrote:
>>>> On Tue, May 12, 2026 at 10:53:05AM -0700, Pierrick Bouvier wrote:
>>>>> On 5/12/2026 10:24 AM, Daniel P. Berrangé wrote:
>>>>>> On Tue, May 12, 2026 at 10:09:53AM -0700, Pierrick Bouvier wrote:
>>>>>>> On 5/12/2026 9:53 AM, Daniel P. Berrangé wrote:
>>>>>>>> On Tue, May 12, 2026 at 09:47:12AM -0700, Pierrick Bouvier wrote:
>>>>>>>>> On 5/12/2026 9:36 AM, Daniel P. Berrangé wrote:
>>>>>>>>>> On Tue, May 12, 2026 at 09:19:45AM -0700, Pierrick Bouvier wrote:
>>>>>>>>>>> On 5/12/2026 9:06 AM, Daniel P. Berrangé wrote:
>>>>>>>>>>>> On Tue, May 12, 2026 at 08:56:54AM -0700, Pierrick Bouvier wrote:
>>>>>>>>>>>>> On 4/24/2026 8:42 AM, Daniel P. Berrangé wrote:
>>>>>>>>>>>>>> The nature of block I/O tests is such that there can be unexpected false
>>>>>>>>>>>>>> positive failures in certain scenarios that have not been encountered
>>>>>>>>>>>>>> before, and sometimes non-deterministic failures that are hard to
>>>>>>>>>>>>>> reproduce.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Before enabling the I/O tests as gating jobs in CI, there needs to be a
>>>>>>>>>>>>>> mechanism to dynamically mark tests as skipped, without having to commit
>>>>>>>>>>>>>> code changes.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> This introduces the QEMU_TEST_IO_SKIP environment variable that is set
>>>>>>>>>>>>>> to a list of FORMAT-OR-PROTOCOL:NAME pairs. The intent is that this
>>>>>>>>>>>>>> variable can be set as a GitLab CI pipeline variable to temporarily
>>>>>>>>>>>>>> disable a test while problems are being debugged.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Reviewed-by: Thomas Huth <thuth@redhat.com>
>>>>>>>>>>>>>> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>  docs/devel/testing/main.rst      |  7 +++++++
>>>>>>>>>>>>>>  tests/qemu-iotests/testrunner.py | 16 ++++++++++++++++
>>>>>>>>>>>>>>  2 files changed, 23 insertions(+)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> diff --git a/docs/devel/testing/main.rst b/docs/devel/testing/main.rst
>>>>>>>>>>>>>> index 797111009a..f779a64415 100644
>>>>>>>>>>>>>> --- a/docs/devel/testing/main.rst
>>>>>>>>>>>>>> +++ b/docs/devel/testing/main.rst
>>>>>>>>>>>>>> @@ -284,6 +284,13 @@ that are specific to certain cache mode.
>>>>>>>>>>>>>>  More options are supported by the ``./check`` script, run ``./check -h`` for
>>>>>>>>>>>>>>  help.
>>>>>>>>>>>>>>  
>>>>>>>>>>>>>> +If a test program is known to be broken, it can be disabled by setting
>>>>>>>>>>>>>> +the ``QEMU_TEST_IO_SKIP`` environment variable with a list of tests to
>>>>>>>>>>>>>> +be skipped. The values are of the form FORMAT-OR-PROTOCOL:NAME, the
>>>>>>>>>>>>>> +leading component can be omitted to skip the test for all formats and
>>>>>>>>>>>>>> +protocols. For example ``export QEMU_TEST_IO_SKIP="luks:149 185 iov-padding``
>>>>>>>>>>>>>> +will skip ``149`` for LUKS only, and ``185`` and ``iov-padding`` for all.
>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>  Writing a new test case
>>>>>>>>>>>>>>  ~~~~~~~~~~~~~~~~~~~~~~~
>>>>>>>>>>>>>>  
>>>>>>>>>>>>>> diff --git a/tests/qemu-iotests/testrunner.py b/tests/qemu-iotests/testrunner.py
>>>>>>>>>>>>>> index dbe2dddc32..ecb5d4529f 100644
>>>>>>>>>>>>>> --- a/tests/qemu-iotests/testrunner.py
>>>>>>>>>>>>>> +++ b/tests/qemu-iotests/testrunner.py
>>>>>>>>>>>>>> @@ -145,6 +145,18 @@ def __init__(self, env: TestEnv, tap: bool = False,
>>>>>>>>>>>>>>  
>>>>>>>>>>>>>>          self._stack: contextlib.ExitStack
>>>>>>>>>>>>>>  
>>>>>>>>>>>>>> +        self.skip = {}
>>>>>>>>>>>>>> +        for rule in os.environ.get("QEMU_TEST_IO_SKIP", "").split(" "):
>>>>>>>>>>>>>> +            rule = rule.strip()
>>>>>>>>>>>>>> +            if rule == "":
>>>>>>>>>>>>>> +                continue
>>>>>>>>>>>>>> +            if ":" in rule:
>>>>>>>>>>>>>> +                fmt, name = rule.split(":")
>>>>>>>>>>>>>> +                if fmt in ("", env.imgfmt, env.imgproto):
>>>>>>>>>>>>>> +                    self.skip[name] = True
>>>>>>>>>>>>>> +            else:
>>>>>>>>>>>>>> +                self.skip[rule] = True
>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>      def __enter__(self) -> 'TestRunner':
>>>>>>>>>>>>>>          self._stack = contextlib.ExitStack()
>>>>>>>>>>>>>>          self._stack.enter_context(self.env)
>>>>>>>>>>>>>> @@ -251,6 +263,10 @@ def do_run_test(self, test: str) -> TestResult:
>>>>>>>>>>>>>>                                description='No qualified output '
>>>>>>>>>>>>>>                                            f'(expected {f_reference})')
>>>>>>>>>>>>>>  
>>>>>>>>>>>>>> +        if f_test.name in self.skip:
>>>>>>>>>>>>>> +            return TestResult(status='not run',
>>>>>>>>>>>>>> +                              description='Listed in QEMU_TEST_IO_SKIP')
>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>          args = [str(f_test.resolve())]
>>>>>>>>>>>>>>          env = self.env.prepare_subprocess(args)
>>>>>>>>>>>>>>  
>>>>>>>>>>>>>
>>>>>>>>>>>>> Why not simply remove the broken tests, and create issues to add them
>>>>>>>>>>>>> again in the future?
>>>>>>>>>>>>
>>>>>>>>>>>> In theory that's what our policy today is, but in practice it is
>>>>>>>>>>>> too much of a burden on the release co-ordinator, to expect them
>>>>>>>>>>>> to create such a patch themselves, or wait on a subsys maintainer
>>>>>>>>>>>> todo it for them.
>>>>>>>>>>>>
>>>>>>>>>>>> They end up just ignoring brokenness in CI which is a bad practice,
>>>>>>>>>>>> and will prevent us ever making CI truely gating or switching to
>>>>>>>>>>>> using MRs for pull requests. This gives us a super-fast way to skip
>>>>>>>>>>>> flaky tests, while the subsystem maintainers figure out the right
>>>>>>>>>>>> permanent answer.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I disagree on this one, merging a single patch doing a git rm, and a git
>>>>>>>>>>> revert later is not more expensive than merging a variable modifying a
>>>>>>>>>>> variable in a yaml file.
>>>>>>>>>>
>>>>>>>>>> Any code changes like that need to be sent back to the subsystem
>>>>>>>>>> maintainer to be acked. IMHO the release manager should not be
>>>>>>>>>> unilaterally deleting tests without peer review.  So that's
>>>>>>>>>> got a non-negligible turn around time, during which CI is broken.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I accept the argument, but it seems like a workaround for a human
>>>>>>>>> process, more than a proper solution to the problem.
>>>>>>>>>
>>>>>>>>> It would be better to have a proper policy for build/test fixes, instead
>>>>>>>>> of implementing local overrides to this.
>>>>>>>>>
>>>>>>>>>> Setting an env variable to skip a problematic test is something
>>>>>>>>>> reasonable to do with zero oversight.
>>>>>>>>>>
>>>>>>>>>>> The issue with this approach is that people running tests locally will
>>>>>>>>>>> not see which tests are skipped, and will see false positives. So you
>>>>>>>>>>> just keep CI green, but not the test base itself.
>>>>>>>>>>
>>>>>>>>>> I would still expect the release manager to file a bug about any
>>>>>>>>>> flaky test they disable via the env var, and the subsystem maintainer
>>>>>>>>>> should still be fixing it or disabling it such that tests won't fail
>>>>>>>>>> more broadly, or deciding to remove it if terminally broken.
>>>>>>>>>>
>>>>>>>>>> We're just decoupling the process so that there is an immediate
>>>>>>>>>> workaround possible. It can also be used by people working in
>>>>>>>>>> their forks - often I've been testing stuff in my fork, but
>>>>>>>>>> see spurious failures because git master has a non-deterministic
>>>>>>>>>> test failure merged. I would like to easily skip those in my fork
>>>>>>>>>> too, without adding extra commits to me working branches, as that
>>>>>>>>>> would require the same commit to be duped into several in-progress
>>>>>>>>>> branches, vs setting the env var once.
>>>>>>>>>>
>>>>>>>>>>> The risk I see is that some tests will stay forever in this skip
>>>>>>>>>>> variable, so it will be dead code for CI, but still alive and failing
>>>>>>>>>>> for people running tests manually who hit the regression.
>>>>>>>>>>
>>>>>>>>>> Again, there should be a bug filed for any flaky test. Anyone can
>>>>>>>>>> do this, if they see it locally or in their fork CI, or in staging
>>>>>>>>>> CI. If no one can see an obvious fix, then anyone can also propose
>>>>>>>>>> to disable the test.
>>>>>>>>>>
>>>>>>>>>>> If you still want an alternative to removing test, implementing a
>>>>>>>>>>> skip_list in tests/qemu-iotests/meson.build is better than an env var
>>>>>>>>>>> IMHO, and achieves the exact same effect, for CI and for users.
>>>>>>>>>>>
>>>>>>>>>>> What do you think?
>>>>>>>>>>
>>>>>>>>>> IMHO there needs to be a way to skip flaky tests which does not
>>>>>>>>>> require code changes as the only available option. Code changes
>>>>>>>>>> are the permanent fix, env var is the immediate workaround.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I'm not sure all this answers to my question about How to ensure users
>>>>>>>>> who run tests and the CI both see the same skip list.
>>>>>>>>>
>>>>>>>>> I don't mind having an env var, a black list in meson or any other
>>>>>>>>> solution, but having different results on a dev machine and in CI is not
>>>>>>>>> a good design. So whatever the solution is, the CI yaml file is not the
>>>>>>>>> proper place to store this information.
>>>>>>>>
>>>>>>>> AFAICT the test 185 that is being skipped in the CI yaml file only
>>>>>>>> fails when run under gitlab. I've never seen a failure running it
>>>>>>>> locally.
>>>>>>>>
>>>>>>>> If it failed locally too, then I'd agree that it should not be
>>>>>>>> skipped in the CI yaml, but universally skipped in all scenarios.
>>>>>>>>
>>>>>>>
>>>>>>> If I get all this correctly, we add a generic mechanic to be able to
>>>>>>> gate CI with block tests just because there is a single test failing
>>>>>>> with a single driver. Is that the right approach?
>>>>>>
>>>>>> The env variable is the generic mechanism.
>>>>>>
>>>>>> The yaml file exclusion for 185 is the special case, but we get
>>>>>> that basically for free with the former.
>>>>>>
>>>>>>> In the future, do we expect to merge code breaking tests?
>>>>>>
>>>>>> Yes. We will certainly merge more non-deterministic tests. We've seen
>>>>>> this over & over again. Something passes CI initially but after a
>>>>>> number of CI pipelines turns out to be flaky
>>>>>>
>>>>>
>>>>> Then we can mark them as flaky in tests/qemu-iotests/meson.build.
>>>>
>>>> That is a long term solution. It does not address the immediate
>>>> time critical goal to have the ability to fix a broken CI pipeline
>>>> immediately by skipping the test without waiting for code changes.
>>>>
>>>>> It seems like you ignore the point that there is a problem between
>>>>> setting something in CI only vs making something that works for all
>>>>> users. I'm not against an env var, I just don't see how it answers this
>>>>> need.
>>>>
>>>> Again, I'm not saying that we fix this only for CI. The env var is
>>>> to allow broken jobs to be immediately skipped, while waiting for
>>>> code changes to permanently skipped/fix the tests. The latter
>>>> addresses it for every scenario.
>>>>
>>>
>>> I might have missed where we have a default value for this env var, out
>>> of yaml file, that makes it apply the exact same set of skip tests for
>>> CI, and for users running tests manually.
>>>
>>> Where is this default applied for both CI and users?
>>>
>>> I understand it's not needed for test 185 which fails only in GitLab,
>>> but as you mentioned, we'll probably have non deterministic tests in the
>>> future, so we need to consider this.
>> 
>> I was considering any change in meson.build to permanently skip a
>> test would be independent of the env var handling, and outside the
>> scope of this series since there's no need for it here.
>>
>
> Where is the default value for this env var applied for both CI and
> users? yaml is for CI only.
>

Hi all,

I think Pierrick's concern is valid, but I see it slightly differently:

In the case of an environment variable, it's possible to set it without
any code change (even to .yaml files), by setting "-o ci.variable=<val>"
at git push time and also by setting directly in GitLab's web interface
under "New Pipeline". So it allows anyone wanting to "just get work
done" to skip broken tests for the duration of a pipeline.

It also works when running locally and skipping some tests for the
single local run.

All temporary changes, of course.

For the case of a flaky test that will need to be disabled for longer
than a single run, could we standardize on a list committed to
meson.build, so it works for both CI and local?  I think so, and I think
it's a good idea. Having recently taking up maintainership of QTests I
sometimes stumble into disabled tests that have been there for years and
everyone forgot about them. Same with the migration subsystem. Having a
single place to check from time to time would be helpful.

Migration, by the way, has been using the env variable approach in qtest
like this:

    /*
     * Our CI system has problems with shared memory.
     * Don't run this test until we find a workaround.
     */
    if (getenv("QEMU_TEST_FLAKY_TESTS")) {

Which is a case in which a single list committed in meson.build wouldn't
help. We actually do want to skip this only for CI.
("problems with shared memory" here means that CI environments simply
don't have enough shared memory space available to run all the tests
that make use of it)

A further complication is that in the migration case above, that skip
does not apply to the entire migration-test, it applies to a sub-test
(./migration-test -p /x86/migration/mode/reboot). Constructs in
meson.build would not be able to skip this, they can only skip the
entire migration-test. There is an -x option that gtester accepts, so
we'd need to write some meson-fu to invoke "./migration-test -x
/x86/migration/mode/reboot -x ..." if any subtest is to be skipped (note
that this is not migration-specific, all qtests that invoke
g_test_add_data* more than once are like this).

Maybe we could work on a proposal to improve this in a generic way for
all test frameworks.

PS: I'd even say there's a lot we could make common between
frameworks. A bunch of what happens around running a test is generic
programming, not tied to a specific programming language or accelerator
(in case of qtest).


>> 
>> With regards,
>> Daniel

Re: [PATCH 14/16] tests: add QEMU_TEST_IO_SKIP for skipping I/O tests

[Daniel P. Berrangé]

On Wed, May 13, 2026 at 11:11:47AM -0300, Fabiano Rosas wrote:
> In the case of an environment variable, it's possible to set it without
> any code change (even to .yaml files), by setting "-o ci.variable=<val>"
> at git push time and also by setting directly in GitLab's web interface
> under "New Pipeline". So it allows anyone wanting to "just get work
> done" to skip broken tests for the duration of a pipeline.
> 
> It also works when running locally and skipping some tests for the
> single local run.
> 
> All temporary changes, of course.
> 
> For the case of a flaky test that will need to be disabled for longer
> than a single run, could we standardize on a list committed to
> meson.build, so it works for both CI and local?  I think so, and I think
> it's a good idea. Having recently taking up maintainership of QTests I
> sometimes stumble into disabled tests that have been there for years and
> everyone forgot about them. Same with the migration subsystem. Having a
> single place to check from time to time would be helpful.

That was one of the reasons why when wwe introduced the "QEMU_TEST_FLAKY_TESTS"
env in tests/functional, we mandated a gitlab issue URL is provided. This 
gives us a record of why it was turned off both in the code and issue tracker.

If we can keep all permanent test disablement behind QEMU_TEST_FLAKY_TESTS
we avoid needing to add logic to meson.build too.


> Migration, by the way, has been using the env variable approach in qtest
> like this:
> 
>     /*
>      * Our CI system has problems with shared memory.
>      * Don't run this test until we find a workaround.
>      */
>     if (getenv("QEMU_TEST_FLAKY_TESTS")) {
> 
> Which is a case in which a single list committed in meson.build wouldn't
> help. We actually do want to skip this only for CI.
> ("problems with shared memory" here means that CI environments simply
> don't have enough shared memory space available to run all the tests
> that make use of it)
> 
> A further complication is that in the migration case above, that skip
> does not apply to the entire migration-test, it applies to a sub-test
> (./migration-test -p /x86/migration/mode/reboot). Constructs in
> meson.build would not be able to skip this, they can only skip the
> entire migration-test. There is an -x option that gtester accepts, so
> we'd need to write some meson-fu to invoke "./migration-test -x
> /x86/migration/mode/reboot -x ..." if any subtest is to be skipped (note
> that this is not migration-specific, all qtests that invoke
> g_test_add_data* more than once are like this).
> 
> Maybe we could work on a proposal to improve this in a generic way for
> all test frameworks.

I have dropped this patch in my v6 posting, as it would indeed be
much more useful if it worked across all the test subsystems.

My v6 has meanwhile introduced the QEMU_TEST_FLAKY_TESTS concept
to the I/O tests (as suggested by Thomas), to skip the dodgy I/O
test universally in all scenarios which satisfies Pierrick's
request.

> PS: I'd even say there's a lot we could make common between
> frameworks. A bunch of what happens around running a test is generic
> programming, not tied to a specific programming language or accelerator
> (in case of qtest).

Yeah, getting everything being represented as meson tests was one
of the things we did to make the frameworks more common.

I think there's scope for having more common env variable names
too. eg we have different env variable names to specify the name
of the QEMU emulator binary which is a bit silly.

With regards,
Daniel
-- 
|: https://berrange.com       ~~        https://hachyderm.io/@berrange :|
|: https://libvirt.org          ~~          https://entangle-photo.org :|
|: https://pixelfed.art/berrange   ~~    https://fstop138.berrange.com :|

Re: [PATCH 07/16] scripts/mtest2make: support optional tests grouping

[Pierrick Bouvier]

On 5/13/2026 3:08 AM, Daniel P. Berrangé wrote:
> On Tue, May 12, 2026 at 08:45:42AM -0700, Pierrick Bouvier wrote:
>> On 4/24/2026 8:41 AM, Daniel P. Berrangé wrote:
>>> Currently tests can be classified into three speed groups depending on
>>> whether the meson suite name ends in '-slow' or '-thorough' or neither.
>>>
>>> This gets turned into make targets that match the name of the meson
>>> suite, with the speed suffix stripped. e.g.
>>>
>>>  * suite=block ->  'make check-block'
>>>  * suite=block-slow -> 'make check-block SPEED=slow'
>>>  * suite=block-thorough -> 'make check-block SPEED=thorough'
>>>
>>> The set of tests under the "thorough" speed, however, can get rather
>>> large and it would be useful to have a way to expose further make
>>> targets for directly running a particular subset of tests.
>>>
>>> This needs a way to run a target without requiring the SPEED variable,
>>> while also not having them enabled by default as if they were 'quick'
>>> tests.
>>>
>>> This modifies mtest2make.py to support this idea by allowing for a new
>>> suffix '-optional' on a suite. When this is present, a correspondingly
>>> named make target will be created without the '-optional' suffix which
>>> will never be run automatically.
>>>
>>> This is intended to be combined with use of other suites. For example,
>>> a single NBD test might be added to two suites, 'block-thorough' and
>>> 'block-nbd-optional'.
>>>
>>> This would allow running it as part of all the block tests with
>>> 'make check-block SPEED=thorough', and as part of a standalone target
>>> 'make check-block-nbd'.
>>>
>>> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
>>> ---
>>>  scripts/mtest2make.py | 22 +++++++++++++++-------
>>>  1 file changed, 15 insertions(+), 7 deletions(-)
>>>
>>> diff --git a/scripts/mtest2make.py b/scripts/mtest2make.py
>>> index 915f02d600..383ea68b16 100644
>>> --- a/scripts/mtest2make.py
>>> +++ b/scripts/mtest2make.py
>>> @@ -22,7 +22,7 @@ def names(self, base):
>>>  print(r'''
>>>  SPEED = quick
>>>  
>>> -.speed.quick = $(sort $(filter-out %-slow %-thorough, $1))
>>> +.speed.quick = $(sort $(filter-out %-slow %-thorough %-optional, $1))
>>>  .speed.slow = $(sort $(filter-out %-thorough, $1))
>>>  .speed.thorough = $(sort $1)
>>>  
>>> @@ -66,10 +66,15 @@ def process_tests(test, targets, suites):
>>>              s = s[:-9]
>>>              suites[s].speeds.add('thorough')
>>>  
>>> +def target_name(suite):
>>> +    if suite.endswith('-optional'):
>>> +        return suite[0:-9]
>>> +    return suite
>>> +
>>>  def emit_prolog(suites, prefix):
>>> -    all_targets = ' '.join((f'{prefix}-{k}'
>>> +    all_targets = ' '.join((f'{prefix}-{target_name(k)}'
>>>                              for k in sorted(suites.keys())))
>>> -    all_xml = ' '.join((f'{prefix}-report-{k}.junit.xml'
>>> +    all_xml = ' '.join((f'{prefix}-report-{target_name(k)}.junit.xml'
>>>                          for k in sorted(suites.keys())))
>>>      print()
>>>      print(f'all-{prefix}-targets = {all_targets}')
>>> @@ -83,14 +88,17 @@ def emit_prolog(suites, prefix):
>>>      print(f'\t$(MAKE) {prefix}$* MTESTARGS="$(MTESTARGS) --logbase {prefix}-report$*" && ln -f meson-logs/$@ .')
>>>  
>>>  def emit_suite(name, suite, prefix):
>>> +    tgtname = target_name(name)
>>>      deps = ' '.join(sorted(suite.deps))
>>>      print()
>>> -    print(f'.{prefix}-{name}.deps = {deps}')
>>> -    print(f'.ninja-goals.check-build += $(.{prefix}-{name}.deps)')
>>> +    print(f'.{prefix}-{tgtname}.deps = {deps}')
>>> +    print(f'.ninja-goals.check-build += $(.{prefix}-{tgtname}.deps)')
>>>  
>>>      names = ' '.join(sorted(suite.names(name)))
>>> -    targets = f'{prefix}-{name} {prefix}-report-{name}.junit.xml'
>>> -    if not name.endswith('-slow') and not name.endswith('-thorough'):
>>> +    targets = f'{prefix}-{tgtname} {prefix}-report-{tgtname}.junit.xml'
>>> +    if not name.endswith('-slow') and \
>>> +       not name.endswith('-thorough') and \
>>> +       not name.endswith('-optional'):
>>>          targets += f' {prefix} {prefix}-report.junit.xml'
>>>      print(f'ifneq ($(filter {targets}, $(MAKECMDGOALS)),)')
>>>      # for the "base" suite possibly add FOO-slow and FOO-thorough
>>
>> Reviewed-by: Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com>
>>
>> Not blocking this patch, thus reviewed, but I think it would be a better
>> abstraction to use meson suites for this (a test can be added in
>> multiple suites), instead of adding more clunk to makefile layer.
> 
> This *is* using meson suites. The mtest2make script takes the list
> of meson suites and generates makefile targets which can be used to
> select execution of the meson suites. This change is exposing more
> of the suites as makefile targets, since make is our primary entry
> point.
> 
> 
> With regards,
> Daniel

Maybe I can't read this patch correctly, but from it, and the message
description, I understand that -optional is a pure Makefile abstraction,
and does not reflect any real meson suite.

```
This modifies mtest2make.py to support this idea by allowing for a new
suffix '-optional' on a suite. When this is present, a correspondingly
named make target will be created without the '-optional' suffix which
will never be run automatically.
```

Regards,
Pierrick

Re: [PATCH 12/16] tests: validate dmsetup result in test 128

[Pierrick Bouvier]

On 5/13/2026 3:11 AM, Daniel P. Berrangé wrote:
> On Tue, May 12, 2026 at 08:53:47AM -0700, Pierrick Bouvier wrote:
>> On 4/24/2026 8:42 AM, Daniel P. Berrangé wrote:
>>> The I/O test 128 uses 'dmsetup create' to create a device, optionally
>>> using sudo to elevate privileges.
>>>
>>> This dmsetup command works in GitLab CI, however, the test then fails
>>> with a missing device name:
>>>
>>>   1..1
>>>   # running raw 128
>>>   not ok raw 128
>>>   ----------------------------------- stderr -----------------------------------
>>>   --- /builds/berrange/qemu/tests/qemu-iotests/128.out
>>>   +++ /builds/berrange/qemu/build/scratch/raw-file-128/128.out.bad
>>>   @@ -1,5 +1,5 @@
>>>    QA output created by 128
>>>
>>>    == reading from error device ==
>>>   -read failed: Input/output error
>>>   +qemu-io: can't open device /dev/mapper/eiodev16546: Could not open '/dev/mapper/eiodev16546': No such file or directory
>>>    *** done
>>>
>>>   (test program exited with status code 1)
>>>
>>> It is believed that this is due to the build env using a manually
>>> populated /dev, such that the device mapper node won't ever appear.
>>> It is not a race, since a test adding a sleep did not result in the
>>> device appearing.
>>>
>>
>> Could that be related to the fact it's running inside a container (if it
>> is). Maybe it's just missing exposing all /dev instead of a limited set
>> of devices.
> 
> We don't have any control over what's present in /dev in test environments,
> so we shouldn't assume /dev automatically exposes devices that we can
> create via device mapper, as it could be locked down.
>

Ok.

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com>

Regards,
Pierrick

Re: [PATCH 14/16] tests: add QEMU_TEST_IO_SKIP for skipping I/O tests

[Pierrick Bouvier]

On 5/12/2026 11:18 PM, Thomas Huth wrote:
> On 12/05/2026 18.36, Daniel P. Berrangé wrote:
> ...
>> IMHO there needs to be a way to skip flaky tests which does not
>> require code changes as the only available option. Code changes
>> are the permanent fix, env var is the immediate workaround.
> 
> The functional tests and the qtests both already use the
> QEMU_TEST_FLAKY_TESTS environment variable to decide whether to run a
> flaky test or not. Could we please use the same mechanism in the
> iotests, too? ... I'd really like to avoid that we have to remember
> specific ways to disable flaky tests for each test subsystem...
>

This sounds like a good idea to reuse the same mechanism.
And should be easy to implement this is in iotest shell script directly.

>  Thanks,
>   Thomas
> 

Re: [PATCH 07/16] scripts/mtest2make: support optional tests grouping

[Daniel P. Berrangé]

On Wed, May 13, 2026 at 08:49:39AM -0700, Pierrick Bouvier wrote:
> On 5/13/2026 3:08 AM, Daniel P. Berrangé wrote:
> > On Tue, May 12, 2026 at 08:45:42AM -0700, Pierrick Bouvier wrote:
> >> On 4/24/2026 8:41 AM, Daniel P. Berrangé wrote:
> >>> Currently tests can be classified into three speed groups depending on
> >>> whether the meson suite name ends in '-slow' or '-thorough' or neither.
> >>>
> >>> This gets turned into make targets that match the name of the meson
> >>> suite, with the speed suffix stripped. e.g.
> >>>
> >>>  * suite=block ->  'make check-block'
> >>>  * suite=block-slow -> 'make check-block SPEED=slow'
> >>>  * suite=block-thorough -> 'make check-block SPEED=thorough'
> >>>
> >>> The set of tests under the "thorough" speed, however, can get rather
> >>> large and it would be useful to have a way to expose further make
> >>> targets for directly running a particular subset of tests.
> >>>
> >>> This needs a way to run a target without requiring the SPEED variable,
> >>> while also not having them enabled by default as if they were 'quick'
> >>> tests.
> >>>
> >>> This modifies mtest2make.py to support this idea by allowing for a new
> >>> suffix '-optional' on a suite. When this is present, a correspondingly
> >>> named make target will be created without the '-optional' suffix which
> >>> will never be run automatically.
> >>>
> >>> This is intended to be combined with use of other suites. For example,
> >>> a single NBD test might be added to two suites, 'block-thorough' and
> >>> 'block-nbd-optional'.
> >>>
> >>> This would allow running it as part of all the block tests with
> >>> 'make check-block SPEED=thorough', and as part of a standalone target
> >>> 'make check-block-nbd'.
> >>>
> >>> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> >>> ---
> >>>  scripts/mtest2make.py | 22 +++++++++++++++-------
> >>>  1 file changed, 15 insertions(+), 7 deletions(-)
> >>>
> >>> diff --git a/scripts/mtest2make.py b/scripts/mtest2make.py
> >>> index 915f02d600..383ea68b16 100644
> >>> --- a/scripts/mtest2make.py
> >>> +++ b/scripts/mtest2make.py
> >>> @@ -22,7 +22,7 @@ def names(self, base):
> >>>  print(r'''
> >>>  SPEED = quick
> >>>  
> >>> -.speed.quick = $(sort $(filter-out %-slow %-thorough, $1))
> >>> +.speed.quick = $(sort $(filter-out %-slow %-thorough %-optional, $1))
> >>>  .speed.slow = $(sort $(filter-out %-thorough, $1))
> >>>  .speed.thorough = $(sort $1)
> >>>  
> >>> @@ -66,10 +66,15 @@ def process_tests(test, targets, suites):
> >>>              s = s[:-9]
> >>>              suites[s].speeds.add('thorough')
> >>>  
> >>> +def target_name(suite):
> >>> +    if suite.endswith('-optional'):
> >>> +        return suite[0:-9]
> >>> +    return suite
> >>> +
> >>>  def emit_prolog(suites, prefix):
> >>> -    all_targets = ' '.join((f'{prefix}-{k}'
> >>> +    all_targets = ' '.join((f'{prefix}-{target_name(k)}'
> >>>                              for k in sorted(suites.keys())))
> >>> -    all_xml = ' '.join((f'{prefix}-report-{k}.junit.xml'
> >>> +    all_xml = ' '.join((f'{prefix}-report-{target_name(k)}.junit.xml'
> >>>                          for k in sorted(suites.keys())))
> >>>      print()
> >>>      print(f'all-{prefix}-targets = {all_targets}')
> >>> @@ -83,14 +88,17 @@ def emit_prolog(suites, prefix):
> >>>      print(f'\t$(MAKE) {prefix}$* MTESTARGS="$(MTESTARGS) --logbase {prefix}-report$*" && ln -f meson-logs/$@ .')
> >>>  
> >>>  def emit_suite(name, suite, prefix):
> >>> +    tgtname = target_name(name)
> >>>      deps = ' '.join(sorted(suite.deps))
> >>>      print()
> >>> -    print(f'.{prefix}-{name}.deps = {deps}')
> >>> -    print(f'.ninja-goals.check-build += $(.{prefix}-{name}.deps)')
> >>> +    print(f'.{prefix}-{tgtname}.deps = {deps}')
> >>> +    print(f'.ninja-goals.check-build += $(.{prefix}-{tgtname}.deps)')
> >>>  
> >>>      names = ' '.join(sorted(suite.names(name)))
> >>> -    targets = f'{prefix}-{name} {prefix}-report-{name}.junit.xml'
> >>> -    if not name.endswith('-slow') and not name.endswith('-thorough'):
> >>> +    targets = f'{prefix}-{tgtname} {prefix}-report-{tgtname}.junit.xml'
> >>> +    if not name.endswith('-slow') and \
> >>> +       not name.endswith('-thorough') and \
> >>> +       not name.endswith('-optional'):
> >>>          targets += f' {prefix} {prefix}-report.junit.xml'
> >>>      print(f'ifneq ($(filter {targets}, $(MAKECMDGOALS)),)')
> >>>      # for the "base" suite possibly add FOO-slow and FOO-thorough
> >>
> >> Reviewed-by: Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com>
> >>
> >> Not blocking this patch, thus reviewed, but I think it would be a better
> >> abstraction to use meson suites for this (a test can be added in
> >> multiple suites), instead of adding more clunk to makefile layer.
> > 
> > This *is* using meson suites. The mtest2make script takes the list
> > of meson suites and generates makefile targets which can be used to
> > select execution of the meson suites. This change is exposing more
> > of the suites as makefile targets, since make is our primary entry
> > point.
> > 
> > 
> > With regards,
> > Daniel
> 
> Maybe I can't read this patch correctly, but from it, and the message
> description, I understand that -optional is a pure Makefile abstraction,
> and does not reflect any real meson suite.

Actually its the other way around. In the next patch in this series
we create meson suites like  "block-nbd-optional" and that gets turned
into a makefile target  "check-block-nbd". The "optional" part of the
meson suite name determines that "check-block-nbd" is NOT added as a
depedency of "check".  IOW, it doesn't get run by "make check", but
must be optionally run through "make check-block-nbd" when needed.

With regards,
Daniel
-- 
|: https://berrange.com       ~~        https://hachyderm.io/@berrange :|
|: https://libvirt.org          ~~          https://entangle-photo.org :|
|: https://pixelfed.art/berrange   ~~    https://fstop138.berrange.com :|

Re: [PATCH 07/16] scripts/mtest2make: support optional tests grouping

[Pierrick Bouvier]

On 5/13/2026 10:15 AM, Daniel P. Berrangé wrote:
> On Wed, May 13, 2026 at 08:49:39AM -0700, Pierrick Bouvier wrote:
>> On 5/13/2026 3:08 AM, Daniel P. Berrangé wrote:
>>> On Tue, May 12, 2026 at 08:45:42AM -0700, Pierrick Bouvier wrote:
>>>> On 4/24/2026 8:41 AM, Daniel P. Berrangé wrote:
>>>>> Currently tests can be classified into three speed groups depending on
>>>>> whether the meson suite name ends in '-slow' or '-thorough' or neither.
>>>>>
>>>>> This gets turned into make targets that match the name of the meson
>>>>> suite, with the speed suffix stripped. e.g.
>>>>>
>>>>>  * suite=block ->  'make check-block'
>>>>>  * suite=block-slow -> 'make check-block SPEED=slow'
>>>>>  * suite=block-thorough -> 'make check-block SPEED=thorough'
>>>>>
>>>>> The set of tests under the "thorough" speed, however, can get rather
>>>>> large and it would be useful to have a way to expose further make
>>>>> targets for directly running a particular subset of tests.
>>>>>
>>>>> This needs a way to run a target without requiring the SPEED variable,
>>>>> while also not having them enabled by default as if they were 'quick'
>>>>> tests.
>>>>>
>>>>> This modifies mtest2make.py to support this idea by allowing for a new
>>>>> suffix '-optional' on a suite. When this is present, a correspondingly
>>>>> named make target will be created without the '-optional' suffix which
>>>>> will never be run automatically.
>>>>>
>>>>> This is intended to be combined with use of other suites. For example,
>>>>> a single NBD test might be added to two suites, 'block-thorough' and
>>>>> 'block-nbd-optional'.
>>>>>
>>>>> This would allow running it as part of all the block tests with
>>>>> 'make check-block SPEED=thorough', and as part of a standalone target
>>>>> 'make check-block-nbd'.
>>>>>
>>>>> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
>>>>> ---
>>>>>  scripts/mtest2make.py | 22 +++++++++++++++-------
>>>>>  1 file changed, 15 insertions(+), 7 deletions(-)
>>>>>
>>>>> diff --git a/scripts/mtest2make.py b/scripts/mtest2make.py
>>>>> index 915f02d600..383ea68b16 100644
>>>>> --- a/scripts/mtest2make.py
>>>>> +++ b/scripts/mtest2make.py
>>>>> @@ -22,7 +22,7 @@ def names(self, base):
>>>>>  print(r'''
>>>>>  SPEED = quick
>>>>>  
>>>>> -.speed.quick = $(sort $(filter-out %-slow %-thorough, $1))
>>>>> +.speed.quick = $(sort $(filter-out %-slow %-thorough %-optional, $1))
>>>>>  .speed.slow = $(sort $(filter-out %-thorough, $1))
>>>>>  .speed.thorough = $(sort $1)
>>>>>  
>>>>> @@ -66,10 +66,15 @@ def process_tests(test, targets, suites):
>>>>>              s = s[:-9]
>>>>>              suites[s].speeds.add('thorough')
>>>>>  
>>>>> +def target_name(suite):
>>>>> +    if suite.endswith('-optional'):
>>>>> +        return suite[0:-9]
>>>>> +    return suite
>>>>> +
>>>>>  def emit_prolog(suites, prefix):
>>>>> -    all_targets = ' '.join((f'{prefix}-{k}'
>>>>> +    all_targets = ' '.join((f'{prefix}-{target_name(k)}'
>>>>>                              for k in sorted(suites.keys())))
>>>>> -    all_xml = ' '.join((f'{prefix}-report-{k}.junit.xml'
>>>>> +    all_xml = ' '.join((f'{prefix}-report-{target_name(k)}.junit.xml'
>>>>>                          for k in sorted(suites.keys())))
>>>>>      print()
>>>>>      print(f'all-{prefix}-targets = {all_targets}')
>>>>> @@ -83,14 +88,17 @@ def emit_prolog(suites, prefix):
>>>>>      print(f'\t$(MAKE) {prefix}$* MTESTARGS="$(MTESTARGS) --logbase {prefix}-report$*" && ln -f meson-logs/$@ .')
>>>>>  
>>>>>  def emit_suite(name, suite, prefix):
>>>>> +    tgtname = target_name(name)
>>>>>      deps = ' '.join(sorted(suite.deps))
>>>>>      print()
>>>>> -    print(f'.{prefix}-{name}.deps = {deps}')
>>>>> -    print(f'.ninja-goals.check-build += $(.{prefix}-{name}.deps)')
>>>>> +    print(f'.{prefix}-{tgtname}.deps = {deps}')
>>>>> +    print(f'.ninja-goals.check-build += $(.{prefix}-{tgtname}.deps)')
>>>>>  
>>>>>      names = ' '.join(sorted(suite.names(name)))
>>>>> -    targets = f'{prefix}-{name} {prefix}-report-{name}.junit.xml'
>>>>> -    if not name.endswith('-slow') and not name.endswith('-thorough'):
>>>>> +    targets = f'{prefix}-{tgtname} {prefix}-report-{tgtname}.junit.xml'
>>>>> +    if not name.endswith('-slow') and \
>>>>> +       not name.endswith('-thorough') and \
>>>>> +       not name.endswith('-optional'):
>>>>>          targets += f' {prefix} {prefix}-report.junit.xml'
>>>>>      print(f'ifneq ($(filter {targets}, $(MAKECMDGOALS)),)')
>>>>>      # for the "base" suite possibly add FOO-slow and FOO-thorough
>>>>
>>>> Reviewed-by: Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com>
>>>>
>>>> Not blocking this patch, thus reviewed, but I think it would be a better
>>>> abstraction to use meson suites for this (a test can be added in
>>>> multiple suites), instead of adding more clunk to makefile layer.
>>>
>>> This *is* using meson suites. The mtest2make script takes the list
>>> of meson suites and generates makefile targets which can be used to
>>> select execution of the meson suites. This change is exposing more
>>> of the suites as makefile targets, since make is our primary entry
>>> point.
>>>
>>>
>>> With regards,
>>> Daniel
>>
>> Maybe I can't read this patch correctly, but from it, and the message
>> description, I understand that -optional is a pure Makefile abstraction,
>> and does not reflect any real meson suite.
> 
> Actually its the other way around. In the next patch in this series
> we create meson suites like  "block-nbd-optional" and that gets turned
> into a makefile target  "check-block-nbd". The "optional" part of the
> meson suite name determines that "check-block-nbd" is NOT added as a
> depedency of "check".  IOW, it doesn't get run by "make check", but
> must be optionally run through "make check-block-nbd" when needed.
>

Oh right.
Just to double check, will everything still run when using
make check-block SPEED=thorough, or is there some new behaviour on this
side?

> With regards,
> Daniel

Re: [PATCH 07/16] scripts/mtest2make: support optional tests grouping

[Daniel P. Berrangé]

On Wed, May 13, 2026 at 10:23:07AM -0700, Pierrick Bouvier wrote:
> On 5/13/2026 10:15 AM, Daniel P. Berrangé wrote:
> > On Wed, May 13, 2026 at 08:49:39AM -0700, Pierrick Bouvier wrote:
> >> On 5/13/2026 3:08 AM, Daniel P. Berrangé wrote:
> >>> On Tue, May 12, 2026 at 08:45:42AM -0700, Pierrick Bouvier wrote:
> >>>> On 4/24/2026 8:41 AM, Daniel P. Berrangé wrote:
> >>>>> Currently tests can be classified into three speed groups depending on
> >>>>> whether the meson suite name ends in '-slow' or '-thorough' or neither.
> >>>>>
> >>>>> This gets turned into make targets that match the name of the meson
> >>>>> suite, with the speed suffix stripped. e.g.
> >>>>>
> >>>>>  * suite=block ->  'make check-block'
> >>>>>  * suite=block-slow -> 'make check-block SPEED=slow'
> >>>>>  * suite=block-thorough -> 'make check-block SPEED=thorough'
> >>>>>
> >>>>> The set of tests under the "thorough" speed, however, can get rather
> >>>>> large and it would be useful to have a way to expose further make
> >>>>> targets for directly running a particular subset of tests.
> >>>>>
> >>>>> This needs a way to run a target without requiring the SPEED variable,
> >>>>> while also not having them enabled by default as if they were 'quick'
> >>>>> tests.
> >>>>>
> >>>>> This modifies mtest2make.py to support this idea by allowing for a new
> >>>>> suffix '-optional' on a suite. When this is present, a correspondingly
> >>>>> named make target will be created without the '-optional' suffix which
> >>>>> will never be run automatically.
> >>>>>
> >>>>> This is intended to be combined with use of other suites. For example,
> >>>>> a single NBD test might be added to two suites, 'block-thorough' and
> >>>>> 'block-nbd-optional'.
> >>>>>
> >>>>> This would allow running it as part of all the block tests with
> >>>>> 'make check-block SPEED=thorough', and as part of a standalone target
> >>>>> 'make check-block-nbd'.
> >>>>>
> >>>>> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> >>>>> ---
> >>>>>  scripts/mtest2make.py | 22 +++++++++++++++-------
> >>>>>  1 file changed, 15 insertions(+), 7 deletions(-)
> >>>>>
> >>>>> diff --git a/scripts/mtest2make.py b/scripts/mtest2make.py
> >>>>> index 915f02d600..383ea68b16 100644
> >>>>> --- a/scripts/mtest2make.py
> >>>>> +++ b/scripts/mtest2make.py
> >>>>> @@ -22,7 +22,7 @@ def names(self, base):
> >>>>>  print(r'''
> >>>>>  SPEED = quick
> >>>>>  
> >>>>> -.speed.quick = $(sort $(filter-out %-slow %-thorough, $1))
> >>>>> +.speed.quick = $(sort $(filter-out %-slow %-thorough %-optional, $1))
> >>>>>  .speed.slow = $(sort $(filter-out %-thorough, $1))
> >>>>>  .speed.thorough = $(sort $1)
> >>>>>  
> >>>>> @@ -66,10 +66,15 @@ def process_tests(test, targets, suites):
> >>>>>              s = s[:-9]
> >>>>>              suites[s].speeds.add('thorough')
> >>>>>  
> >>>>> +def target_name(suite):
> >>>>> +    if suite.endswith('-optional'):
> >>>>> +        return suite[0:-9]
> >>>>> +    return suite
> >>>>> +
> >>>>>  def emit_prolog(suites, prefix):
> >>>>> -    all_targets = ' '.join((f'{prefix}-{k}'
> >>>>> +    all_targets = ' '.join((f'{prefix}-{target_name(k)}'
> >>>>>                              for k in sorted(suites.keys())))
> >>>>> -    all_xml = ' '.join((f'{prefix}-report-{k}.junit.xml'
> >>>>> +    all_xml = ' '.join((f'{prefix}-report-{target_name(k)}.junit.xml'
> >>>>>                          for k in sorted(suites.keys())))
> >>>>>      print()
> >>>>>      print(f'all-{prefix}-targets = {all_targets}')
> >>>>> @@ -83,14 +88,17 @@ def emit_prolog(suites, prefix):
> >>>>>      print(f'\t$(MAKE) {prefix}$* MTESTARGS="$(MTESTARGS) --logbase {prefix}-report$*" && ln -f meson-logs/$@ .')
> >>>>>  
> >>>>>  def emit_suite(name, suite, prefix):
> >>>>> +    tgtname = target_name(name)
> >>>>>      deps = ' '.join(sorted(suite.deps))
> >>>>>      print()
> >>>>> -    print(f'.{prefix}-{name}.deps = {deps}')
> >>>>> -    print(f'.ninja-goals.check-build += $(.{prefix}-{name}.deps)')
> >>>>> +    print(f'.{prefix}-{tgtname}.deps = {deps}')
> >>>>> +    print(f'.ninja-goals.check-build += $(.{prefix}-{tgtname}.deps)')
> >>>>>  
> >>>>>      names = ' '.join(sorted(suite.names(name)))
> >>>>> -    targets = f'{prefix}-{name} {prefix}-report-{name}.junit.xml'
> >>>>> -    if not name.endswith('-slow') and not name.endswith('-thorough'):
> >>>>> +    targets = f'{prefix}-{tgtname} {prefix}-report-{tgtname}.junit.xml'
> >>>>> +    if not name.endswith('-slow') and \
> >>>>> +       not name.endswith('-thorough') and \
> >>>>> +       not name.endswith('-optional'):
> >>>>>          targets += f' {prefix} {prefix}-report.junit.xml'
> >>>>>      print(f'ifneq ($(filter {targets}, $(MAKECMDGOALS)),)')
> >>>>>      # for the "base" suite possibly add FOO-slow and FOO-thorough
> >>>>
> >>>> Reviewed-by: Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com>
> >>>>
> >>>> Not blocking this patch, thus reviewed, but I think it would be a better
> >>>> abstraction to use meson suites for this (a test can be added in
> >>>> multiple suites), instead of adding more clunk to makefile layer.
> >>>
> >>> This *is* using meson suites. The mtest2make script takes the list
> >>> of meson suites and generates makefile targets which can be used to
> >>> select execution of the meson suites. This change is exposing more
> >>> of the suites as makefile targets, since make is our primary entry
> >>> point.
> >>>
> >>>
> >>> With regards,
> >>> Daniel
> >>
> >> Maybe I can't read this patch correctly, but from it, and the message
> >> description, I understand that -optional is a pure Makefile abstraction,
> >> and does not reflect any real meson suite.
> > 
> > Actually its the other way around. In the next patch in this series
> > we create meson suites like  "block-nbd-optional" and that gets turned
> > into a makefile target  "check-block-nbd". The "optional" part of the
> > meson suite name determines that "check-block-nbd" is NOT added as a
> > depedency of "check".  IOW, it doesn't get run by "make check", but
> > must be optionally run through "make check-block-nbd" when needed.
> >
> 
> Oh right.
> Just to double check, will everything still run when using
> make check-block SPEED=thorough, or is there some new behaviour on this
> side?

Yes, this commit doesn't affect that existing make target, so it
operates as before.

Patch 5 does however fix a pre-existing problem - currently even
with SPEED=thorough most of the qcow2 tests didn't get run, only
the "quick" tests. With this series SPEED=thorough correctly does
what people thought it was already doing.

With regards,
Daniel
-- 
|: https://berrange.com       ~~        https://hachyderm.io/@berrange :|
|: https://libvirt.org          ~~          https://entangle-photo.org :|
|: https://pixelfed.art/berrange   ~~    https://fstop138.berrange.com :|

Re: [PATCH 07/16] scripts/mtest2make: support optional tests grouping

[Pierrick Bouvier]

On 5/13/2026 10:26 AM, Daniel P. Berrangé wrote:
> On Wed, May 13, 2026 at 10:23:07AM -0700, Pierrick Bouvier wrote:
>> On 5/13/2026 10:15 AM, Daniel P. Berrangé wrote:
>>> On Wed, May 13, 2026 at 08:49:39AM -0700, Pierrick Bouvier wrote:
>>>> On 5/13/2026 3:08 AM, Daniel P. Berrangé wrote:
>>>>> On Tue, May 12, 2026 at 08:45:42AM -0700, Pierrick Bouvier wrote:
>>>>>> On 4/24/2026 8:41 AM, Daniel P. Berrangé wrote:
>>>>>>> Currently tests can be classified into three speed groups depending on
>>>>>>> whether the meson suite name ends in '-slow' or '-thorough' or neither.
>>>>>>>
>>>>>>> This gets turned into make targets that match the name of the meson
>>>>>>> suite, with the speed suffix stripped. e.g.
>>>>>>>
>>>>>>>  * suite=block ->  'make check-block'
>>>>>>>  * suite=block-slow -> 'make check-block SPEED=slow'
>>>>>>>  * suite=block-thorough -> 'make check-block SPEED=thorough'
>>>>>>>
>>>>>>> The set of tests under the "thorough" speed, however, can get rather
>>>>>>> large and it would be useful to have a way to expose further make
>>>>>>> targets for directly running a particular subset of tests.
>>>>>>>
>>>>>>> This needs a way to run a target without requiring the SPEED variable,
>>>>>>> while also not having them enabled by default as if they were 'quick'
>>>>>>> tests.
>>>>>>>
>>>>>>> This modifies mtest2make.py to support this idea by allowing for a new
>>>>>>> suffix '-optional' on a suite. When this is present, a correspondingly
>>>>>>> named make target will be created without the '-optional' suffix which
>>>>>>> will never be run automatically.
>>>>>>>
>>>>>>> This is intended to be combined with use of other suites. For example,
>>>>>>> a single NBD test might be added to two suites, 'block-thorough' and
>>>>>>> 'block-nbd-optional'.
>>>>>>>
>>>>>>> This would allow running it as part of all the block tests with
>>>>>>> 'make check-block SPEED=thorough', and as part of a standalone target
>>>>>>> 'make check-block-nbd'.
>>>>>>>
>>>>>>> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
>>>>>>> ---
>>>>>>>  scripts/mtest2make.py | 22 +++++++++++++++-------
>>>>>>>  1 file changed, 15 insertions(+), 7 deletions(-)
>>>>>>>
>>>>>>> diff --git a/scripts/mtest2make.py b/scripts/mtest2make.py
>>>>>>> index 915f02d600..383ea68b16 100644
>>>>>>> --- a/scripts/mtest2make.py
>>>>>>> +++ b/scripts/mtest2make.py
>>>>>>> @@ -22,7 +22,7 @@ def names(self, base):
>>>>>>>  print(r'''
>>>>>>>  SPEED = quick
>>>>>>>  
>>>>>>> -.speed.quick = $(sort $(filter-out %-slow %-thorough, $1))
>>>>>>> +.speed.quick = $(sort $(filter-out %-slow %-thorough %-optional, $1))
>>>>>>>  .speed.slow = $(sort $(filter-out %-thorough, $1))
>>>>>>>  .speed.thorough = $(sort $1)
>>>>>>>  
>>>>>>> @@ -66,10 +66,15 @@ def process_tests(test, targets, suites):
>>>>>>>              s = s[:-9]
>>>>>>>              suites[s].speeds.add('thorough')
>>>>>>>  
>>>>>>> +def target_name(suite):
>>>>>>> +    if suite.endswith('-optional'):
>>>>>>> +        return suite[0:-9]
>>>>>>> +    return suite
>>>>>>> +
>>>>>>>  def emit_prolog(suites, prefix):
>>>>>>> -    all_targets = ' '.join((f'{prefix}-{k}'
>>>>>>> +    all_targets = ' '.join((f'{prefix}-{target_name(k)}'
>>>>>>>                              for k in sorted(suites.keys())))
>>>>>>> -    all_xml = ' '.join((f'{prefix}-report-{k}.junit.xml'
>>>>>>> +    all_xml = ' '.join((f'{prefix}-report-{target_name(k)}.junit.xml'
>>>>>>>                          for k in sorted(suites.keys())))
>>>>>>>      print()
>>>>>>>      print(f'all-{prefix}-targets = {all_targets}')
>>>>>>> @@ -83,14 +88,17 @@ def emit_prolog(suites, prefix):
>>>>>>>      print(f'\t$(MAKE) {prefix}$* MTESTARGS="$(MTESTARGS) --logbase {prefix}-report$*" && ln -f meson-logs/$@ .')
>>>>>>>  
>>>>>>>  def emit_suite(name, suite, prefix):
>>>>>>> +    tgtname = target_name(name)
>>>>>>>      deps = ' '.join(sorted(suite.deps))
>>>>>>>      print()
>>>>>>> -    print(f'.{prefix}-{name}.deps = {deps}')
>>>>>>> -    print(f'.ninja-goals.check-build += $(.{prefix}-{name}.deps)')
>>>>>>> +    print(f'.{prefix}-{tgtname}.deps = {deps}')
>>>>>>> +    print(f'.ninja-goals.check-build += $(.{prefix}-{tgtname}.deps)')
>>>>>>>  
>>>>>>>      names = ' '.join(sorted(suite.names(name)))
>>>>>>> -    targets = f'{prefix}-{name} {prefix}-report-{name}.junit.xml'
>>>>>>> -    if not name.endswith('-slow') and not name.endswith('-thorough'):
>>>>>>> +    targets = f'{prefix}-{tgtname} {prefix}-report-{tgtname}.junit.xml'
>>>>>>> +    if not name.endswith('-slow') and \
>>>>>>> +       not name.endswith('-thorough') and \
>>>>>>> +       not name.endswith('-optional'):
>>>>>>>          targets += f' {prefix} {prefix}-report.junit.xml'
>>>>>>>      print(f'ifneq ($(filter {targets}, $(MAKECMDGOALS)),)')
>>>>>>>      # for the "base" suite possibly add FOO-slow and FOO-thorough
>>>>>>
>>>>>> Reviewed-by: Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com>
>>>>>>
>>>>>> Not blocking this patch, thus reviewed, but I think it would be a better
>>>>>> abstraction to use meson suites for this (a test can be added in
>>>>>> multiple suites), instead of adding more clunk to makefile layer.
>>>>>
>>>>> This *is* using meson suites. The mtest2make script takes the list
>>>>> of meson suites and generates makefile targets which can be used to
>>>>> select execution of the meson suites. This change is exposing more
>>>>> of the suites as makefile targets, since make is our primary entry
>>>>> point.
>>>>>
>>>>>
>>>>> With regards,
>>>>> Daniel
>>>>
>>>> Maybe I can't read this patch correctly, but from it, and the message
>>>> description, I understand that -optional is a pure Makefile abstraction,
>>>> and does not reflect any real meson suite.
>>>
>>> Actually its the other way around. In the next patch in this series
>>> we create meson suites like  "block-nbd-optional" and that gets turned
>>> into a makefile target  "check-block-nbd". The "optional" part of the
>>> meson suite name determines that "check-block-nbd" is NOT added as a
>>> depedency of "check".  IOW, it doesn't get run by "make check", but
>>> must be optionally run through "make check-block-nbd" when needed.
>>>
>>
>> Oh right.
>> Just to double check, will everything still run when using
>> make check-block SPEED=thorough, or is there some new behaviour on this
>> side?
> 
> Yes, this commit doesn't affect that existing make target, so it
> operates as before.
> 
> Patch 5 does however fix a pre-existing problem - currently even
> with SPEED=thorough most of the qcow2 tests didn't get run, only
> the "quick" tests. With this series SPEED=thorough correctly does
> what people thought it was already doing.
>

Sounds good, thanks Daniel.

> With regards,
> Daniel