From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1694620406; x=1695225206; darn=redhat.com; h=content-transfer-encoding:mime-version:message-id:in-reply-to:date :subject:cc:to:from:user-agent:references:from:to:cc:subject:date :message-id:reply-to; bh=KbpgIZY0S+jrnQeTU9MHb5yuccFEI2qq5nwURR5Z7vY=; b=dMUDIZQhfw8AKFV8SsuGFg21LZVt1LSSqDZJWNAcla2EEkNJpn0ZHHA8oiMEChMdzz zSav/zHphflE61NnAwUwzgISvjiU6WHk4hMMKNzP3rqJMNcQTiCxDkFCPVBqjz/DnDl9 3vBeOKSpSUWJdJOnNFd55WIBCktMe+HTUSNf7bfHCtIZOywQ19auYZGo1W61sHnNrfsz VSMZOEixcQ1UDGAK+nVAqPSetQVljAP2KvmFhGBov+2bVjMvsGFOtZ9jb1jsSCubUUvl eoRwBj43fcm85ezISVpx2zF9fsHijxbcrsJNHOe1bbXRxTY/gpLu9OhBWp72kOIVbR3P jJww== References: From: Alex =?utf-8?Q?Benn=C3=A9e?= Date: Wed, 13 Sep 2023 16:38:12 +0100 In-reply-to: Message-ID: <87msxqrrwr.fsf@linaro.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Subject: Re: [Virtio-fs] [BUG] virtio-fs: Corruption when running binaries from virtiofsd-backed fs List-Id: Development discussions about virtio-fs List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Erik Schilling Cc: virtualization@lists.linux-foundation.org, linux-fsdevel@vger.kernel.org, virtio-fs@redhat.com, Richard Henderson , Weiwei Li , Vivek Goyal , Stefan Hajnoczi , Miklos Szeredi , Manos Pitsidianakis , Viresh Kumar , linux-kernel@vger.kernel.org, qemu-devel@nongnu.org, German Maglione , Hanna Czenczek , Philippe =?utf-8?Q?Mathieu-Daud=C3=A9?= "Erik Schilling" writes: > CCing a few more people as suggested by stefanha on #qemu. (Add philmd who's tracking the MIPs failure) > On Wed Sep 13, 2023 at 8:18 AM CEST, Erik Schilling wrote: >> On Fri Sep 1, 2023 at 12:37 PM CEST, Erik Schilling wrote: >> > On Wed Aug 30, 2023 at 10:20 AM CEST, Erik Schilling wrote: >> > > Hi all! >> > > >> > > Some days ago I posted to #virtiofs:matrix.org, describing that I am >> > > observing what looks like a corruption when executing programs from = a >> > > virtiofs-based filesystem. >> > > >> > > Over the last few days I spent more time drilling into the problem. >> > > This is an attempt at summarizing my findings in order to see what o= ther >> > > people think about this. >> > > >> > > When running binaries mounted from virtiofs they may either: fail wi= th a >> > > segfault, fail with badaddr, get stuck or - sometimes - succeed. >> > > >> > > Environment: >> > > Host: Fedora 38 running 6.4.11-200.fc38.x86_64 >> > > Guest: Yocto-based image: 6.4.9-yocto-standard, aarch64 >> > > virtiofsd: latest main + some debug prints [1] >> > > QEMU: built from recent git [2] >> > > >> > > virtiofsd invocation: >> > > RUST_LOG=3D"debug" ./virtiofsd --seccomp=3Dnone --sandbox=3Dnone \ >> > > --socket-path "fs.sock0" --shared-dir $PWD/share-dir/ --cache=3D= never >> > > >> > > QEMU invocation: >> > > ~/projects/qemu/build/qemu-system-aarch64 -kernel Image -machine v= irt \ >> > > -cpu cortex-a57 \ >> > > -serial mon:stdio \ >> > > -device virtio-net-pci,netdev=3Dnet0 \ >> > > -netdev user,id=3Dnet0,hostfwd=3Dtcp::2223-:22 \ >> > > -display none -m 2048 -smp 4 \ >> > > -object memory-backend-memfd,id=3Dmem,size=3D2048M,share=3Don \ >> > > -numa node,memdev=3Dmem \ >> > > -hda trs-overlay-guest.qcow2 \ >> > > -chardev socket,id=3Dchar0,path=3D"fs.sock0" \ >> > > -device vhost-user-fs-pci,queue-size=3D1024,chardev=3Dchar0,tag= =3D/dev/root \ >> > > -append 'root=3D/dev/vda2 ro log_buf_len=3D8M' >> > > >> > > I figured that launching virtiofsd with --cache=3Dalways masks the >> > > problem. Therefore, I set --cache=3Dnever, but I think I observed no >> > > difference compared to the default setting (auto). >> > > >> > > Adding logging to virtiofsd and kernel _feeled_ like it made the pro= blem >> > > harder to reproduce - leaving me with the impression that some race = is >> > > happening on somewhere. >> > > >> > > Trying to rule out that virtiofsd is returning corrupted data, I add= ed >> > > some logging and hashsum calculation hacks to it [1]. The hashes che= ck >> > > out across multiple accesses and the order and kind of queued messag= es >> > > is exactly the same in both the error case and crash case. fio was a= lso >> > > unable to find any errors with a naive job description [3]. >> > > >> > > Next, I tried to capture info on the guest side. This became a bit >> > > tricky since the crashes became pretty rare once I followed a fixed >> > > pattern of starting log capture, running perf and trying to reproduc= e >> > > the problem. Ultimately, I had the most consistent results with >> > > immediately running a program twice: >> > > >> > > /mnt/ld-linux-aarch64.so.1 /mnt/ls.coreutils /; \ >> > > /mnt/ld-linux-aarch64.so.1 /mnt/ls.coreutils / >> > > >> > > (/mnt being the virtiofs mount) >> > > >> > > For collecting logs, I made a hack to the guest kernel in order to d= ump >> > > the page content after receiving the virtiofs responses [4]. Reprodu= cing >> > > the problem with this, leaves me with logs that seem to suggest that >> > > virtiofsd is returning identical content, but the guest kernel seems= to >> > > receive differing pages: >> > > >> > > good-kernel [5]: >> > > kernel: virtio_fs_wake_pending_and_unlock: opcode 3 unique 0x312 n= odeid 0x1 in.len 56 out.len 104 >> > > kernel: virtiofs virtio1: virtio_fs_vq_done requests.0 >> > > kernel: virtio_fs_wake_pending_and_unlock: opcode 1 unique 0x314 n= odeid 0x1 in.len 53 out.len 128 >> > > kernel: virtiofs virtio1: virtio_fs_vq_done requests.0 >> > > kernel: virtio_fs_wake_pending_and_unlock: opcode 3 unique 0x316 n= odeid 0x29 in.len 56 out.len 104 >> > > kernel: virtiofs virtio1: virtio_fs_vq_done requests.0 >> > > kernel: virtio_fs_wake_pending_and_unlock: opcode 14 unique 0x318 = nodeid 0x29 in.len 48 out.len 16 >> > > kernel: virtiofs virtio1: virtio_fs_vq_done requests.0 >> > > kernel: virtio_fs_wake_pending_and_unlock: opcode 15 unique 0x31a = nodeid 0x29 in.len 80 out.len 832 >> > > kernel: virtiofs virtio1: virtio_fs_vq_done requests.0 >> > > kernel: virtio_fs: page: 000000006996d520 >> > > kernel: virtio_fs: to: 00000000de590c14 >> > > kernel: virtio_fs rsp:00 00 00 00 00 00 00 00 00 00 00 00 00 00 00= 00 ................ >> > > kernel: virtio_fs rsp:00 00 00 00 00 00 00 00 00 00 00 00 00 00 00= 00 ................ >> > > kernel: virtio_fs rsp:00 00 00 00 00 00 00 00 00 00 00 00 00 00 00= 00 ................ >> > > kernel: virtio_fs rsp:00 00 00 00 00 00 00 00 00 00 00 00 00 00 00= 00 ................ >> > > [...] >> > > >> > > bad-kernel [6]: >> > > kernel: virtio_fs_wake_pending_and_unlock: opcode 3 unique 0x162 n= odeid 0x1 in.len 56 out.len 104 >> > > kernel: virtiofs virtio1: virtio_fs_vq_done requests.0 >> > > kernel: virtio_fs_wake_pending_and_unlock: opcode 1 unique 0x164 n= odeid 0x1 in.len 53 out.len 128 >> > > kernel: virtiofs virtio1: virtio_fs_vq_done requests.0 >> > > kernel: virtio_fs_wake_pending_and_unlock: opcode 3 unique 0x166 n= odeid 0x16 in.len 56 out.len 104 >> > > kernel: virtiofs virtio1: virtio_fs_vq_done requests.0 >> > > kernel: virtio_fs_wake_pending_and_unlock: opcode 14 unique 0x168 = nodeid 0x16 in.len 48 out.len 16 >> > > kernel: virtiofs virtio1: virtio_fs_vq_done requests.0 >> > > kernel: virtio_fs_wake_pending_and_unlock: opcode 15 unique 0x16a = nodeid 0x16 in.len 80 out.len 832 >> > > kernel: virtiofs virtio1: virtio_fs_vq_done requests.0 >> > > kernel: virtio_fs: page: 000000006ce9a559 >> > > kernel: virtio_fs: to: 000000007ae8b946 Are these the copying from virtio to buffer cache? I would assume they trigger -d trace:memory_notdirty_write_access if so. >> > > kernel: virtio_fs rsp:00 00 00 00 00 00 00 00 00 00 00 00 00 00 00= 00 ................ >> > > kernel: virtio_fs rsp:00 00 00 00 00 00 00 00 00 00 00 00 00 00 00= 00 ................ >> > > kernel: virtio_fs rsp:80 40 de c8 ff ff 00 00 cc 2b 62 ae ff ff 00= 00 .@.......+b..... >> > > kernel: virtio_fs rsp:02 4e de c8 ff ff 00 00 00 00 00 00 00 00 00= 00 .N.............. >> > > [...] >> > > >> > > When looking at the corresponding output from virtiofsd, it claims t= o >> > > have returned identical data: >> > > >> > > good-virtiofsd [7]: >> > > [DEBUG virtiofsd::server] Received request: opcode=3DRead (15), in= ode=3D41, unique=3D794, pid=3D481 >> > > [src/server.rs:618] r.read_obj().map_err(Error::DecodeMessage)? = =3D ReadIn { >> > > fh: 31, >> > > offset: 0, >> > > size: 832, >> > > read_flags: 2, >> > > lock_owner: 6838554705639967244, >> > > flags: 131072, >> > > padding: 0, >> > > } >> > > [src/file_traits.rs:161] hash =3D 2308490450751364994 >> > > [DEBUG virtiofsd::server] Replying OK, header: OutHeader { len: 84= 8, error: 0, unique: 794 } >> > > >> > > bad-virtiofsd [8]: >> > > [DEBUG virtiofsd::server] Received request: opcode=3DRead (15), in= ode=3D22, unique=3D362, pid=3D406 >> > > [src/server.rs:618] r.read_obj().map_err(Error::DecodeMessage)? = =3D ReadIn { >> > > fh: 12, >> > > offset: 0, >> > > size: 832, >> > > read_flags: 2, >> > > lock_owner: 6181120926258395554, >> > > flags: 131072, >> > > padding: 0, >> > > } >> > > [src/file_traits.rs:161] hash =3D 2308490450751364994 >> > > [DEBUG virtiofsd::server] Replying OK, header: OutHeader { len: 84= 8, error: 0, unique: 362 } >> > > >> > > The "corruption" only seems to happen in this one page, all other pa= ges >> > > are identical between runs (except that the bad run terminates earli= er). >> > > >> > > What do the experts think here? To me it feels a bit like some kind = of >> > > corruption is going on. Or am I misinterpreting things here? >> > > >> > > Which further analysis steps would you suggest? >> > > >> > > >> > > Further notes: >> > > >> > > After collecting the above results, I realized that running the gues= t >> > > with -smp 1 makes the problems a lot worse. So maybe that is a bette= r >> > > choice when trying to reproduce it. >> > > >> > > Repo with my scripts is available at: >> > > https://git.codelinaro.org/erik_schilling/jira-orko-65-bootstrap-k3s= -config/ >> > > >> > > The scripts are just quick and dirty implementations and are not >> > > particulary portable. >> > >> > Summary of my testing during the last few days: >> > >> > Testing with KCSAN revealed a few cases that look like missing READ_ON= CE >> > annotations (will send patches separately). But nothing of that was >> > related to the immediate problem. I tested instrument_read() and anoth= er >> > round of logging with a delay to virtio_fs_request_complete. It looks >> > like the buffer get corrupted before entering that function. KCSAN >> > or manual sleeps + prints did not show any corruption while in that >> > function. >> > >> > KASAN did not report any issues. >> > >> > Patching virtiofsd to do an additional copy and going through rust-vmm= 's >> > .copy_to() function did not change the behaviour. >> > >> > I will mostly be off next week, will continue analysis afterwards. Hap= py >> > to hear about suggestions of other things to try :). >> >> Back from a week of vacation... >> >> Summary of what was discussed on #virtiofs:matrix.org: >> >> The issue only seems to happen in QEMU TCG scenarios (I tested aarch64 >> and x86_64 on x86_64, wizzard on Matrix tested arm32). >> >> CCing qemu-devel. Maybe someone has some hints on where to focus the >> debugging efforts? >> >> I am trying to build a complex monster script of tracing the relevant >> addresses in order to figure out whether the guest or host does the >> writes. But I am happy to hear about more clever ideas :). > > After hearing about investigations of bugs in other virtio scenarios > that seem to be caused by QEMU [9], I tested some older QEMU versions. > > Indeed, a882b5712373171d3bd53cd82ddab4453ddef468 did not show the buggy > behaviour. So I did a bisect: > > git bisect start > # status: waiting for both good and bad commits > # good: [a882b5712373171d3bd53cd82ddab4453ddef468] Revert "virtio: > introduce macro IRTIO_CONFIG_IRQ_IDX" > git bisect good a882b5712373171d3bd53cd82ddab4453ddef468 > # status: waiting for bad commit, 1 good commit known > # bad: [9ef497755afc252fb8e060c9ea6b0987abfd20b6] Merge tag > 'pull-vfio-20230911' of https://github.com/legoater/qemu into staging > git bisect bad 9ef497755afc252fb8e060c9ea6b0987abfd20b6 > # skip: [3ba5fe46ea4456a16e2f47ab8e75943b54879c4e] Merge tag > 'mips-20221108' of https://github.com/philmd/qemu into staging > git bisect skip 3ba5fe46ea4456a16e2f47ab8e75943b54879c4e > # skip: [ade760a2f63804b7ab1839fbc3e5ddbf30538718] Merge tag > 'pull-request-2022-11-08' of https://gitlab.com/thuth/qemu into > staging > git bisect skip ade760a2f63804b7ab1839fbc3e5ddbf30538718 > # good: [ad2ca2e3f762b0cb98eb976002569795b270aef1] target/xtensa: Dro= p tcg_temp_free > git bisect good ad2ca2e3f762b0cb98eb976002569795b270aef1 > # bad: [19a720b74fde7e859d19f12c66a72e545947a657] Merge tag > 'tracing-pull-request' of https://gitlab.com/stefanha/qemu into > staging > git bisect bad 19a720b74fde7e859d19f12c66a72e545947a657 > # bad: [29d9efca16080211f107b540f04d1ed3c12c63b0] arm/Kconfig: Do > not build TCG-only boards on a KVM-only build > git bisect bad 29d9efca16080211f107b540f04d1ed3c12c63b0 > # good: [9636e513255362c4a329e3e5fb2c97dab3c5ce47] Merge tag > 'misc-next-pull-request' of https://gitlab.com/berrange/qemu into > staging > git bisect good 9636e513255362c4a329e3e5fb2c97dab3c5ce47 > # bad: [45608654aa63ca2b311d6cb761e1522f2128e00e] Merge tag > 'pull-tpm-2023-04-20-1' of https://github.com/stefanberger/qemu-tpm > into staging > git bisect bad 45608654aa63ca2b311d6cb761e1522f2128e00e > # good: [1ff4a81bd3efb207992f1da267886fe0c4df764f] tcg: use QTree ins= tead of GTree > git bisect good 1ff4a81bd3efb207992f1da267886fe0c4df764f > # bad: [9ed98cae151368cc89c4bb77c9f325f7185e8f09] block-backend: > ignore inserted state in blk_co_nb_sectors > git bisect bad 9ed98cae151368cc89c4bb77c9f325f7185e8f09 > # good: [c8cb603293fd329f2a62ade76ec9de3f462fc5c3] tests/avocado: Tes= t Xen guest support under KVM > git bisect good c8cb603293fd329f2a62ade76ec9de3f462fc5c3 > # bad: [64f1c63d87208e28e8e38c4ab514ada1728960ef] Merge tag > 'pull_error_handle_fix_use_after_free.v1' of > https://github.com/stefanberger/qemu-tpm into staging > git bisect bad 64f1c63d87208e28e8e38c4ab514ada1728960ef > # good: [8a712df4d4d736b7fe6441626677bfd271d95b15] Merge tag > 'pull-for-8.0-040423-2' of https://gitlab.com/stsquad/qemu into > staging > git bisect good 8a712df4d4d736b7fe6441626677bfd271d95b15 > # bad: [7d0334e49111787ae19fbc8d29ff6e7347f0605e] Merge tag > 'pull-tcg-20230404' of https://gitlab.com/rth7680/qemu into staging > git bisect bad 7d0334e49111787ae19fbc8d29ff6e7347f0605e > # bad: [3371802fba3f7be4465f8a5e5777d43d556676ef] accel/tcg: Fix jump= cache set in cpu_exec_loop > git bisect bad 3371802fba3f7be4465f8a5e5777d43d556676ef > # good: [6cda41daa2162b8e1048124655ba02a8c2b762b4] Revert > "linux-user/arm: Take more care allocating commpage" > git bisect good 6cda41daa2162b8e1048124655ba02a8c2b762b4 > # skip: [c83574392e0af108a643347712564f6749906413] accel/tcg: Fix ove= rwrite problems of tcg_cflags > git bisect skip c83574392e0af108a643347712564f6749906413 > # only skipped commits left to test > # possible first bad commit: > [3371802fba3f7be4465f8a5e5777d43d556676ef] accel/tcg: Fix jump cache > set in cpu_exec_loop It should be possible to rule out the jump cache in HEAD by patching out the two: if (likely(tb && tb->pc =3D=3D pc && tb->cs_base =3D=3D cs_base && tb->flags =3D=3D flags && tb_cflags(tb) =3D=3D cflags)) { return tb; } functions to if(0) { return tb } in tb_lookup. That can rule out if a corrupted jump cache is responsible (at the cost of running a bit slower). Apropos of earlier tb_jmp_cache_inval_tb() is responsible for removing entries of TB's that are invalidated for whatever reason from the cache. > # possible first bad commit: > [c83574392e0af108a643347712564f6749906413] accel/tcg: Fix overwrite > problems of tcg_cflags > > I had an inclusive test in the end where c83574392e did not yield in me > being able to start the VM. It's interesting that is involved the PCREL stuff. This is a relatively recent optimisation that allows us to re-use translations for physical pages that are mapped into virtual memory multiple times. To do that the target translator has to support CF_PCREL and translate only knowing the PC as an offset into the page. > > Whether one of these contains a bug or whether only new behaviour of > QEMU revealed a bug somewhere else is of course still to be figured out. > > [9] https://gitlab.com/qemu-project/qemu/-/issues/1866 This is follow up on #1826, #1834, #1846 which is all broadly to do with how QEMU deals with updates to it's memory maps (e.g. when PCI devices are remapped/reconfigured). We are unsure if the MIPS architecture should be executing some sort of barrier around reconfiguration that can ensure we exit the block or we have to detect mid-block IO accesses icount style. > > - Erik > >> >> - Erik >> >> > >> > Good weekend, >> > >> > - Erik >> > >> > >> > > >> > > - Erik >> > > >> > > [1] https://gitlab.com/ablu/virtiofsd/-/commit/18fd0c1849e15bc55fbdd= 6e1f169801b2b03da1f >> > > [2] https://gitlab.com/qemu-project/qemu/-/commit/50e7a40af372ee5931= c99ef7390f5d3d6fbf6ec4 >> > > [3] >> > > https://git.codelinaro.org/erik_schilling/jira-orko-65-bootstrap-k3s= -config/-/blob/397a6310dea35973025e3d61f46090bf0c092762/share-dir/write-and= -verify-mmap.fio >> > > [4] https://github.com/Ablu/linux/commit/3880b9f8affb01aeabb0a04fe76= ad7701dc0bb95 >> > > [5] Line 12923: >> > > https://git.codelinaro.org/erik_schilling/jira-orko-65-bootstrap-k3s= -config/-/blob/main/logs/2023-08-29%2013%3A42%3A35%2B02%3A00/good-drop-bad-= 1.txt >> > > [6] Line 12923: >> > > https://git.codelinaro.org/erik_schilling/jira-orko-65-bootstrap-k3s= -config/-/blob/main/logs/2023-08-29%2013%3A42%3A35%2B02%3A00/good-bad-1.txt >> > > [7] >> > > https://git.codelinaro.org/erik_schilling/jira-orko-65-bootstrap-k3s= -config/-/blob/main/logs/2023-08-29%2013%3A42%3A35%2B02%3A00/virtiofsd.txt#= L2538-2549 >> > > [8] >> > > https://git.codelinaro.org/erik_schilling/jira-orko-65-bootstrap-k3s= -config/-/blob/main/logs/2023-08-29%2013%3A42%3A35%2B02%3A00/virtiofsd.txt#= L1052-1063 --=20 Alex Benn=C3=A9e Virtualisation Tech Lead @ Linaro From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 033FAEDEC7C for ; Wed, 13 Sep 2023 15:53:33 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 99263600CC; Wed, 13 Sep 2023 15:53:33 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org 99263600CC Authentication-Results: smtp3.osuosl.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.a=rsa-sha256 header.s=google header.b=TxUhHZ9O X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id j8Sq0OvWhDMW; Wed, 13 Sep 2023 15:53:32 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp3.osuosl.org (Postfix) with ESMTPS id 8220C605AF; Wed, 13 Sep 2023 15:53:31 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org 8220C605AF Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 5AEFEC0071; Wed, 13 Sep 2023 15:53:31 +0000 (UTC) Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 38B9FC0032 for ; Wed, 13 Sep 2023 15:53:30 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id F2B85416A8 for ; Wed, 13 Sep 2023 15:53:29 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org F2B85416A8 Authentication-Results: smtp2.osuosl.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.a=rsa-sha256 header.s=google header.b=TxUhHZ9O X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id j-Wtt6A_TpyO for ; Wed, 13 Sep 2023 15:53:28 +0000 (UTC) Received: from mail-wm1-x332.google.com (mail-wm1-x332.google.com [IPv6:2a00:1450:4864:20::332]) by smtp2.osuosl.org (Postfix) with ESMTPS id D66AF405F8 for ; Wed, 13 Sep 2023 15:53:27 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org D66AF405F8 Received: by mail-wm1-x332.google.com with SMTP id 5b1f17b1804b1-4018af103bcso81545e9.1 for ; Wed, 13 Sep 2023 08:53:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1694620406; x=1695225206; darn=lists.linux-foundation.org; h=content-transfer-encoding:mime-version:message-id:in-reply-to:date :subject:cc:to:from:user-agent:references:from:to:cc:subject:date :message-id:reply-to; bh=KbpgIZY0S+jrnQeTU9MHb5yuccFEI2qq5nwURR5Z7vY=; b=TxUhHZ9O8vimXOQp3kO80nkgQgWmzmnvsDJpNvkuVmzu2ETAbUWe89dFQog2RJTED8 z9yEcHUkZJMxXsPPEzon9UmZuVAzuPS3RVs//HxzZdq8oEzJdKZUpeK41v/jN+GAstfC HQ6knDGYc2tr65PvSMI7+vZul81PmQqJixEcVhzBpImjs8iUVprV1ZLiAR4FfFS1XuKR TtPlJ0C46LL5+76PPuVCW0WhO+sUDvmvz4NS4z1cA6EqKAK5+chQenrJ5lZQuTt+SNSF GQeZgimdhxBizXiUmxJpkH17yqT8TvOK1XfFkWAWjroflfSQhOX8Gdk9uXBU/m3t203r 3Tfg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694620406; x=1695225206; h=content-transfer-encoding:mime-version:message-id:in-reply-to:date :subject:cc:to:from:user-agent:references:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=KbpgIZY0S+jrnQeTU9MHb5yuccFEI2qq5nwURR5Z7vY=; b=xMxlayrsWt3ZUpxp/N0VizwvW8eXZhkb8M9s4m+SQ9kPX4oKYPSYCHGyg5fcC1kSk1 J0jBKh0pKCxE22X3rgXL3faaROaBSWIVs8VIYuHiR+XVPNX2snRL4Zektu97zX/X4wVR 6urKD9prq/Av08ZZq8zjodwAtn/CC28XnBZjUGm1v6KP/KqvneVbZem3BgWToWKdC7Mv QF6kdVUN6fJhr8dLKjFJDigOj5M/1xcKvxmzcRavu8KpYkoO0I6JwG8yAjnKRtmwLzxj 5PW1xVJmXtmuQP/Cy8zLOP04kdbR4aTQHC8o4q75H2d+szUkMCF2GWqkUbYa1jtnuDEG orVQ== X-Gm-Message-State: AOJu0YwT8gDJUewovInVtrx2FC3UP0ok1vpN/c7mBw+EXoloSIwrRVpA 7tiGeQ0a9n3+0S2G60CLKdRk9w== X-Google-Smtp-Source: AGHT+IG7uN/b0G0olfsBzT5JVx50eogSPAxxETPGl3o5DM9I5OGZFQgctwuNG29LxQ5Cb8CEXzO01g== X-Received: by 2002:a05:600c:44c9:b0:3ff:a95b:9751 with SMTP id f9-20020a05600c44c900b003ffa95b9751mr4604899wmo.7.1694620405692; Wed, 13 Sep 2023 08:53:25 -0700 (PDT) Received: from zen.linaroharston ([85.9.250.243]) by smtp.gmail.com with ESMTPSA id z20-20020a7bc7d4000000b003feae747ff2sm2400657wmk.35.2023.09.13.08.53.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Sep 2023 08:53:24 -0700 (PDT) Received: from zen (localhost [127.0.0.1]) by zen.linaroharston (Postfix) with ESMTP id 496A41FFBB; Wed, 13 Sep 2023 16:53:24 +0100 (BST) References: User-agent: mu4e 1.11.17; emacs 29.1.50 From: Alex =?utf-8?Q?Benn=C3=A9e?= To: Erik Schilling Subject: Re: [BUG] virtio-fs: Corruption when running binaries from virtiofsd-backed fs Date: Wed, 13 Sep 2023 16:38:12 +0100 In-reply-to: Message-ID: <87msxqrrwr.fsf@linaro.org> MIME-Version: 1.0 Cc: Manos Pitsidianakis , Philippe =?utf-8?Q?Mathieu-Daud=C3=A9?= , Miklos Szeredi , qemu-devel@nongnu.org, Viresh Kumar , Weiwei Li , Richard Henderson , linux-kernel@vger.kernel.org, virtualization@lists.linux-foundation.org, virtio-fs@redhat.com, German Maglione , Hanna Czenczek , Stefan Hajnoczi , linux-fsdevel@vger.kernel.org, Vivek Goyal X-BeenThere: virtualization@lists.linux-foundation.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Linux virtualization List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Errors-To: virtualization-bounces@lists.linux-foundation.org Sender: "Virtualization" CiJFcmlrIFNjaGlsbGluZyIgPGVyaWsuc2NoaWxsaW5nQGxpbmFyby5vcmc+IHdyaXRlczoKCj4g Q0NpbmcgYSBmZXcgbW9yZSBwZW9wbGUgYXMgc3VnZ2VzdGVkIGJ5IHN0ZWZhbmhhIG9uICNxZW11 LgoKKEFkZCBwaGlsbWQgd2hvJ3MgdHJhY2tpbmcgdGhlIE1JUHMgZmFpbHVyZSkKCj4gT24gV2Vk IFNlcCAxMywgMjAyMyBhdCA4OjE4IEFNIENFU1QsIEVyaWsgU2NoaWxsaW5nIHdyb3RlOgo+PiBP biBGcmkgU2VwIDEsIDIwMjMgYXQgMTI6MzcgUE0gQ0VTVCwgRXJpayBTY2hpbGxpbmcgd3JvdGU6 Cj4+ID4gT24gV2VkIEF1ZyAzMCwgMjAyMyBhdCAxMDoyMCBBTSBDRVNULCBFcmlrIFNjaGlsbGlu ZyB3cm90ZToKPj4gPiA+IEhpIGFsbCEKPj4gPiA+Cj4+ID4gPiBTb21lIGRheXMgYWdvIEkgcG9z dGVkIHRvICN2aXJ0aW9mczptYXRyaXgub3JnLCBkZXNjcmliaW5nIHRoYXQgSSBhbQo+PiA+ID4g b2JzZXJ2aW5nIHdoYXQgbG9va3MgbGlrZSBhIGNvcnJ1cHRpb24gd2hlbiBleGVjdXRpbmcgcHJv Z3JhbXMgZnJvbSBhCj4+ID4gPiB2aXJ0aW9mcy1iYXNlZCBmaWxlc3lzdGVtLgo+PiA+ID4KPj4g PiA+IE92ZXIgdGhlIGxhc3QgZmV3IGRheXMgSSBzcGVudCBtb3JlIHRpbWUgZHJpbGxpbmcgaW50 byB0aGUgcHJvYmxlbS4KPj4gPiA+IFRoaXMgaXMgYW4gYXR0ZW1wdCBhdCBzdW1tYXJpemluZyBt eSBmaW5kaW5ncyBpbiBvcmRlciB0byBzZWUgd2hhdCBvdGhlcgo+PiA+ID4gcGVvcGxlIHRoaW5r IGFib3V0IHRoaXMuCj4+ID4gPgo+PiA+ID4gV2hlbiBydW5uaW5nIGJpbmFyaWVzIG1vdW50ZWQg ZnJvbSB2aXJ0aW9mcyB0aGV5IG1heSBlaXRoZXI6IGZhaWwgd2l0aCBhCj4+ID4gPiBzZWdmYXVs dCwgZmFpbCB3aXRoIGJhZGFkZHIsIGdldCBzdHVjayBvciAtIHNvbWV0aW1lcyAtIHN1Y2NlZWQu Cj4+ID4gPgo+PiA+ID4gRW52aXJvbm1lbnQ6Cj4+ID4gPiAgIEhvc3Q6IEZlZG9yYSAzOCBydW5u aW5nIDYuNC4xMS0yMDAuZmMzOC54ODZfNjQKPj4gPiA+ICAgR3Vlc3Q6IFlvY3RvLWJhc2VkIGlt YWdlOiA2LjQuOS15b2N0by1zdGFuZGFyZCwgYWFyY2g2NAo+PiA+ID4gICB2aXJ0aW9mc2Q6IGxh dGVzdCBtYWluICsgc29tZSBkZWJ1ZyBwcmludHMgWzFdCj4+ID4gPiAgIFFFTVU6IGJ1aWx0IGZy b20gcmVjZW50IGdpdCBbMl0KPj4gPiA+Cj4+ID4gPiB2aXJ0aW9mc2QgaW52b2NhdGlvbjoKPj4g PiA+ICAgUlVTVF9MT0c9ImRlYnVnIiAuL3ZpcnRpb2ZzZCAtLXNlY2NvbXA9bm9uZSAtLXNhbmRi b3g9bm9uZSBcCj4+ID4gPiAgICAgLS1zb2NrZXQtcGF0aCAiZnMuc29jazAiIC0tc2hhcmVkLWRp ciAkUFdEL3NoYXJlLWRpci8gLS1jYWNoZT1uZXZlcgo+PiA+ID4KPj4gPiA+IFFFTVUgaW52b2Nh dGlvbjoKPj4gPiA+ICAgfi9wcm9qZWN0cy9xZW11L2J1aWxkL3FlbXUtc3lzdGVtLWFhcmNoNjQg LWtlcm5lbCBJbWFnZSAtbWFjaGluZSB2aXJ0IFwKPj4gPiA+ICAgICAtY3B1IGNvcnRleC1hNTcg XAo+PiA+ID4gICAgIC1zZXJpYWwgbW9uOnN0ZGlvIFwKPj4gPiA+ICAgICAtZGV2aWNlIHZpcnRp by1uZXQtcGNpLG5ldGRldj1uZXQwIFwKPj4gPiA+ICAgICAtbmV0ZGV2IHVzZXIsaWQ9bmV0MCxo b3N0ZndkPXRjcDo6MjIyMy06MjIgXAo+PiA+ID4gICAgIC1kaXNwbGF5IG5vbmUgLW0gMjA0OCAt c21wIDQgXAo+PiA+ID4gICAgIC1vYmplY3QgbWVtb3J5LWJhY2tlbmQtbWVtZmQsaWQ9bWVtLHNp emU9MjA0OE0sc2hhcmU9b24gXAo+PiA+ID4gICAgIC1udW1hIG5vZGUsbWVtZGV2PW1lbSBcCj4+ ID4gPiAgICAgLWhkYSB0cnMtb3ZlcmxheS1ndWVzdC5xY293MiBcCj4+ID4gPiAgICAgLWNoYXJk ZXYgc29ja2V0LGlkPWNoYXIwLHBhdGg9ImZzLnNvY2swIiBcCj4+ID4gPiAgICAgLWRldmljZSB2 aG9zdC11c2VyLWZzLXBjaSxxdWV1ZS1zaXplPTEwMjQsY2hhcmRldj1jaGFyMCx0YWc9L2Rldi9y b290IFwKPj4gPiA+ICAgICAtYXBwZW5kICdyb290PS9kZXYvdmRhMiBybyBsb2dfYnVmX2xlbj04 TScKPj4gPiA+Cj4+ID4gPiBJIGZpZ3VyZWQgdGhhdCBsYXVuY2hpbmcgdmlydGlvZnNkIHdpdGgg LS1jYWNoZT1hbHdheXMgbWFza3MgdGhlCj4+ID4gPiBwcm9ibGVtLiBUaGVyZWZvcmUsIEkgc2V0 IC0tY2FjaGU9bmV2ZXIsIGJ1dCBJIHRoaW5rIEkgb2JzZXJ2ZWQgbm8KPj4gPiA+IGRpZmZlcmVu Y2UgY29tcGFyZWQgdG8gdGhlIGRlZmF1bHQgc2V0dGluZyAoYXV0bykuCj4+ID4gPgo+PiA+ID4g QWRkaW5nIGxvZ2dpbmcgdG8gdmlydGlvZnNkIGFuZCBrZXJuZWwgX2ZlZWxlZF8gbGlrZSBpdCBt YWRlIHRoZSBwcm9ibGVtCj4+ID4gPiBoYXJkZXIgdG8gcmVwcm9kdWNlIC0gbGVhdmluZyBtZSB3 aXRoIHRoZSBpbXByZXNzaW9uIHRoYXQgc29tZSByYWNlIGlzCj4+ID4gPiBoYXBwZW5pbmcgb24g c29tZXdoZXJlLgo+PiA+ID4KPj4gPiA+IFRyeWluZyB0byBydWxlIG91dCB0aGF0IHZpcnRpb2Zz ZCBpcyByZXR1cm5pbmcgY29ycnVwdGVkIGRhdGEsIEkgYWRkZWQKPj4gPiA+IHNvbWUgbG9nZ2lu ZyBhbmQgaGFzaHN1bSBjYWxjdWxhdGlvbiBoYWNrcyB0byBpdCBbMV0uIFRoZSBoYXNoZXMgY2hl Y2sKPj4gPiA+IG91dCBhY3Jvc3MgbXVsdGlwbGUgYWNjZXNzZXMgYW5kIHRoZSBvcmRlciBhbmQg a2luZCBvZiBxdWV1ZWQgbWVzc2FnZXMKPj4gPiA+IGlzIGV4YWN0bHkgdGhlIHNhbWUgaW4gYm90 aCB0aGUgZXJyb3IgY2FzZSBhbmQgY3Jhc2ggY2FzZS4gZmlvIHdhcyBhbHNvCj4+ID4gPiB1bmFi bGUgdG8gZmluZCBhbnkgZXJyb3JzIHdpdGggYSBuYWl2ZSBqb2IgZGVzY3JpcHRpb24gWzNdLgo+ PiA+ID4KPj4gPiA+IE5leHQsIEkgdHJpZWQgdG8gY2FwdHVyZSBpbmZvIG9uIHRoZSBndWVzdCBz aWRlLiBUaGlzIGJlY2FtZSBhIGJpdAo+PiA+ID4gdHJpY2t5IHNpbmNlIHRoZSBjcmFzaGVzIGJl Y2FtZSBwcmV0dHkgcmFyZSBvbmNlIEkgZm9sbG93ZWQgYSBmaXhlZAo+PiA+ID4gcGF0dGVybiBv ZiBzdGFydGluZyBsb2cgY2FwdHVyZSwgcnVubmluZyBwZXJmIGFuZCB0cnlpbmcgdG8gcmVwcm9k dWNlCj4+ID4gPiB0aGUgcHJvYmxlbS4gVWx0aW1hdGVseSwgSSBoYWQgdGhlIG1vc3QgY29uc2lz dGVudCByZXN1bHRzIHdpdGgKPj4gPiA+IGltbWVkaWF0ZWx5IHJ1bm5pbmcgYSBwcm9ncmFtIHR3 aWNlOgo+PiA+ID4KPj4gPiA+ICAgL21udC9sZC1saW51eC1hYXJjaDY0LnNvLjEgL21udC9scy5j b3JldXRpbHMgLzsgXAo+PiA+ID4gICAgIC9tbnQvbGQtbGludXgtYWFyY2g2NC5zby4xIC9tbnQv bHMuY29yZXV0aWxzIC8KPj4gPiA+Cj4+ID4gPiAgICgvbW50IGJlaW5nIHRoZSB2aXJ0aW9mcyBt b3VudCkKPj4gPiA+Cj4+ID4gPiBGb3IgY29sbGVjdGluZyBsb2dzLCBJIG1hZGUgYSBoYWNrIHRv IHRoZSBndWVzdCBrZXJuZWwgaW4gb3JkZXIgdG8gZHVtcAo+PiA+ID4gdGhlIHBhZ2UgY29udGVu dCBhZnRlciByZWNlaXZpbmcgdGhlIHZpcnRpb2ZzIHJlc3BvbnNlcyBbNF0uIFJlcHJvZHVjaW5n Cj4+ID4gPiB0aGUgcHJvYmxlbSB3aXRoIHRoaXMsIGxlYXZlcyBtZSB3aXRoIGxvZ3MgdGhhdCBz ZWVtIHRvIHN1Z2dlc3QgdGhhdAo+PiA+ID4gdmlydGlvZnNkIGlzIHJldHVybmluZyBpZGVudGlj YWwgY29udGVudCwgYnV0IHRoZSBndWVzdCBrZXJuZWwgc2VlbXMgdG8KPj4gPiA+IHJlY2VpdmUg ZGlmZmVyaW5nIHBhZ2VzOgo+PiA+ID4KPj4gPiA+IGdvb2Qta2VybmVsIFs1XToKPj4gPiA+ICAg a2VybmVsOiB2aXJ0aW9fZnNfd2FrZV9wZW5kaW5nX2FuZF91bmxvY2s6IG9wY29kZSAzIHVuaXF1 ZSAweDMxMiBub2RlaWQgMHgxIGluLmxlbiA1NiBvdXQubGVuIDEwNAo+PiA+ID4gICBrZXJuZWw6 IHZpcnRpb2ZzIHZpcnRpbzE6IHZpcnRpb19mc192cV9kb25lIHJlcXVlc3RzLjAKPj4gPiA+ICAg a2VybmVsOiB2aXJ0aW9fZnNfd2FrZV9wZW5kaW5nX2FuZF91bmxvY2s6IG9wY29kZSAxIHVuaXF1 ZSAweDMxNCBub2RlaWQgMHgxIGluLmxlbiA1MyBvdXQubGVuIDEyOAo+PiA+ID4gICBrZXJuZWw6 IHZpcnRpb2ZzIHZpcnRpbzE6IHZpcnRpb19mc192cV9kb25lIHJlcXVlc3RzLjAKPj4gPiA+ICAg a2VybmVsOiB2aXJ0aW9fZnNfd2FrZV9wZW5kaW5nX2FuZF91bmxvY2s6IG9wY29kZSAzIHVuaXF1 ZSAweDMxNiBub2RlaWQgMHgyOSBpbi5sZW4gNTYgb3V0LmxlbiAxMDQKPj4gPiA+ICAga2VybmVs OiB2aXJ0aW9mcyB2aXJ0aW8xOiB2aXJ0aW9fZnNfdnFfZG9uZSByZXF1ZXN0cy4wCj4+ID4gPiAg IGtlcm5lbDogdmlydGlvX2ZzX3dha2VfcGVuZGluZ19hbmRfdW5sb2NrOiBvcGNvZGUgMTQgdW5p cXVlIDB4MzE4IG5vZGVpZCAweDI5IGluLmxlbiA0OCBvdXQubGVuIDE2Cj4+ID4gPiAgIGtlcm5l bDogdmlydGlvZnMgdmlydGlvMTogdmlydGlvX2ZzX3ZxX2RvbmUgcmVxdWVzdHMuMAo+PiA+ID4g ICBrZXJuZWw6IHZpcnRpb19mc193YWtlX3BlbmRpbmdfYW5kX3VubG9jazogb3Bjb2RlIDE1IHVu aXF1ZSAweDMxYSBub2RlaWQgMHgyOSBpbi5sZW4gODAgb3V0LmxlbiA4MzIKPj4gPiA+ICAga2Vy bmVsOiB2aXJ0aW9mcyB2aXJ0aW8xOiB2aXJ0aW9fZnNfdnFfZG9uZSByZXF1ZXN0cy4wCj4+ID4g PiAgIGtlcm5lbDogdmlydGlvX2ZzOiBwYWdlOiAwMDAwMDAwMDY5OTZkNTIwCj4+ID4gPiAgIGtl cm5lbDogdmlydGlvX2ZzOiB0bzogMDAwMDAwMDBkZTU5MGMxNAo+PiA+ID4gICBrZXJuZWw6IHZp cnRpb19mcyByc3A6MDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAg MDAgIC4uLi4uLi4uLi4uLi4uLi4KPj4gPiA+ICAga2VybmVsOiB2aXJ0aW9fZnMgcnNwOjAwIDAw IDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwICAuLi4uLi4uLi4uLi4u Li4uCj4+ID4gPiAgIGtlcm5lbDogdmlydGlvX2ZzIHJzcDowMCAwMCAwMCAwMCAwMCAwMCAwMCAw MCAwMCAwMCAwMCAwMCAwMCAwMCAwMCAwMCAgLi4uLi4uLi4uLi4uLi4uLgo+PiA+ID4gICBrZXJu ZWw6IHZpcnRpb19mcyByc3A6MDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAg MDAgMDAgMDAgIC4uLi4uLi4uLi4uLi4uLi4KPj4gPiA+ICAgWy4uLl0KPj4gPiA+Cj4+ID4gPiBi YWQta2VybmVsIFs2XToKPj4gPiA+ICAga2VybmVsOiB2aXJ0aW9fZnNfd2FrZV9wZW5kaW5nX2Fu ZF91bmxvY2s6IG9wY29kZSAzIHVuaXF1ZSAweDE2MiBub2RlaWQgMHgxIGluLmxlbiA1NiBvdXQu bGVuIDEwNAo+PiA+ID4gICBrZXJuZWw6IHZpcnRpb2ZzIHZpcnRpbzE6IHZpcnRpb19mc192cV9k b25lIHJlcXVlc3RzLjAKPj4gPiA+ICAga2VybmVsOiB2aXJ0aW9fZnNfd2FrZV9wZW5kaW5nX2Fu ZF91bmxvY2s6IG9wY29kZSAxIHVuaXF1ZSAweDE2NCBub2RlaWQgMHgxIGluLmxlbiA1MyBvdXQu bGVuIDEyOAo+PiA+ID4gICBrZXJuZWw6IHZpcnRpb2ZzIHZpcnRpbzE6IHZpcnRpb19mc192cV9k b25lIHJlcXVlc3RzLjAKPj4gPiA+ICAga2VybmVsOiB2aXJ0aW9fZnNfd2FrZV9wZW5kaW5nX2Fu ZF91bmxvY2s6IG9wY29kZSAzIHVuaXF1ZSAweDE2NiBub2RlaWQgMHgxNiBpbi5sZW4gNTYgb3V0 LmxlbiAxMDQKPj4gPiA+ICAga2VybmVsOiB2aXJ0aW9mcyB2aXJ0aW8xOiB2aXJ0aW9fZnNfdnFf ZG9uZSByZXF1ZXN0cy4wCj4+ID4gPiAgIGtlcm5lbDogdmlydGlvX2ZzX3dha2VfcGVuZGluZ19h bmRfdW5sb2NrOiBvcGNvZGUgMTQgdW5pcXVlIDB4MTY4IG5vZGVpZCAweDE2IGluLmxlbiA0OCBv dXQubGVuIDE2Cj4+ID4gPiAgIGtlcm5lbDogdmlydGlvZnMgdmlydGlvMTogdmlydGlvX2ZzX3Zx X2RvbmUgcmVxdWVzdHMuMAo+PiA+ID4gICBrZXJuZWw6IHZpcnRpb19mc193YWtlX3BlbmRpbmdf YW5kX3VubG9jazogb3Bjb2RlIDE1IHVuaXF1ZSAweDE2YSBub2RlaWQgMHgxNiBpbi5sZW4gODAg b3V0LmxlbiA4MzIKPj4gPiA+ICAga2VybmVsOiB2aXJ0aW9mcyB2aXJ0aW8xOiB2aXJ0aW9fZnNf dnFfZG9uZSByZXF1ZXN0cy4wCj4+ID4gPiAgIGtlcm5lbDogdmlydGlvX2ZzOiBwYWdlOiAwMDAw MDAwMDZjZTlhNTU5Cj4+ID4gPiAgIGtlcm5lbDogdmlydGlvX2ZzOiB0bzogMDAwMDAwMDA3YWU4 Yjk0NgoKQXJlIHRoZXNlIHRoZSBjb3B5aW5nIGZyb20gdmlydGlvIHRvIGJ1ZmZlciBjYWNoZT8g SSB3b3VsZCBhc3N1bWUgdGhleQp0cmlnZ2VyIC1kIHRyYWNlOm1lbW9yeV9ub3RkaXJ0eV93cml0 ZV9hY2Nlc3MgaWYgc28uCgo+PiA+ID4gICBrZXJuZWw6IHZpcnRpb19mcyByc3A6MDAgMDAgMDAg MDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgIC4uLi4uLi4uLi4uLi4uLi4K Pj4gPiA+ICAga2VybmVsOiB2aXJ0aW9fZnMgcnNwOjAwIDAwIDAwIDAwIDAwIDAwIDAwIDAwIDAw IDAwIDAwIDAwIDAwIDAwIDAwIDAwICAuLi4uLi4uLi4uLi4uLi4uCj4+ID4gPiAgIGtlcm5lbDog dmlydGlvX2ZzIHJzcDo4MCA0MCBkZSBjOCBmZiBmZiAwMCAwMCBjYyAyYiA2MiBhZSBmZiBmZiAw MCAwMCAgLkAuLi4uLi4uK2IuLi4uLgo+PiA+ID4gICBrZXJuZWw6IHZpcnRpb19mcyByc3A6MDIg NGUgZGUgYzggZmYgZmYgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgMDAgIC5OLi4uLi4uLi4u Li4uLi4KPj4gPiA+ICAgWy4uLl0KPj4gPiA+Cj4+ID4gPiBXaGVuIGxvb2tpbmcgYXQgdGhlIGNv cnJlc3BvbmRpbmcgb3V0cHV0IGZyb20gdmlydGlvZnNkLCBpdCBjbGFpbXMgdG8KPj4gPiA+IGhh dmUgcmV0dXJuZWQgaWRlbnRpY2FsIGRhdGE6Cj4+ID4gPgo+PiA+ID4gZ29vZC12aXJ0aW9mc2Qg WzddOgo+PiA+ID4gICBbREVCVUcgdmlydGlvZnNkOjpzZXJ2ZXJdIFJlY2VpdmVkIHJlcXVlc3Q6 IG9wY29kZT1SZWFkICgxNSksIGlub2RlPTQxLCB1bmlxdWU9Nzk0LCBwaWQ9NDgxCj4+ID4gPiAg IFtzcmMvc2VydmVyLnJzOjYxOF0gci5yZWFkX29iaigpLm1hcF9lcnIoRXJyb3I6OkRlY29kZU1l c3NhZ2UpPyA9IFJlYWRJbiB7Cj4+ID4gPiAgICAgICBmaDogMzEsCj4+ID4gPiAgICAgICBvZmZz ZXQ6IDAsCj4+ID4gPiAgICAgICBzaXplOiA4MzIsCj4+ID4gPiAgICAgICByZWFkX2ZsYWdzOiAy LAo+PiA+ID4gICAgICAgbG9ja19vd25lcjogNjgzODU1NDcwNTYzOTk2NzI0NCwKPj4gPiA+ICAg ICAgIGZsYWdzOiAxMzEwNzIsCj4+ID4gPiAgICAgICBwYWRkaW5nOiAwLAo+PiA+ID4gICB9Cj4+ ID4gPiAgIFtzcmMvZmlsZV90cmFpdHMucnM6MTYxXSBoYXNoID0gMjMwODQ5MDQ1MDc1MTM2NDk5 NAo+PiA+ID4gICBbREVCVUcgdmlydGlvZnNkOjpzZXJ2ZXJdIFJlcGx5aW5nIE9LLCBoZWFkZXI6 IE91dEhlYWRlciB7IGxlbjogODQ4LCBlcnJvcjogMCwgdW5pcXVlOiA3OTQgfQo+PiA+ID4KPj4g PiA+IGJhZC12aXJ0aW9mc2QgWzhdOgo+PiA+ID4gICBbREVCVUcgdmlydGlvZnNkOjpzZXJ2ZXJd IFJlY2VpdmVkIHJlcXVlc3Q6IG9wY29kZT1SZWFkICgxNSksIGlub2RlPTIyLCB1bmlxdWU9MzYy LCBwaWQ9NDA2Cj4+ID4gPiAgIFtzcmMvc2VydmVyLnJzOjYxOF0gci5yZWFkX29iaigpLm1hcF9l cnIoRXJyb3I6OkRlY29kZU1lc3NhZ2UpPyA9IFJlYWRJbiB7Cj4+ID4gPiAgICAgICBmaDogMTIs Cj4+ID4gPiAgICAgICBvZmZzZXQ6IDAsCj4+ID4gPiAgICAgICBzaXplOiA4MzIsCj4+ID4gPiAg ICAgICByZWFkX2ZsYWdzOiAyLAo+PiA+ID4gICAgICAgbG9ja19vd25lcjogNjE4MTEyMDkyNjI1 ODM5NTU1NCwKPj4gPiA+ICAgICAgIGZsYWdzOiAxMzEwNzIsCj4+ID4gPiAgICAgICBwYWRkaW5n OiAwLAo+PiA+ID4gICB9Cj4+ID4gPiAgIFtzcmMvZmlsZV90cmFpdHMucnM6MTYxXSBoYXNoID0g MjMwODQ5MDQ1MDc1MTM2NDk5NAo+PiA+ID4gICBbREVCVUcgdmlydGlvZnNkOjpzZXJ2ZXJdIFJl cGx5aW5nIE9LLCBoZWFkZXI6IE91dEhlYWRlciB7IGxlbjogODQ4LCBlcnJvcjogMCwgdW5pcXVl OiAzNjIgfQo+PiA+ID4KPj4gPiA+IFRoZSAiY29ycnVwdGlvbiIgb25seSBzZWVtcyB0byBoYXBw ZW4gaW4gdGhpcyBvbmUgcGFnZSwgYWxsIG90aGVyIHBhZ2VzCj4+ID4gPiBhcmUgaWRlbnRpY2Fs IGJldHdlZW4gcnVucyAoZXhjZXB0IHRoYXQgdGhlIGJhZCBydW4gdGVybWluYXRlcyBlYXJsaWVy KS4KPj4gPiA+Cj4+ID4gPiBXaGF0IGRvIHRoZSBleHBlcnRzIHRoaW5rIGhlcmU/IFRvIG1lIGl0 IGZlZWxzIGEgYml0IGxpa2Ugc29tZSBraW5kIG9mCj4+ID4gPiBjb3JydXB0aW9uIGlzIGdvaW5n IG9uLiBPciBhbSBJIG1pc2ludGVycHJldGluZyB0aGluZ3MgaGVyZT8KPj4gPiA+Cj4+ID4gPiBX aGljaCBmdXJ0aGVyIGFuYWx5c2lzIHN0ZXBzIHdvdWxkIHlvdSBzdWdnZXN0Pwo+PiA+ID4KPj4g PiA+Cj4+ID4gPiBGdXJ0aGVyIG5vdGVzOgo+PiA+ID4KPj4gPiA+IEFmdGVyIGNvbGxlY3Rpbmcg dGhlIGFib3ZlIHJlc3VsdHMsIEkgcmVhbGl6ZWQgdGhhdCBydW5uaW5nIHRoZSBndWVzdAo+PiA+ ID4gd2l0aCAtc21wIDEgbWFrZXMgdGhlIHByb2JsZW1zIGEgbG90IHdvcnNlLiBTbyBtYXliZSB0 aGF0IGlzIGEgYmV0dGVyCj4+ID4gPiBjaG9pY2Ugd2hlbiB0cnlpbmcgdG8gcmVwcm9kdWNlIGl0 Lgo+PiA+ID4KPj4gPiA+IFJlcG8gd2l0aCBteSBzY3JpcHRzIGlzIGF2YWlsYWJsZSBhdDoKPj4g PiA+IGh0dHBzOi8vZ2l0LmNvZGVsaW5hcm8ub3JnL2VyaWtfc2NoaWxsaW5nL2ppcmEtb3Jrby02 NS1ib290c3RyYXAtazNzLWNvbmZpZy8KPj4gPiA+Cj4+ID4gPiBUaGUgc2NyaXB0cyBhcmUganVz dCBxdWljayBhbmQgZGlydHkgaW1wbGVtZW50YXRpb25zIGFuZCBhcmUgbm90Cj4+ID4gPiBwYXJ0 aWN1bGFyeSBwb3J0YWJsZS4KPj4gPgo+PiA+IFN1bW1hcnkgb2YgbXkgdGVzdGluZyBkdXJpbmcg dGhlIGxhc3QgZmV3IGRheXM6Cj4+ID4KPj4gPiBUZXN0aW5nIHdpdGggS0NTQU4gcmV2ZWFsZWQg YSBmZXcgY2FzZXMgdGhhdCBsb29rIGxpa2UgbWlzc2luZyBSRUFEX09OQ0UKPj4gPiBhbm5vdGF0 aW9ucyAod2lsbCBzZW5kIHBhdGNoZXMgc2VwYXJhdGVseSkuIEJ1dCBub3RoaW5nIG9mIHRoYXQg d2FzCj4+ID4gcmVsYXRlZCB0byB0aGUgaW1tZWRpYXRlIHByb2JsZW0uIEkgdGVzdGVkIGluc3Ry dW1lbnRfcmVhZCgpIGFuZCBhbm90aGVyCj4+ID4gcm91bmQgb2YgbG9nZ2luZyB3aXRoIGEgZGVs YXkgdG8gdmlydGlvX2ZzX3JlcXVlc3RfY29tcGxldGUuIEl0IGxvb2tzCj4+ID4gbGlrZSB0aGUg YnVmZmVyIGdldCBjb3JydXB0ZWQgYmVmb3JlIGVudGVyaW5nIHRoYXQgZnVuY3Rpb24uIEtDU0FO Cj4+ID4gb3IgbWFudWFsIHNsZWVwcyArIHByaW50cyBkaWQgbm90IHNob3cgYW55IGNvcnJ1cHRp b24gd2hpbGUgaW4gdGhhdAo+PiA+IGZ1bmN0aW9uLgo+PiA+Cj4+ID4gS0FTQU4gZGlkIG5vdCBy ZXBvcnQgYW55IGlzc3Vlcy4KPj4gPgo+PiA+IFBhdGNoaW5nIHZpcnRpb2ZzZCB0byBkbyBhbiBh ZGRpdGlvbmFsIGNvcHkgYW5kIGdvaW5nIHRocm91Z2ggcnVzdC12bW0ncwo+PiA+IC5jb3B5X3Rv KCkgZnVuY3Rpb24gZGlkIG5vdCBjaGFuZ2UgdGhlIGJlaGF2aW91ci4KPj4gPgo+PiA+IEkgd2ls bCBtb3N0bHkgYmUgb2ZmIG5leHQgd2Vlaywgd2lsbCBjb250aW51ZSBhbmFseXNpcyBhZnRlcndh cmRzLiBIYXBweQo+PiA+IHRvIGhlYXIgYWJvdXQgc3VnZ2VzdGlvbnMgb2Ygb3RoZXIgdGhpbmdz IHRvIHRyeSA6KS4KPj4KPj4gQmFjayBmcm9tIGEgd2VlayBvZiB2YWNhdGlvbi4uLgo+Pgo+PiBT dW1tYXJ5IG9mIHdoYXQgd2FzIGRpc2N1c3NlZCBvbiAjdmlydGlvZnM6bWF0cml4Lm9yZzoKPj4K Pj4gVGhlIGlzc3VlIG9ubHkgc2VlbXMgdG8gaGFwcGVuIGluIFFFTVUgVENHIHNjZW5hcmlvcyAo SSB0ZXN0ZWQgYWFyY2g2NAo+PiBhbmQgeDg2XzY0IG9uIHg4Nl82NCwgd2l6emFyZCBvbiBNYXRy aXggdGVzdGVkIGFybTMyKS4KPj4KPj4gQ0NpbmcgcWVtdS1kZXZlbC4gTWF5YmUgc29tZW9uZSBo YXMgc29tZSBoaW50cyBvbiB3aGVyZSB0byBmb2N1cyB0aGUKPj4gZGVidWdnaW5nIGVmZm9ydHM/ Cj4+Cj4+IEkgYW0gdHJ5aW5nIHRvIGJ1aWxkIGEgY29tcGxleCBtb25zdGVyIHNjcmlwdCBvZiB0 cmFjaW5nIHRoZSByZWxldmFudAo+PiBhZGRyZXNzZXMgaW4gb3JkZXIgdG8gZmlndXJlIG91dCB3 aGV0aGVyIHRoZSBndWVzdCBvciBob3N0IGRvZXMgdGhlCj4+IHdyaXRlcy4gQnV0IEkgYW0gaGFw cHkgdG8gaGVhciBhYm91dCBtb3JlIGNsZXZlciBpZGVhcyA6KS4KPgo+IEFmdGVyIGhlYXJpbmcg YWJvdXQgaW52ZXN0aWdhdGlvbnMgb2YgYnVncyBpbiBvdGhlciB2aXJ0aW8gc2NlbmFyaW9zCj4g dGhhdCBzZWVtIHRvIGJlIGNhdXNlZCBieSBRRU1VIFs5XSwgSSB0ZXN0ZWQgc29tZSBvbGRlciBR RU1VIHZlcnNpb25zLgo+Cj4gSW5kZWVkLCBhODgyYjU3MTIzNzMxNzFkM2JkNTNjZDgyZGRhYjQ0 NTNkZGVmNDY4IGRpZCBub3Qgc2hvdyB0aGUgYnVnZ3kKPiBiZWhhdmlvdXIuIFNvIEkgZGlkIGEg YmlzZWN0Ogo+Cj4gICAgIGdpdCBiaXNlY3Qgc3RhcnQKPiAgICAgIyBzdGF0dXM6IHdhaXRpbmcg Zm9yIGJvdGggZ29vZCBhbmQgYmFkIGNvbW1pdHMKPiAgICAgIyBnb29kOiBbYTg4MmI1NzEyMzcz MTcxZDNiZDUzY2Q4MmRkYWI0NDUzZGRlZjQ2OF0gUmV2ZXJ0ICJ2aXJ0aW86Cj4gaW50cm9kdWNl IG1hY3JvIElSVElPX0NPTkZJR19JUlFfSURYIgo+ICAgICBnaXQgYmlzZWN0IGdvb2QgYTg4MmI1 NzEyMzczMTcxZDNiZDUzY2Q4MmRkYWI0NDUzZGRlZjQ2OAo+ICAgICAjIHN0YXR1czogd2FpdGlu ZyBmb3IgYmFkIGNvbW1pdCwgMSBnb29kIGNvbW1pdCBrbm93bgo+ICAgICAjIGJhZDogWzllZjQ5 Nzc1NWFmYzI1MmZiOGUwNjBjOWVhNmIwOTg3YWJmZDIwYjZdIE1lcmdlIHRhZwo+ICdwdWxsLXZm aW8tMjAyMzA5MTEnIG9mIGh0dHBzOi8vZ2l0aHViLmNvbS9sZWdvYXRlci9xZW11IGludG8gc3Rh Z2luZwo+ICAgICBnaXQgYmlzZWN0IGJhZCA5ZWY0OTc3NTVhZmMyNTJmYjhlMDYwYzllYTZiMDk4 N2FiZmQyMGI2Cj4gICAgICMgc2tpcDogWzNiYTVmZTQ2ZWE0NDU2YTE2ZTJmNDdhYjhlNzU5NDNi NTQ4NzljNGVdIE1lcmdlIHRhZwo+ICdtaXBzLTIwMjIxMTA4JyBvZiBodHRwczovL2dpdGh1Yi5j b20vcGhpbG1kL3FlbXUgaW50byBzdGFnaW5nCj4gICAgIGdpdCBiaXNlY3Qgc2tpcCAzYmE1ZmU0 NmVhNDQ1NmExNmUyZjQ3YWI4ZTc1OTQzYjU0ODc5YzRlCj4gICAgICMgc2tpcDogW2FkZTc2MGEy ZjYzODA0YjdhYjE4MzlmYmMzZTVkZGJmMzA1Mzg3MThdIE1lcmdlIHRhZwo+ICdwdWxsLXJlcXVl c3QtMjAyMi0xMS0wOCcgb2YgaHR0cHM6Ly9naXRsYWIuY29tL3RodXRoL3FlbXUgaW50bwo+IHN0 YWdpbmcKPiAgICAgZ2l0IGJpc2VjdCBza2lwIGFkZTc2MGEyZjYzODA0YjdhYjE4MzlmYmMzZTVk ZGJmMzA1Mzg3MTgKPiAgICAgIyBnb29kOiBbYWQyY2EyZTNmNzYyYjBjYjk4ZWI5NzYwMDI1Njk3 OTViMjcwYWVmMV0gdGFyZ2V0L3h0ZW5zYTogRHJvcCB0Y2dfdGVtcF9mcmVlCj4gICAgIGdpdCBi aXNlY3QgZ29vZCBhZDJjYTJlM2Y3NjJiMGNiOThlYjk3NjAwMjU2OTc5NWIyNzBhZWYxCj4gICAg ICMgYmFkOiBbMTlhNzIwYjc0ZmRlN2U4NTlkMTlmMTJjNjZhNzJlNTQ1OTQ3YTY1N10gTWVyZ2Ug dGFnCj4gJ3RyYWNpbmctcHVsbC1yZXF1ZXN0JyBvZiBodHRwczovL2dpdGxhYi5jb20vc3RlZmFu aGEvcWVtdSBpbnRvCj4gc3RhZ2luZwo+ICAgICBnaXQgYmlzZWN0IGJhZCAxOWE3MjBiNzRmZGU3 ZTg1OWQxOWYxMmM2NmE3MmU1NDU5NDdhNjU3Cj4gICAgICMgYmFkOiBbMjlkOWVmY2ExNjA4MDIx MWYxMDdiNTQwZjA0ZDFlZDNjMTJjNjNiMF0gYXJtL0tjb25maWc6IERvCj4gbm90IGJ1aWxkIFRD Ry1vbmx5IGJvYXJkcyBvbiBhIEtWTS1vbmx5IGJ1aWxkCj4gICAgIGdpdCBiaXNlY3QgYmFkIDI5 ZDllZmNhMTYwODAyMTFmMTA3YjU0MGYwNGQxZWQzYzEyYzYzYjAKPiAgICAgIyBnb29kOiBbOTYz NmU1MTMyNTUzNjJjNGEzMjllM2U1ZmIyYzk3ZGFiM2M1Y2U0N10gTWVyZ2UgdGFnCj4gJ21pc2Mt bmV4dC1wdWxsLXJlcXVlc3QnIG9mIGh0dHBzOi8vZ2l0bGFiLmNvbS9iZXJyYW5nZS9xZW11IGlu dG8KPiBzdGFnaW5nCj4gICAgIGdpdCBiaXNlY3QgZ29vZCA5NjM2ZTUxMzI1NTM2MmM0YTMyOWUz ZTVmYjJjOTdkYWIzYzVjZTQ3Cj4gICAgICMgYmFkOiBbNDU2MDg2NTRhYTYzY2EyYjMxMWQ2Y2I3 NjFlMTUyMmYyMTI4ZTAwZV0gTWVyZ2UgdGFnCj4gJ3B1bGwtdHBtLTIwMjMtMDQtMjAtMScgb2Yg aHR0cHM6Ly9naXRodWIuY29tL3N0ZWZhbmJlcmdlci9xZW11LXRwbQo+IGludG8gc3RhZ2luZwo+ ICAgICBnaXQgYmlzZWN0IGJhZCA0NTYwODY1NGFhNjNjYTJiMzExZDZjYjc2MWUxNTIyZjIxMjhl MDBlCj4gICAgICMgZ29vZDogWzFmZjRhODFiZDNlZmIyMDc5OTJmMWRhMjY3ODg2ZmUwYzRkZjc2 NGZdIHRjZzogdXNlIFFUcmVlIGluc3RlYWQgb2YgR1RyZWUKPiAgICAgZ2l0IGJpc2VjdCBnb29k IDFmZjRhODFiZDNlZmIyMDc5OTJmMWRhMjY3ODg2ZmUwYzRkZjc2NGYKPiAgICAgIyBiYWQ6IFs5 ZWQ5OGNhZTE1MTM2OGNjODljNGJiNzdjOWYzMjVmNzE4NWU4ZjA5XSBibG9jay1iYWNrZW5kOgo+ IGlnbm9yZSBpbnNlcnRlZCBzdGF0ZSBpbiBibGtfY29fbmJfc2VjdG9ycwo+ICAgICBnaXQgYmlz ZWN0IGJhZCA5ZWQ5OGNhZTE1MTM2OGNjODljNGJiNzdjOWYzMjVmNzE4NWU4ZjA5Cj4gICAgICMg Z29vZDogW2M4Y2I2MDMyOTNmZDMyOWYyYTYyYWRlNzZlYzlkZTNmNDYyZmM1YzNdIHRlc3RzL2F2 b2NhZG86IFRlc3QgWGVuIGd1ZXN0IHN1cHBvcnQgdW5kZXIgS1ZNCj4gICAgIGdpdCBiaXNlY3Qg Z29vZCBjOGNiNjAzMjkzZmQzMjlmMmE2MmFkZTc2ZWM5ZGUzZjQ2MmZjNWMzCj4gICAgICMgYmFk OiBbNjRmMWM2M2Q4NzIwOGUyOGU4ZTM4YzRhYjUxNGFkYTE3Mjg5NjBlZl0gTWVyZ2UgdGFnCj4g J3B1bGxfZXJyb3JfaGFuZGxlX2ZpeF91c2VfYWZ0ZXJfZnJlZS52MScgb2YKPiBodHRwczovL2dp dGh1Yi5jb20vc3RlZmFuYmVyZ2VyL3FlbXUtdHBtIGludG8gc3RhZ2luZwo+ICAgICBnaXQgYmlz ZWN0IGJhZCA2NGYxYzYzZDg3MjA4ZTI4ZThlMzhjNGFiNTE0YWRhMTcyODk2MGVmCj4gICAgICMg Z29vZDogWzhhNzEyZGY0ZDRkNzM2YjdmZTY0NDE2MjY2NzdiZmQyNzFkOTViMTVdIE1lcmdlIHRh Zwo+ICdwdWxsLWZvci04LjAtMDQwNDIzLTInIG9mIGh0dHBzOi8vZ2l0bGFiLmNvbS9zdHNxdWFk L3FlbXUgaW50bwo+IHN0YWdpbmcKPiAgICAgZ2l0IGJpc2VjdCBnb29kIDhhNzEyZGY0ZDRkNzM2 YjdmZTY0NDE2MjY2NzdiZmQyNzFkOTViMTUKPiAgICAgIyBiYWQ6IFs3ZDAzMzRlNDkxMTE3ODdh ZTE5ZmJjOGQyOWZmNmU3MzQ3ZjA2MDVlXSBNZXJnZSB0YWcKPiAncHVsbC10Y2ctMjAyMzA0MDQn IG9mIGh0dHBzOi8vZ2l0bGFiLmNvbS9ydGg3NjgwL3FlbXUgaW50byBzdGFnaW5nCj4gICAgIGdp dCBiaXNlY3QgYmFkIDdkMDMzNGU0OTExMTc4N2FlMTlmYmM4ZDI5ZmY2ZTczNDdmMDYwNWUKPiAg ICAgIyBiYWQ6IFszMzcxODAyZmJhM2Y3YmU0NDY1ZjhhNWU1Nzc3ZDQzZDU1NjY3NmVmXSBhY2Nl bC90Y2c6IEZpeCBqdW1wIGNhY2hlIHNldCBpbiBjcHVfZXhlY19sb29wCj4gICAgIGdpdCBiaXNl Y3QgYmFkIDMzNzE4MDJmYmEzZjdiZTQ0NjVmOGE1ZTU3NzdkNDNkNTU2Njc2ZWYKPiAgICAgIyBn b29kOiBbNmNkYTQxZGFhMjE2MmI4ZTEwNDgxMjQ2NTViYTAyYThjMmI3NjJiNF0gUmV2ZXJ0Cj4g ImxpbnV4LXVzZXIvYXJtOiBUYWtlIG1vcmUgY2FyZSBhbGxvY2F0aW5nIGNvbW1wYWdlIgo+ICAg ICBnaXQgYmlzZWN0IGdvb2QgNmNkYTQxZGFhMjE2MmI4ZTEwNDgxMjQ2NTViYTAyYThjMmI3NjJi NAo+ICAgICAjIHNraXA6IFtjODM1NzQzOTJlMGFmMTA4YTY0MzM0NzcxMjU2NGY2NzQ5OTA2NDEz XSBhY2NlbC90Y2c6IEZpeCBvdmVyd3JpdGUgcHJvYmxlbXMgb2YgdGNnX2NmbGFncwo+ICAgICBn aXQgYmlzZWN0IHNraXAgYzgzNTc0MzkyZTBhZjEwOGE2NDMzNDc3MTI1NjRmNjc0OTkwNjQxMwo+ ICAgICAjIG9ubHkgc2tpcHBlZCBjb21taXRzIGxlZnQgdG8gdGVzdAo+ICAgICAjIHBvc3NpYmxl IGZpcnN0IGJhZCBjb21taXQ6Cj4gWzMzNzE4MDJmYmEzZjdiZTQ0NjVmOGE1ZTU3NzdkNDNkNTU2 Njc2ZWZdIGFjY2VsL3RjZzogRml4IGp1bXAgY2FjaGUKPiBzZXQgaW4gY3B1X2V4ZWNfbG9vcAoK SXQgc2hvdWxkIGJlIHBvc3NpYmxlIHRvIHJ1bGUgb3V0IHRoZSBqdW1wIGNhY2hlIGluIEhFQUQg YnkgcGF0Y2hpbmcgb3V0CnRoZSB0d286CgogICAgICAgIGlmIChsaWtlbHkodGIgJiYKICAgICAg ICAgICAgICAgICAgIHRiLT5wYyA9PSBwYyAmJgogICAgICAgICAgICAgICAgICAgdGItPmNzX2Jh c2UgPT0gY3NfYmFzZSAmJgogICAgICAgICAgICAgICAgICAgdGItPmZsYWdzID09IGZsYWdzICYm CiAgICAgICAgICAgICAgICAgICB0Yl9jZmxhZ3ModGIpID09IGNmbGFncykpIHsKICAgICAgICAg ICAgcmV0dXJuIHRiOwogICAgICAgIH0KCmZ1bmN0aW9ucyB0byBpZigwKSB7IHJldHVybiB0YiB9 IGluIHRiX2xvb2t1cC4gVGhhdCBjYW4gcnVsZSBvdXQgaWYgYQpjb3JydXB0ZWQganVtcCBjYWNo ZSBpcyByZXNwb25zaWJsZSAoYXQgdGhlIGNvc3Qgb2YgcnVubmluZyBhIGJpdApzbG93ZXIpLgoK QXByb3BvcyBvZiBlYXJsaWVyIHRiX2ptcF9jYWNoZV9pbnZhbF90YigpIGlzIHJlc3BvbnNpYmxl IGZvciByZW1vdmluZwplbnRyaWVzIG9mIFRCJ3MgdGhhdCBhcmUgaW52YWxpZGF0ZWQgZm9yIHdo YXRldmVyIHJlYXNvbiBmcm9tIHRoZSBjYWNoZS4KCj4gICAgICMgcG9zc2libGUgZmlyc3QgYmFk IGNvbW1pdDoKPiBbYzgzNTc0MzkyZTBhZjEwOGE2NDMzNDc3MTI1NjRmNjc0OTkwNjQxM10gYWNj ZWwvdGNnOiBGaXggb3ZlcndyaXRlCj4gcHJvYmxlbXMgb2YgdGNnX2NmbGFncwo+Cj4gSSBoYWQg YW4gaW5jbHVzaXZlIHRlc3QgaW4gdGhlIGVuZCB3aGVyZSBjODM1NzQzOTJlIGRpZCBub3QgeWll bGQgaW4gbWUKPiBiZWluZyBhYmxlIHRvIHN0YXJ0IHRoZSBWTS4KCkl0J3MgaW50ZXJlc3Rpbmcg dGhhdCBpcyBpbnZvbHZlZCB0aGUgUENSRUwgc3R1ZmYuIFRoaXMgaXMgYQpyZWxhdGl2ZWx5IHJl Y2VudCBvcHRpbWlzYXRpb24gdGhhdCBhbGxvd3MgdXMgdG8gcmUtdXNlIHRyYW5zbGF0aW9ucyBm b3IKcGh5c2ljYWwgcGFnZXMgdGhhdCBhcmUgbWFwcGVkIGludG8gdmlydHVhbCBtZW1vcnkgbXVs dGlwbGUgdGltZXMuIFRvIGRvCnRoYXQgdGhlIHRhcmdldCB0cmFuc2xhdG9yIGhhcyB0byBzdXBw b3J0IENGX1BDUkVMIGFuZCB0cmFuc2xhdGUgb25seQprbm93aW5nIHRoZSBQQyBhcyBhbiBvZmZz ZXQgaW50byB0aGUgcGFnZS4KCj4KPiBXaGV0aGVyIG9uZSBvZiB0aGVzZSBjb250YWlucyBhIGJ1 ZyBvciB3aGV0aGVyIG9ubHkgbmV3IGJlaGF2aW91ciBvZgo+IFFFTVUgcmV2ZWFsZWQgYSBidWcg c29tZXdoZXJlIGVsc2UgaXMgb2YgY291cnNlIHN0aWxsIHRvIGJlIGZpZ3VyZWQgb3V0Lgo+Cj4g WzldIGh0dHBzOi8vZ2l0bGFiLmNvbS9xZW11LXByb2plY3QvcWVtdS8tL2lzc3Vlcy8xODY2CgpU aGlzIGlzIGZvbGxvdyB1cCBvbiAjMTgyNiwgIzE4MzQsICMxODQ2IHdoaWNoIGlzIGFsbCBicm9h ZGx5IHRvIGRvIHdpdGgKaG93IFFFTVUgZGVhbHMgd2l0aCB1cGRhdGVzIHRvIGl0J3MgbWVtb3J5 IG1hcHMgKGUuZy4gd2hlbiBQQ0kgZGV2aWNlcwphcmUgcmVtYXBwZWQvcmVjb25maWd1cmVkKS4g V2UgYXJlIHVuc3VyZSBpZiB0aGUgTUlQUyBhcmNoaXRlY3R1cmUKc2hvdWxkIGJlIGV4ZWN1dGlu ZyBzb21lIHNvcnQgb2YgYmFycmllciBhcm91bmQgcmVjb25maWd1cmF0aW9uIHRoYXQgY2FuCmVu c3VyZSB3ZSBleGl0IHRoZSBibG9jayBvciB3ZSBoYXZlIHRvIGRldGVjdCBtaWQtYmxvY2sgSU8g YWNjZXNzZXMKaWNvdW50IHN0eWxlLgoKPgo+IC0gRXJpawo+Cj4+Cj4+IC0gRXJpawo+Pgo+PiA+ Cj4+ID4gR29vZCB3ZWVrZW5kLAo+PiA+Cj4+ID4gLSBFcmlrCj4+ID4KPj4gPgo+PiA+ID4KPj4g PiA+IC0gRXJpawo+PiA+ID4KPj4gPiA+IFsxXSBodHRwczovL2dpdGxhYi5jb20vYWJsdS92aXJ0 aW9mc2QvLS9jb21taXQvMThmZDBjMTg0OWUxNWJjNTVmYmRkNmUxZjE2OTgwMWIyYjAzZGExZgo+ PiA+ID4gWzJdIGh0dHBzOi8vZ2l0bGFiLmNvbS9xZW11LXByb2plY3QvcWVtdS8tL2NvbW1pdC81 MGU3YTQwYWYzNzJlZTU5MzFjOTllZjczOTBmNWQzZDZmYmY2ZWM0Cj4+ID4gPiBbM10KPj4gPiA+ IGh0dHBzOi8vZ2l0LmNvZGVsaW5hcm8ub3JnL2VyaWtfc2NoaWxsaW5nL2ppcmEtb3Jrby02NS1i b290c3RyYXAtazNzLWNvbmZpZy8tL2Jsb2IvMzk3YTYzMTBkZWEzNTk3MzAyNWUzZDYxZjQ2MDkw YmYwYzA5Mjc2Mi9zaGFyZS1kaXIvd3JpdGUtYW5kLXZlcmlmeS1tbWFwLmZpbwo+PiA+ID4gWzRd IGh0dHBzOi8vZ2l0aHViLmNvbS9BYmx1L2xpbnV4L2NvbW1pdC8zODgwYjlmOGFmZmIwMWFlYWJi MGEwNGZlNzZhZDc3MDFkYzBiYjk1Cj4+ID4gPiBbNV0gTGluZSAxMjkyMzoKPj4gPiA+IGh0dHBz Oi8vZ2l0LmNvZGVsaW5hcm8ub3JnL2VyaWtfc2NoaWxsaW5nL2ppcmEtb3Jrby02NS1ib290c3Ry YXAtazNzLWNvbmZpZy8tL2Jsb2IvbWFpbi9sb2dzLzIwMjMtMDgtMjklMjAxMyUzQTQyJTNBMzUl MkIwMiUzQTAwL2dvb2QtZHJvcC1iYWQtMS50eHQKPj4gPiA+IFs2XSBMaW5lIDEyOTIzOgo+PiA+ ID4gaHR0cHM6Ly9naXQuY29kZWxpbmFyby5vcmcvZXJpa19zY2hpbGxpbmcvamlyYS1vcmtvLTY1 LWJvb3RzdHJhcC1rM3MtY29uZmlnLy0vYmxvYi9tYWluL2xvZ3MvMjAyMy0wOC0yOSUyMDEzJTNB NDIlM0EzNSUyQjAyJTNBMDAvZ29vZC1iYWQtMS50eHQKPj4gPiA+IFs3XQo+PiA+ID4gaHR0cHM6 Ly9naXQuY29kZWxpbmFyby5vcmcvZXJpa19zY2hpbGxpbmcvamlyYS1vcmtvLTY1LWJvb3RzdHJh cC1rM3MtY29uZmlnLy0vYmxvYi9tYWluL2xvZ3MvMjAyMy0wOC0yOSUyMDEzJTNBNDIlM0EzNSUy QjAyJTNBMDAvdmlydGlvZnNkLnR4dCNMMjUzOC0yNTQ5Cj4+ID4gPiBbOF0KPj4gPiA+IGh0dHBz Oi8vZ2l0LmNvZGVsaW5hcm8ub3JnL2VyaWtfc2NoaWxsaW5nL2ppcmEtb3Jrby02NS1ib290c3Ry YXAtazNzLWNvbmZpZy8tL2Jsb2IvbWFpbi9sb2dzLzIwMjMtMDgtMjklMjAxMyUzQTQyJTNBMzUl MkIwMiUzQTAwL3ZpcnRpb2ZzZC50eHQjTDEwNTItMTA2MwoKCi0tIApBbGV4IEJlbm7DqWUKVmly dHVhbGlzYXRpb24gVGVjaCBMZWFkIEAgTGluYXJvCl9fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fClZpcnR1YWxpemF0aW9uIG1haWxpbmcgbGlzdApWaXJ0dWFs aXphdGlvbkBsaXN0cy5saW51eC1mb3VuZGF0aW9uLm9yZwpodHRwczovL2xpc3RzLmxpbnV4Zm91 bmRhdGlvbi5vcmcvbWFpbG1hbi9saXN0aW5mby92aXJ0dWFsaXphdGlvbg== From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 183E6EDEC7B for ; Wed, 13 Sep 2023 15:53:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230020AbjIMPxc (ORCPT ); Wed, 13 Sep 2023 11:53:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44890 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229809AbjIMPxb (ORCPT ); Wed, 13 Sep 2023 11:53:31 -0400 Received: from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com [IPv6:2a00:1450:4864:20::32d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 81A28CCD for ; Wed, 13 Sep 2023 08:53:27 -0700 (PDT) Received: by mail-wm1-x32d.google.com with SMTP id 5b1f17b1804b1-401187f8071so234105e9.0 for ; Wed, 13 Sep 2023 08:53:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1694620406; x=1695225206; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:message-id:in-reply-to:date :subject:cc:to:from:user-agent:references:from:to:cc:subject:date :message-id:reply-to; bh=KbpgIZY0S+jrnQeTU9MHb5yuccFEI2qq5nwURR5Z7vY=; b=sxYmXU2lIBjYAH7eyGQ7Hq0KSDiSJbcuWKC5hJh6RMEFmziORpHB23AgUDZGBTmoU+ OfnKIQGj3KooB4ATvF7qZJAHay3Z4kqrE18YFWFPM6qZVjLoUq+bUrxatuu4orC5otbC bIyMLOsTcqyt2UcXOrIz0fyasLIA6KQUQqB4dAnONm9GlgcjJkK635H7HaU+FGUPBIN2 6s1gXx3ZFqUJKaHfhfwtpEJXPFLaOd03DK8db64v8ZIiV0sC+LDCXfTUVjfEoylHqtTr JOdqdHZimZscMOpUJt3r6kdWK+XiaeUZjR48JcHb3swXHgG76YPCuNI7eVUqdxZv7bvQ iMUw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694620406; x=1695225206; h=content-transfer-encoding:mime-version:message-id:in-reply-to:date :subject:cc:to:from:user-agent:references:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=KbpgIZY0S+jrnQeTU9MHb5yuccFEI2qq5nwURR5Z7vY=; b=XdHzzDxqV9tjSbGb/iFIedt6ezLUP/DWjgFAfo4DQFGcy4MOf0hRN8/VO0XrrZ9hmw WBRDppGeU2QjgnI/oaBU5OkV5bc9yxVeUNIZDpCFh2PfpRgna432lLLwUzNVj2RrdExb eGVwbW9BY76unbMPAuV2M5Vc2O3XZrKlEm/mLGb6i5IVP2PALpwLILkPryAk9O0cY/YN ovmUliCf0LOCEY1DpPvvu4HQkwZ0a0DMsMYjd6xPflORjbNiUs0kZ2wVjmyPD0yoDnxK Jy7R5WtWz2/E6Z7KK36PhXWxojRLY+JdqDFIL8z+H02bYamQexf+4NMLOckhxLfKmH7x 4FOA== X-Gm-Message-State: AOJu0YxBL38YVKDgOU86u5hI5L8CyDN+jTZT/ImWSG0ZIoWedsX5mXIO Mw+H1P6wnN6vSnRXyvLbPlm5gQ== X-Google-Smtp-Source: AGHT+IG7uN/b0G0olfsBzT5JVx50eogSPAxxETPGl3o5DM9I5OGZFQgctwuNG29LxQ5Cb8CEXzO01g== X-Received: by 2002:a05:600c:44c9:b0:3ff:a95b:9751 with SMTP id f9-20020a05600c44c900b003ffa95b9751mr4604899wmo.7.1694620405692; Wed, 13 Sep 2023 08:53:25 -0700 (PDT) Received: from zen.linaroharston ([85.9.250.243]) by smtp.gmail.com with ESMTPSA id z20-20020a7bc7d4000000b003feae747ff2sm2400657wmk.35.2023.09.13.08.53.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Sep 2023 08:53:24 -0700 (PDT) Received: from zen (localhost [127.0.0.1]) by zen.linaroharston (Postfix) with ESMTP id 496A41FFBB; Wed, 13 Sep 2023 16:53:24 +0100 (BST) References: User-agent: mu4e 1.11.17; emacs 29.1.50 From: Alex =?utf-8?Q?Benn=C3=A9e?= To: Erik Schilling Cc: virtualization@lists.linux-foundation.org, linux-fsdevel@vger.kernel.org, virtio-fs@redhat.com, Richard Henderson , Weiwei Li , Vivek Goyal , Stefan Hajnoczi , Miklos Szeredi , Manos Pitsidianakis , Viresh Kumar , linux-kernel@vger.kernel.org, qemu-devel@nongnu.org, German Maglione , Hanna Czenczek , Philippe =?utf-8?Q?Mathieu-Daud=C3=A9?= Subject: Re: [BUG] virtio-fs: Corruption when running binaries from virtiofsd-backed fs Date: Wed, 13 Sep 2023 16:38:12 +0100 In-reply-to: Message-ID: <87msxqrrwr.fsf@linaro.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org "Erik Schilling" writes: > CCing a few more people as suggested by stefanha on #qemu. (Add philmd who's tracking the MIPs failure) > On Wed Sep 13, 2023 at 8:18 AM CEST, Erik Schilling wrote: >> On Fri Sep 1, 2023 at 12:37 PM CEST, Erik Schilling wrote: >> > On Wed Aug 30, 2023 at 10:20 AM CEST, Erik Schilling wrote: >> > > Hi all! >> > > >> > > Some days ago I posted to #virtiofs:matrix.org, describing that I am >> > > observing what looks like a corruption when executing programs from a >> > > virtiofs-based filesystem. >> > > >> > > Over the last few days I spent more time drilling into the problem. >> > > This is an attempt at summarizing my findings in order to see what o= ther >> > > people think about this. >> > > >> > > When running binaries mounted from virtiofs they may either: fail wi= th a >> > > segfault, fail with badaddr, get stuck or - sometimes - succeed. >> > > >> > > Environment: >> > > Host: Fedora 38 running 6.4.11-200.fc38.x86_64 >> > > Guest: Yocto-based image: 6.4.9-yocto-standard, aarch64 >> > > virtiofsd: latest main + some debug prints [1] >> > > QEMU: built from recent git [2] >> > > >> > > virtiofsd invocation: >> > > RUST_LOG=3D"debug" ./virtiofsd --seccomp=3Dnone --sandbox=3Dnone \ >> > > --socket-path "fs.sock0" --shared-dir $PWD/share-dir/ --cache=3D= never >> > > >> > > QEMU invocation: >> > > ~/projects/qemu/build/qemu-system-aarch64 -kernel Image -machine v= irt \ >> > > -cpu cortex-a57 \ >> > > -serial mon:stdio \ >> > > -device virtio-net-pci,netdev=3Dnet0 \ >> > > -netdev user,id=3Dnet0,hostfwd=3Dtcp::2223-:22 \ >> > > -display none -m 2048 -smp 4 \ >> > > -object memory-backend-memfd,id=3Dmem,size=3D2048M,share=3Don \ >> > > -numa node,memdev=3Dmem \ >> > > -hda trs-overlay-guest.qcow2 \ >> > > -chardev socket,id=3Dchar0,path=3D"fs.sock0" \ >> > > -device vhost-user-fs-pci,queue-size=3D1024,chardev=3Dchar0,tag= =3D/dev/root \ >> > > -append 'root=3D/dev/vda2 ro log_buf_len=3D8M' >> > > >> > > I figured that launching virtiofsd with --cache=3Dalways masks the >> > > problem. Therefore, I set --cache=3Dnever, but I think I observed no >> > > difference compared to the default setting (auto). >> > > >> > > Adding logging to virtiofsd and kernel _feeled_ like it made the pro= blem >> > > harder to reproduce - leaving me with the impression that some race = is >> > > happening on somewhere. >> > > >> > > Trying to rule out that virtiofsd is returning corrupted data, I add= ed >> > > some logging and hashsum calculation hacks to it [1]. The hashes che= ck >> > > out across multiple accesses and the order and kind of queued messag= es >> > > is exactly the same in both the error case and crash case. fio was a= lso >> > > unable to find any errors with a naive job description [3]. >> > > >> > > Next, I tried to capture info on the guest side. This became a bit >> > > tricky since the crashes became pretty rare once I followed a fixed >> > > pattern of starting log capture, running perf and trying to reproduce >> > > the problem. Ultimately, I had the most consistent results with >> > > immediately running a program twice: >> > > >> > > /mnt/ld-linux-aarch64.so.1 /mnt/ls.coreutils /; \ >> > > /mnt/ld-linux-aarch64.so.1 /mnt/ls.coreutils / >> > > >> > > (/mnt being the virtiofs mount) >> > > >> > > For collecting logs, I made a hack to the guest kernel in order to d= ump >> > > the page content after receiving the virtiofs responses [4]. Reprodu= cing >> > > the problem with this, leaves me with logs that seem to suggest that >> > > virtiofsd is returning identical content, but the guest kernel seems= to >> > > receive differing pages: >> > > >> > > good-kernel [5]: >> > > kernel: virtio_fs_wake_pending_and_unlock: opcode 3 unique 0x312 n= odeid 0x1 in.len 56 out.len 104 >> > > kernel: virtiofs virtio1: virtio_fs_vq_done requests.0 >> > > kernel: virtio_fs_wake_pending_and_unlock: opcode 1 unique 0x314 n= odeid 0x1 in.len 53 out.len 128 >> > > kernel: virtiofs virtio1: virtio_fs_vq_done requests.0 >> > > kernel: virtio_fs_wake_pending_and_unlock: opcode 3 unique 0x316 n= odeid 0x29 in.len 56 out.len 104 >> > > kernel: virtiofs virtio1: virtio_fs_vq_done requests.0 >> > > kernel: virtio_fs_wake_pending_and_unlock: opcode 14 unique 0x318 = nodeid 0x29 in.len 48 out.len 16 >> > > kernel: virtiofs virtio1: virtio_fs_vq_done requests.0 >> > > kernel: virtio_fs_wake_pending_and_unlock: opcode 15 unique 0x31a = nodeid 0x29 in.len 80 out.len 832 >> > > kernel: virtiofs virtio1: virtio_fs_vq_done requests.0 >> > > kernel: virtio_fs: page: 000000006996d520 >> > > kernel: virtio_fs: to: 00000000de590c14 >> > > kernel: virtio_fs rsp:00 00 00 00 00 00 00 00 00 00 00 00 00 00 00= 00 ................ >> > > kernel: virtio_fs rsp:00 00 00 00 00 00 00 00 00 00 00 00 00 00 00= 00 ................ >> > > kernel: virtio_fs rsp:00 00 00 00 00 00 00 00 00 00 00 00 00 00 00= 00 ................ >> > > kernel: virtio_fs rsp:00 00 00 00 00 00 00 00 00 00 00 00 00 00 00= 00 ................ >> > > [...] >> > > >> > > bad-kernel [6]: >> > > kernel: virtio_fs_wake_pending_and_unlock: opcode 3 unique 0x162 n= odeid 0x1 in.len 56 out.len 104 >> > > kernel: virtiofs virtio1: virtio_fs_vq_done requests.0 >> > > kernel: virtio_fs_wake_pending_and_unlock: opcode 1 unique 0x164 n= odeid 0x1 in.len 53 out.len 128 >> > > kernel: virtiofs virtio1: virtio_fs_vq_done requests.0 >> > > kernel: virtio_fs_wake_pending_and_unlock: opcode 3 unique 0x166 n= odeid 0x16 in.len 56 out.len 104 >> > > kernel: virtiofs virtio1: virtio_fs_vq_done requests.0 >> > > kernel: virtio_fs_wake_pending_and_unlock: opcode 14 unique 0x168 = nodeid 0x16 in.len 48 out.len 16 >> > > kernel: virtiofs virtio1: virtio_fs_vq_done requests.0 >> > > kernel: virtio_fs_wake_pending_and_unlock: opcode 15 unique 0x16a = nodeid 0x16 in.len 80 out.len 832 >> > > kernel: virtiofs virtio1: virtio_fs_vq_done requests.0 >> > > kernel: virtio_fs: page: 000000006ce9a559 >> > > kernel: virtio_fs: to: 000000007ae8b946 Are these the copying from virtio to buffer cache? I would assume they trigger -d trace:memory_notdirty_write_access if so. >> > > kernel: virtio_fs rsp:00 00 00 00 00 00 00 00 00 00 00 00 00 00 00= 00 ................ >> > > kernel: virtio_fs rsp:00 00 00 00 00 00 00 00 00 00 00 00 00 00 00= 00 ................ >> > > kernel: virtio_fs rsp:80 40 de c8 ff ff 00 00 cc 2b 62 ae ff ff 00= 00 .@.......+b..... >> > > kernel: virtio_fs rsp:02 4e de c8 ff ff 00 00 00 00 00 00 00 00 00= 00 .N.............. >> > > [...] >> > > >> > > When looking at the corresponding output from virtiofsd, it claims to >> > > have returned identical data: >> > > >> > > good-virtiofsd [7]: >> > > [DEBUG virtiofsd::server] Received request: opcode=3DRead (15), in= ode=3D41, unique=3D794, pid=3D481 >> > > [src/server.rs:618] r.read_obj().map_err(Error::DecodeMessage)? = =3D ReadIn { >> > > fh: 31, >> > > offset: 0, >> > > size: 832, >> > > read_flags: 2, >> > > lock_owner: 6838554705639967244, >> > > flags: 131072, >> > > padding: 0, >> > > } >> > > [src/file_traits.rs:161] hash =3D 2308490450751364994 >> > > [DEBUG virtiofsd::server] Replying OK, header: OutHeader { len: 84= 8, error: 0, unique: 794 } >> > > >> > > bad-virtiofsd [8]: >> > > [DEBUG virtiofsd::server] Received request: opcode=3DRead (15), in= ode=3D22, unique=3D362, pid=3D406 >> > > [src/server.rs:618] r.read_obj().map_err(Error::DecodeMessage)? = =3D ReadIn { >> > > fh: 12, >> > > offset: 0, >> > > size: 832, >> > > read_flags: 2, >> > > lock_owner: 6181120926258395554, >> > > flags: 131072, >> > > padding: 0, >> > > } >> > > [src/file_traits.rs:161] hash =3D 2308490450751364994 >> > > [DEBUG virtiofsd::server] Replying OK, header: OutHeader { len: 84= 8, error: 0, unique: 362 } >> > > >> > > The "corruption" only seems to happen in this one page, all other pa= ges >> > > are identical between runs (except that the bad run terminates earli= er). >> > > >> > > What do the experts think here? To me it feels a bit like some kind = of >> > > corruption is going on. Or am I misinterpreting things here? >> > > >> > > Which further analysis steps would you suggest? >> > > >> > > >> > > Further notes: >> > > >> > > After collecting the above results, I realized that running the guest >> > > with -smp 1 makes the problems a lot worse. So maybe that is a better >> > > choice when trying to reproduce it. >> > > >> > > Repo with my scripts is available at: >> > > https://git.codelinaro.org/erik_schilling/jira-orko-65-bootstrap-k3s= -config/ >> > > >> > > The scripts are just quick and dirty implementations and are not >> > > particulary portable. >> > >> > Summary of my testing during the last few days: >> > >> > Testing with KCSAN revealed a few cases that look like missing READ_ON= CE >> > annotations (will send patches separately). But nothing of that was >> > related to the immediate problem. I tested instrument_read() and anoth= er >> > round of logging with a delay to virtio_fs_request_complete. It looks >> > like the buffer get corrupted before entering that function. KCSAN >> > or manual sleeps + prints did not show any corruption while in that >> > function. >> > >> > KASAN did not report any issues. >> > >> > Patching virtiofsd to do an additional copy and going through rust-vmm= 's >> > .copy_to() function did not change the behaviour. >> > >> > I will mostly be off next week, will continue analysis afterwards. Hap= py >> > to hear about suggestions of other things to try :). >> >> Back from a week of vacation... >> >> Summary of what was discussed on #virtiofs:matrix.org: >> >> The issue only seems to happen in QEMU TCG scenarios (I tested aarch64 >> and x86_64 on x86_64, wizzard on Matrix tested arm32). >> >> CCing qemu-devel. Maybe someone has some hints on where to focus the >> debugging efforts? >> >> I am trying to build a complex monster script of tracing the relevant >> addresses in order to figure out whether the guest or host does the >> writes. But I am happy to hear about more clever ideas :). > > After hearing about investigations of bugs in other virtio scenarios > that seem to be caused by QEMU [9], I tested some older QEMU versions. > > Indeed, a882b5712373171d3bd53cd82ddab4453ddef468 did not show the buggy > behaviour. So I did a bisect: > > git bisect start > # status: waiting for both good and bad commits > # good: [a882b5712373171d3bd53cd82ddab4453ddef468] Revert "virtio: > introduce macro IRTIO_CONFIG_IRQ_IDX" > git bisect good a882b5712373171d3bd53cd82ddab4453ddef468 > # status: waiting for bad commit, 1 good commit known > # bad: [9ef497755afc252fb8e060c9ea6b0987abfd20b6] Merge tag > 'pull-vfio-20230911' of https://github.com/legoater/qemu into staging > git bisect bad 9ef497755afc252fb8e060c9ea6b0987abfd20b6 > # skip: [3ba5fe46ea4456a16e2f47ab8e75943b54879c4e] Merge tag > 'mips-20221108' of https://github.com/philmd/qemu into staging > git bisect skip 3ba5fe46ea4456a16e2f47ab8e75943b54879c4e > # skip: [ade760a2f63804b7ab1839fbc3e5ddbf30538718] Merge tag > 'pull-request-2022-11-08' of https://gitlab.com/thuth/qemu into > staging > git bisect skip ade760a2f63804b7ab1839fbc3e5ddbf30538718 > # good: [ad2ca2e3f762b0cb98eb976002569795b270aef1] target/xtensa: Dro= p tcg_temp_free > git bisect good ad2ca2e3f762b0cb98eb976002569795b270aef1 > # bad: [19a720b74fde7e859d19f12c66a72e545947a657] Merge tag > 'tracing-pull-request' of https://gitlab.com/stefanha/qemu into > staging > git bisect bad 19a720b74fde7e859d19f12c66a72e545947a657 > # bad: [29d9efca16080211f107b540f04d1ed3c12c63b0] arm/Kconfig: Do > not build TCG-only boards on a KVM-only build > git bisect bad 29d9efca16080211f107b540f04d1ed3c12c63b0 > # good: [9636e513255362c4a329e3e5fb2c97dab3c5ce47] Merge tag > 'misc-next-pull-request' of https://gitlab.com/berrange/qemu into > staging > git bisect good 9636e513255362c4a329e3e5fb2c97dab3c5ce47 > # bad: [45608654aa63ca2b311d6cb761e1522f2128e00e] Merge tag > 'pull-tpm-2023-04-20-1' of https://github.com/stefanberger/qemu-tpm > into staging > git bisect bad 45608654aa63ca2b311d6cb761e1522f2128e00e > # good: [1ff4a81bd3efb207992f1da267886fe0c4df764f] tcg: use QTree ins= tead of GTree > git bisect good 1ff4a81bd3efb207992f1da267886fe0c4df764f > # bad: [9ed98cae151368cc89c4bb77c9f325f7185e8f09] block-backend: > ignore inserted state in blk_co_nb_sectors > git bisect bad 9ed98cae151368cc89c4bb77c9f325f7185e8f09 > # good: [c8cb603293fd329f2a62ade76ec9de3f462fc5c3] tests/avocado: Tes= t Xen guest support under KVM > git bisect good c8cb603293fd329f2a62ade76ec9de3f462fc5c3 > # bad: [64f1c63d87208e28e8e38c4ab514ada1728960ef] Merge tag > 'pull_error_handle_fix_use_after_free.v1' of > https://github.com/stefanberger/qemu-tpm into staging > git bisect bad 64f1c63d87208e28e8e38c4ab514ada1728960ef > # good: [8a712df4d4d736b7fe6441626677bfd271d95b15] Merge tag > 'pull-for-8.0-040423-2' of https://gitlab.com/stsquad/qemu into > staging > git bisect good 8a712df4d4d736b7fe6441626677bfd271d95b15 > # bad: [7d0334e49111787ae19fbc8d29ff6e7347f0605e] Merge tag > 'pull-tcg-20230404' of https://gitlab.com/rth7680/qemu into staging > git bisect bad 7d0334e49111787ae19fbc8d29ff6e7347f0605e > # bad: [3371802fba3f7be4465f8a5e5777d43d556676ef] accel/tcg: Fix jump= cache set in cpu_exec_loop > git bisect bad 3371802fba3f7be4465f8a5e5777d43d556676ef > # good: [6cda41daa2162b8e1048124655ba02a8c2b762b4] Revert > "linux-user/arm: Take more care allocating commpage" > git bisect good 6cda41daa2162b8e1048124655ba02a8c2b762b4 > # skip: [c83574392e0af108a643347712564f6749906413] accel/tcg: Fix ove= rwrite problems of tcg_cflags > git bisect skip c83574392e0af108a643347712564f6749906413 > # only skipped commits left to test > # possible first bad commit: > [3371802fba3f7be4465f8a5e5777d43d556676ef] accel/tcg: Fix jump cache > set in cpu_exec_loop It should be possible to rule out the jump cache in HEAD by patching out the two: if (likely(tb && tb->pc =3D=3D pc && tb->cs_base =3D=3D cs_base && tb->flags =3D=3D flags && tb_cflags(tb) =3D=3D cflags)) { return tb; } functions to if(0) { return tb } in tb_lookup. That can rule out if a corrupted jump cache is responsible (at the cost of running a bit slower). Apropos of earlier tb_jmp_cache_inval_tb() is responsible for removing entries of TB's that are invalidated for whatever reason from the cache. > # possible first bad commit: > [c83574392e0af108a643347712564f6749906413] accel/tcg: Fix overwrite > problems of tcg_cflags > > I had an inclusive test in the end where c83574392e did not yield in me > being able to start the VM. It's interesting that is involved the PCREL stuff. This is a relatively recent optimisation that allows us to re-use translations for physical pages that are mapped into virtual memory multiple times. To do that the target translator has to support CF_PCREL and translate only knowing the PC as an offset into the page. > > Whether one of these contains a bug or whether only new behaviour of > QEMU revealed a bug somewhere else is of course still to be figured out. > > [9] https://gitlab.com/qemu-project/qemu/-/issues/1866 This is follow up on #1826, #1834, #1846 which is all broadly to do with how QEMU deals with updates to it's memory maps (e.g. when PCI devices are remapped/reconfigured). We are unsure if the MIPS architecture should be executing some sort of barrier around reconfiguration that can ensure we exit the block or we have to detect mid-block IO accesses icount style. > > - Erik > >> >> - Erik >> >> > >> > Good weekend, >> > >> > - Erik >> > >> > >> > > >> > > - Erik >> > > >> > > [1] https://gitlab.com/ablu/virtiofsd/-/commit/18fd0c1849e15bc55fbdd= 6e1f169801b2b03da1f >> > > [2] https://gitlab.com/qemu-project/qemu/-/commit/50e7a40af372ee5931= c99ef7390f5d3d6fbf6ec4 >> > > [3] >> > > https://git.codelinaro.org/erik_schilling/jira-orko-65-bootstrap-k3s= -config/-/blob/397a6310dea35973025e3d61f46090bf0c092762/share-dir/write-and= -verify-mmap.fio >> > > [4] https://github.com/Ablu/linux/commit/3880b9f8affb01aeabb0a04fe76= ad7701dc0bb95 >> > > [5] Line 12923: >> > > https://git.codelinaro.org/erik_schilling/jira-orko-65-bootstrap-k3s= -config/-/blob/main/logs/2023-08-29%2013%3A42%3A35%2B02%3A00/good-drop-bad-= 1.txt >> > > [6] Line 12923: >> > > https://git.codelinaro.org/erik_schilling/jira-orko-65-bootstrap-k3s= -config/-/blob/main/logs/2023-08-29%2013%3A42%3A35%2B02%3A00/good-bad-1.txt >> > > [7] >> > > https://git.codelinaro.org/erik_schilling/jira-orko-65-bootstrap-k3s= -config/-/blob/main/logs/2023-08-29%2013%3A42%3A35%2B02%3A00/virtiofsd.txt#= L2538-2549 >> > > [8] >> > > https://git.codelinaro.org/erik_schilling/jira-orko-65-bootstrap-k3s= -config/-/blob/main/logs/2023-08-29%2013%3A42%3A35%2B02%3A00/virtiofsd.txt#= L1052-1063 --=20 Alex Benn=C3=A9e Virtualisation Tech Lead @ Linaro