From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D5672C61D85 for ; Tue, 21 Nov 2023 17:25:12 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1r5UUc-0000Ok-9A; Tue, 21 Nov 2023 12:24:58 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1r5UUb-0000OV-6U for qemu-devel@nongnu.org; Tue, 21 Nov 2023 12:24:57 -0500 Received: from mail-wm1-x32e.google.com ([2a00:1450:4864:20::32e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1r5UUZ-0006O1-Ay for qemu-devel@nongnu.org; Tue, 21 Nov 2023 12:24:56 -0500 Received: by mail-wm1-x32e.google.com with SMTP id 5b1f17b1804b1-40806e4106dso19844315e9.1 for ; Tue, 21 Nov 2023 09:24:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1700587494; x=1701192294; darn=nongnu.org; h=content-transfer-encoding:mime-version:message-id:date:user-agent :references:in-reply-to:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=fJQ/onAn2xH3uj0/HxmqCjRnsjukcL+sLDS7EdDUbac=; b=SkY5BI5u1sfnM3mVVaZ/YPZo1sAK4Qd9mBeHticSLkoH1mOUrtKgSL4OYz3YDd3kyJ YcEdh1ekwVPW4NXz//byp82fOMtQlNDAjCqf5QEDD0zlOoFY4jA7rN4Gm2DidOdA4AT1 6RxWJ5JthkRsvy2FAc8jc6e4h7TGqLdJIXP0ZUZb2NCVKm4i+Ug55odCIjFEgSxaia7S mWOHJsNWWEbHqzDf2gP3izTNMNjt7vuj2BdXZybey5mB9ICGdDYWsmwIs5ykoENZInTI 6fVVffwNvJssDpEGrTf24GX+oEFF6N+Ms9ff6STUdoxl0owrix7KYir3Y8D7WSKLzPAr +OcQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700587494; x=1701192294; h=content-transfer-encoding:mime-version:message-id:date:user-agent :references:in-reply-to:subject:cc:to:from:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=fJQ/onAn2xH3uj0/HxmqCjRnsjukcL+sLDS7EdDUbac=; b=w8wyt7S8omsiBDarBCua61i5MLTudbL32yfvl0n3YDctO8mN77W604fz+nL2bi48Be lY1tRdEPPICI6u/928rkV+QR3WrqsycKm6QUGDJF4BMFlylEigfmVGFgHNHKwfeKfbtT lfobLb6UJPTPeGbiMg5M0x3jYkw+HeaQz1beOk2YgOXLLkvhaGvCIImjKT5Q53Bh0I53 sVFOSG6HlsG73SMtRkAO9aBqMmSrUv32eZXwldSbbMUx1rOI5Nkzn3wXqMTBYkqJuME6 wfwtQ43CSjLkkY3nE0mKBPImKC15ZJms2z/lqKYBxoHqzoj33GwkcEEmOnHiyIJwaWOp sUew== X-Gm-Message-State: AOJu0Yy0/XsK2FNB20TzfSQgup73UPibUeykF6tVGF24vS8hIgoyxwDt Qtgku1pRclCjly1FGLUYy7aTmw== X-Google-Smtp-Source: AGHT+IFkr9I6QCeo/tXCoh3jHDkkk8mcpYSfEX/zxoJFHyfFIU2b8NvKqrMKQHzwq3AOeU/s/Jdw/Q== X-Received: by 2002:a05:600c:45c7:b0:401:c8b9:4b86 with SMTP id s7-20020a05600c45c700b00401c8b94b86mr12067wmo.9.1700587493671; Tue, 21 Nov 2023 09:24:53 -0800 (PST) Received: from draig.lan ([85.9.250.243]) by smtp.gmail.com with ESMTPSA id d34-20020a05600c4c2200b0040a48430837sm21650332wmp.13.2023.11.21.09.24.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Nov 2023 09:24:53 -0800 (PST) Received: from draig (localhost [IPv6:::1]) by draig.lan (Postfix) with ESMTP id C5D185F74B; Tue, 21 Nov 2023 17:24:52 +0000 (GMT) From: =?utf-8?Q?Alex_Benn=C3=A9e?= To: Mikhail Tyutin Cc: "qemu-devel@nongnu.org" , Richard Henderson , "erdnaxe@crans.org" , "ma.mandourr@gmail.com" Subject: Re: Instruction virtual address in TCG Plugins In-Reply-To: (Mikhail Tyutin's message of "Tue, 21 Nov 2023 16:39:25 +0000") References: <87leb1xtdx.fsf@draig.linaro.org> <874jhoy54t.fsf@draig.linaro.org> User-Agent: mu4e 1.11.25; emacs 29.1 Date: Tue, 21 Nov 2023 17:24:52 +0000 Message-ID: <878r6rf28r.fsf@draig.linaro.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Received-SPF: pass client-ip=2a00:1450:4864:20::32e; envelope-from=alex.bennee@linaro.org; helo=mail-wm1-x32e.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Mikhail Tyutin writes: >> >> > I suspect it is because of memory mappings by guest OS that changes= virtual addresses for that block. >> >> > >> >> > I also looked at gen_empty_udata_cb() function and considered to ex= tend plugin API to pass a program counter >> >> > value as additional callback argument. I thought it would always gi= ve me valid virtual address of an instruction. >> >> > Unfortunately, I didn't find a way to get value of that register in= architecture agnostic way (it is 'pc' member in >> >> > CPUArchState structure). >> >> >> >> When we merge the register api you should be able to do that. Although >> >> during testing I realised that PC acted funny compared to everything >> >> else because we don't actually update the shadow register every >> >> instruction. >> > >> > We implemented similar API to read registers (by coincidence, I posted= this patch at the same time as the API you >> > mentioned) and I observe similar behavior. As far as I see, CPU state = is only updated in between of executed translation >> > blocks. Switching to 'singlestep' mode helps to fix that, but executio= n overhead is huge. >> > >> > There is also blocks 'chaining' mechanism which is likely contributes = to corrupted blocks vaddr inside of callbacks. >> > My guess is that 'pc' value for those chained blocks points to the fir= st block of entire chain. Unfortunately, It is very >> > hard to debug, because I can only see block chains when I run whole Li= nux guest OS. Does Qemu has small test >> > application to trigger long enough chain of translation blocks? >>=20 >> No all registers should be resolved by the end of any block. There is >> currently no optimisation of register usage between TBs. If you are >> seeing PC corruption that would be a bug - but fundamentally things >> would break pretty quick if tb_lookup() and friends didn't have an >> accurate PC. > > I managed to root cause source of corrupted addresses in plugin callbacks. > There were basically 2 problems: > > 1. Memory IO operations force TCG to create special translation blocks to > process that memory load/store operation. The plugin gets notification for > this translation block as well, but instrumentation callbacks other than > memory ones are silently ignored. To make it correct, the plugin has to m= atch > instruction execution callback from previous TB to memory callback from t= hat > special TB. The fix was to expose internal =E2=80=98memOnly=E2=80=99 TB f= lag to the plugin to > handle such TBs differently. Are you talking about the CF_MEMI_ONLY compile flag? We added this to avoid double counting executed instructions. Has there been a clash with the other changes to always cpu_recompile_io? This was a change added to fix: https://gitlab.com/qemu-project/qemu/-/issues/1866 Richard is going to look at optimising the cpu_recompile_io code so we "lock in" a shortened translation once we discover a block is doing MMIO. See https://linaro.atlassian.net/browse/QEMU-605 for an overview. > 2. Another problem is related to interrupts handling. Since we can insert= pre- > callback on instructions only, the plugin is not aware if instruction is > actually executed or interrupted by an interrupt or exception. In fact, it > mistakenly interprets all interrupted instructions as executed. Adding API > to receive interrupt notification and appropriate handling of it fixes > the problem. We don't process any interrupts until the start of each block so no asynchronous IRQs should interrupt execution. However it is possible that any given instruction could generate a synchronous exception so if you need a precise count of execution you need to instrument every single instruction. With enough knowledge the plugin could avoid instrumenting stuff that will never fault but that relies on baking additional knowledge into the plugin. Generally its only memory operations that can fault (although I guess FPU and some more esoteric integer ops can). > > I will send those patches for review shortly and thank you for dissuading= me > from going to wrong direction! --=20 Alex Benn=C3=A9e Virtualisation Tech Lead @ Linaro