From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 66763C2B9F8 for ; Tue, 25 May 2021 05:31:28 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 91AF461400 for ; Tue, 25 May 2021 05:31:27 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 91AF461400 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=gibson.dropbear.id.au Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:44770 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1llPf4-0000rB-Ei for qemu-devel@archiver.kernel.org; Tue, 25 May 2021 01:31:26 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:46180) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1llPdk-0007lL-Ia; Tue, 25 May 2021 01:30:05 -0400 Received: from bilbo.ozlabs.org ([203.11.71.1]:44359 helo=ozlabs.org) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1llPdh-0001HP-Ib; Tue, 25 May 2021 01:30:04 -0400 Received: by ozlabs.org (Postfix, from userid 1007) id 4Fq2hh02gYz9s24; Tue, 25 May 2021 15:29:55 +1000 (AEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gibson.dropbear.id.au; s=201602; t=1621920596; bh=JDLuRYHpN8gMCEckLjIoX+tAarkN5v+s4hb+kSdpRQw=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=pStaVQPJ4tbKOjJaQcQ3E3ZRYQHArq9nPPSRmJSPo3MOan4ujJuPrpqAMdENt2YAO DByg0adpltfnOgLDEaVeAP4OBDEMSD9tsxm++Fu7Ijwa/+JHNCe60Yx1jV3MDfA5y0 LhJjx7beHlHlN9Axv0ASAXetO2RilGtRqZYDDceQ= Date: Tue, 25 May 2021 15:24:53 +1000 From: David Gibson To: Alexey Kardashevskiy Subject: Re: [PATCH qemu v20] spapr: Implement Open Firmware client interface Message-ID: References: <5825cde5-a408-a438-116d-5a9d9113a52a@ozlabs.ru> <8527c8d2-c1e7-b3f8-0bda-529ba3864701@ozlabs.ru> <4f6ceca3-5f18-fe70-18f9-4efde8feb1ed@ozlabs.ru> <7a4e47e5-59b-9132-eafd-d84d8b73f5c@eik.bme.hu> <17fbb016-2e7-d57e-bedd-1ae7814fb860@eik.bme.hu> <939a489f-40de-da33-dd7-9fd1f5eb190@eik.bme.hu> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="uFv4duyxikqw4fwP" Content-Disposition: inline In-Reply-To: Received-SPF: pass client-ip=203.11.71.1; envelope-from=dgibson@ozlabs.org; helo=ozlabs.org X-Spam_score_int: -17 X-Spam_score: -1.8 X-Spam_bar: - X-Spam_report: (-1.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HEADER_FROM_DIFFERENT_DOMAINS=0.249, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: qemu-ppc@nongnu.org, qemu-devel@nongnu.org Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" --uFv4duyxikqw4fwP Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, May 24, 2021 at 10:46:26PM +1000, Alexey Kardashevskiy wrote: >=20 >=20 > On 24/05/2021 20:55, BALATON Zoltan wrote: > > On Mon, 24 May 2021, David Gibson wrote: > > > On Sun, May 23, 2021 at 07:09:26PM +0200, BALATON Zoltan wrote: > > > > On Sun, 23 May 2021, BALATON Zoltan wrote: > > > > > On Sun, 23 May 2021, Alexey Kardashevskiy wrote: > > > > > > One thing to note about PCI is that normally I think the client > > > > > > expects the firmware to do PCI probing and SLOF does it. But VOF > > > > > > does not and Linux scans PCI bus(es) itself. Might be a problem= for > > > > > > you kernel. > > > > >=20 > > > > > I'm not sure what info does MorphOS get from the device tree > > > > > and what it > > > > > probes itself but I think it may at least need device ids > > > > > and info about > > > > > the PCI bus to be able to access the config regs, after that it s= hould > > > > > set the devices up hopefully. I could add these from the board co= de to > > > > > device tree so VOF does not need to do anything about it. However= I'm > > > > > not getting to that point yet because it crashes on something tha= t it's > > > > > missing and couldn't yet find out what is that. > > > > >=20 > > > > > I'd like to get Linux working now as that would be enough to test= this > > > > > and then if for MorphOS we still need a ROM it's not a problem if= at > > > > > least we can boot Linux without the original firmware. But I can'= t make > > > > > Linux open a serial console and I don't know what it needs for th= at. Do > > > > > you happen to know? I've looked at the sources in > > > > > Linux/arch/powerpc but > > > > > not sure how it would find and open a serial port on pegasos2. It= seems > > > > > to work with the board firmware and now I can get it to boot with= VOF > > > > > but then it does not open serial so it probably needs something i= n the > > > > > device tree or expects the firmware to set something up that we s= hould > > > > > add in pegasos2.c when using VOF. > > > >=20 > > > > I've now found that Linux uses rtas methods read-pci-config and > > > > write-pci-config for PCI access on pegasos2 so this means that we'll > > > > probably need rtas too (I hoped we could get away without it if > > > > it were only > > > > used for shutdown/reboot or so but seems Linux needs it for PCI > > > > as well and > > > > does not scan the bus and won't find some devices without it). > > >=20 > > > Yes, definitely sounds like you'll need an RTAS implementation. > >=20 > > I plan to fix that after managed to get serial working as that seems to > > not need it. If I delete the rtas-size property from /rtas on the > > original firmware that makes Linux skip instantiating rtas, but I still > > get serial output just not accessing PCI devices. So I think it should > > work and keeps things simpler at first. Then I'll try rtas later. > >=20 > > > > While VOF can do rtas, this causes a problem with the hypercall > > > > method using > > > > sc 1 that goes through vhyp but trips the assert in ppc_store_sdr1(= ) so > > > > cannot work after guest is past quiesce. > > >=20 > > > > So the question is why is that > > > > assert there > > >=20 > > > Ah.. right.=A0 So, vhyp was designed for the PAPR use case, where we > > > want to model the CPU when it's in supervisor and user mode, but not > > > when it's in hypervisor mode.=A0 We want qemu to mimic the behaviour = of > > > the hypervisor, rather than attempting to actually execute hypervisor > > > code in the virtual CPU. > > >=20 > > > On systems that have a hypervisor mode, SDR1 is hypervisor privileged, > > > so it makes no sense for the guest to attempt to set it.=A0 That shou= ld > > > be caught by the general SPR code and turned into a 0x700, hence the > > > assert() if we somehow reach ppc_store_sdr1(). > > >=20 > > > So, we are seeing a problem here because you want the 'sc 1' > > > interception of vhyp, but not the rest of the stuff that goes with it. > > >=20 > > > > and would using sc 1 for hypercalls on pegasos2 cause other > > > > problems later even if the assert could be removed? > > >=20 > > > At least in the short term, I think you probably can remove the > > > assert.=A0 In your case the 'sc 1' calls aren't truly to a hypervisor, > > > but a special case escape to qemu for the firmware emulation.=A0 I th= ink > > > it's unlikely to cause problems later, because nothing on a 32-bit > > > system should be attempting an 'sc 1'.=A0 The only thing I can think = of > > > that would fail is some test case which explicitly verified that 'sc > > > 1' triggered a 0x700 (SIGILL from userspace). > >=20 > > OK so the assert should check if the CPU has an HV bit. I think there > > was a #detine for that somewhere that I can add to the assert then I can > > try that. What I wasn't sure about is that sc 1 would conflict with the > > guest's usage of normal sc calls or are these going through different > > paths and only sc 1 will trigger vhyp callback not affecting notmal sc > > calls? (Or if this causes an otherwise unnecessary VM exit on KVM even > > when it works then maybe looking for a different way in the future might > > be needed. But for now if this works with modifying the assert to allow > > this on ppc32 then I go for that as that's the simplest way for now.) > >=20 > > > > Can somebody who knows > > > > more about it explain this please? If this cannot be resolved > > > > then we may > > > > need a different hypercall method on pegasos2 (I've considered > > > > MOL OSI or > > > > are there other options? I may use some advice from people who know= it > > > > better, especially the possible interaction with KVM later as > > > > the long term > > > > goal with pegasos2 is to be able to run with KVM on PPC hardware > > > > eventually.) > > >=20 > > > Right, you might need an alternative method eventually.=A0 Really any > > > illegal instruction for your cpu is a possible candidate.=A0 Bear in > > > mind that this is *not* truly a hypercall interface, instead it's > > > something we're special casing for the purposes of faking the > > > firmware. > > >=20 > > > The "attn" instruction used on BookE might be a reasonable candidate > > > (assuming it doesn't conflict with something on 32-bit BookS) - that's > > > often used for things like signalling the attention of hardware > > > debuggers, and this is somewhat akin. > > >=20 > > > Mostly it's just a matter of working out what would be least messy to > > > intercept in the TCG instruction decoding path. > >=20 > > I'll wait for the current ongoing reorganisations to settle for that. If > > an alternative is needed I was considering the interface used by Mac on > > Linux: > >=20 > > https://lists.nongnu.org/archive/html/qemu-ppc/2021-03/msg00047.html > >=20 > > becuase there are some paravirtual drivers I think that use these on Mac > > OS X so this might also be useful for that use case for Mac emulation. > > But that seems very similar just checking for magic values at a normal > > syscall which means all syscalls will be intercepted anyway. In that > > case if sc 1 does not interfere with normal sc instructions then it may > > be better to keep that as the invalid instruction we trap on. > >=20 > > > > But this also means that if that assert cannot be dropped or > > > > there may be other problems with sc 1 hypercalls then we maybe > > > > cannot have > > > > the same vof.bin and we'll need a separate version that I would lik= e to > > > > avoid if possible so if there's a simple way to keep it working or = make > > > > vof.bin use alternate hypercall method without needing a separate b= inary > > > > that would be the direction I'd tend to go. Even if we need a seoar= ate > > > > version I'd like to keep as much common as possible. > > > >=20 > > > > I've tested that the missing rtas is not the reason for getting > > > > no output > > > > via serial though, as even when disabling rtas on pegasos2.rom > > > > it boots and > > > > I still get serial output just some PCI devices are not detected > > > > (such as > > > > USB, the video card and the not emulated ethernet port but these ar= e not > > > > fatal so it might even work as a first try without rtas, just to bo= ot a > > > > Linux kernel for testing it would be enough if I can fix the > > > > serial output). > > > > I still don't know why it's not finding serial but I think it > > > > may be some > > > > missing or wrong info in the device tree I generat. I'll try to foc= us on > > > > this for now and leave the above rtas question for later. > > >=20 > > > Oh.. another thought on that.=A0 You have an ISA serial port on Pegas= os, > > > I believe.=A0 I wonder if the PCI->ISA bridge needs some configuratio= n / > > > initialization that the firmware is expected to do.=A0 If so you'll n= eed > > > to mimic that setup in qemu for the VOF case. > >=20 > > That's what I begin to think because I've added everything to the device > > tree that I thought could be needed and I still don't get it working so > > it may need some config from the firmware. But how do I access device > > registers from board code? I've tried adding a machine reset method and > > write to memory mapped device registers but all my attempts failed. I've > > tried cpu_stl_le_data and even memory_region_dispatch_write but these > > did not get to the device. What's the way to access guest mmio regs from > > QEMU? >=20 > If we know that that serial is sitting behind PCI->ISA bridge (is it?), I Uh.. maybe. I think ISA bridges at least sometimes behave differently =66rom regular PCI devices or bridges, because legacy. Also note that it's probably IO space you need to map in, not MMIO space. > think you need to assign a BAR to that bridge, do some ISA setup (no idea > which) and enable that bridge (write MEMORY to PCI_COMMAND), this should > enable its registers. >=20 > In pseries we add "linux,pci-probe-only"=3D0 which makes Linux do all the > above instead of relying on the firmware doing BAR assignment. >=20 >=20 --=20 David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson --uFv4duyxikqw4fwP Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEdfRlhq5hpmzETofcbDjKyiDZs5IFAmCsiiUACgkQbDjKyiDZ s5KNUA//SDQxvzewyJaPyaadt9JlLB+Q9s5H1Wqu+1vyq7lOIYYy96ozrYhcAazN hERL5WwCKHnX73H3kCaKNFZ0RruPYO+SZR1q4Xw4UuDqqimjmR7ETMPDMCbpBzJw ZZc6KE31cFN4pvAr80Wows0o+RdJJiVYz7N1qaCCqHcLVwIHOUKCDscSHt2SRuDg HV1gQS5ZYjED+4cfR/q877QlzugtwrOTyA2GLuzpNspXDsbg7SNDxj63n6judF/c K9vgkZTPeCe77pE4ekUCqQdSAGB6lcntwilYCJOw2TD/4+9Fx3Et4bNsVpyYPNcC W1HdVGHa0WC4RPOfbLQzS9rJBvp1xah2ZFHrXs/KzVmGlsgzJ+s2+YNDP6SYWHc6 tsfr9B6EpTOH0A4uFCur/MaY6tQ7+nbquDt/kwVrt0rfevtjvQZINBPoTsPembs5 JdXYBrLSDQeovQxdnEABVHnbqB3zjgcWVpJ3JU9WtcQPbhYUqzy9xjq+jNxmX9in DP5AuQGKxQnBsDOwLn7JE2JuBuKyUhK3AZRorwaL+FJ4MkoJAI1z7mwCq0e196cf ad0FEF964x0+BSNE8/D0xh1lAlS9DkxFnfIjlUMqCtbhL2Zi+awG1RNyVlxpvuBS 5aHoebRqXlwMLFZ56NrUsefY9afr0lIIAVRBm1YYXXLCJHPBZjo= =TgDB -----END PGP SIGNATURE----- --uFv4duyxikqw4fwP--