From: Greg Kurz <groug@kaod.org>
To: "Cédric Le Goater" <clg@kaod.org>
Cc: qemu-ppc@nongnu.org, Gustavo Romero <gromero@linux.ibm.com>,
Satheesh Rajendran <sathnaga@linux.vnet.ibm.com>,
qemu-devel@nongnu.org, David Gibson <david@gibson.dropbear.id.au>
Subject: Re: [PATCH v2 3/4] spapr/xive: Allocate IPIs independently from the other sources
Date: Thu, 5 Nov 2020 12:39:34 +0100 [thread overview]
Message-ID: <20201105123934.50ad1c39@bahia.lan> (raw)
In-Reply-To: <2d837671-ec7a-2eb8-c9f3-10d031ee4fde@kaod.org>
On Thu, 5 Nov 2020 09:37:27 +0100
Cédric Le Goater <clg@kaod.org> wrote:
> On 8/20/20 3:45 PM, Cédric Le Goater wrote:
> > The vCPU IPIs are now allocated in kvmppc_xive_cpu_connect() when the
> > vCPU connects to the KVM device and not when all the sources are reset
> > in kvmppc_xive_source_reset()
>
> This patch is introducing a regression when vsmt is in used.
>
Well, it isn't exactly when vsmt is used, it is when vsmt is set
to a different value than the one which is passed to -smp threads,
otherwise you always get consecutive vcpu ids.
> https://bugs.launchpad.net/qemu/+bug/1900241
>
In this report we have threads=1, so depending on vsmt this gives
the following vcpu ids:
-M vsmt=1 -smp 8,cores=8,threads=1
kvmppc_xive_reset_ipi_on_cpu: cpu_index=0 vcpu_id=0
kvmppc_xive_reset_ipi_on_cpu: cpu_index=1 vcpu_id=1
kvmppc_xive_reset_ipi_on_cpu: cpu_index=2 vcpu_id=2
kvmppc_xive_reset_ipi_on_cpu: cpu_index=3 vcpu_id=3
kvmppc_xive_reset_ipi_on_cpu: cpu_index=4 vcpu_id=4
kvmppc_xive_reset_ipi_on_cpu: cpu_index=5 vcpu_id=5
kvmppc_xive_reset_ipi_on_cpu: cpu_index=6 vcpu_id=6
kvmppc_xive_reset_ipi_on_cpu: cpu_index=7 vcpu_id=7
-M vsmt=2 -smp 8,cores=8,threads=1
kvmppc_xive_reset_ipi_on_cpu: cpu_index=0 vcpu_id=0
kvmppc_xive_reset_ipi_on_cpu: cpu_index=1 vcpu_id=2
kvmppc_xive_reset_ipi_on_cpu: cpu_index=2 vcpu_id=4
kvmppc_xive_reset_ipi_on_cpu: cpu_index=3 vcpu_id=6
kvmppc_xive_reset_ipi_on_cpu: cpu_index=4 vcpu_id=8
kvmppc_xive_reset_ipi_on_cpu: cpu_index=5 vcpu_id=10
kvmppc_xive_reset_ipi_on_cpu: cpu_index=6 vcpu_id=12
kvmppc_xive_reset_ipi_on_cpu: cpu_index=7 vcpu_id=14
-M vsmt=4 -smp 8,cores=8,threads=1
kvmppc_xive_reset_ipi_on_cpu: cpu_index=0 vcpu_id=0
kvmppc_xive_reset_ipi_on_cpu: cpu_index=1 vcpu_id=4
kvmppc_xive_reset_ipi_on_cpu: cpu_index=2 vcpu_id=8
kvmppc_xive_reset_ipi_on_cpu: cpu_index=3 vcpu_id=12
kvmppc_xive_reset_ipi_on_cpu: cpu_index=4 vcpu_id=16
kvmppc_xive_reset_ipi_on_cpu: cpu_index=5 vcpu_id=20
kvmppc_xive_reset_ipi_on_cpu: cpu_index=6 vcpu_id=24
kvmppc_xive_reset_ipi_on_cpu: cpu_index=7 vcpu_id=28
-M vsmt=8 -smp 8,cores=8,threads=1
kvmppc_xive_reset_ipi_on_cpu: cpu_index=0 vcpu_id=0
kvmppc_xive_reset_ipi_on_cpu: cpu_index=1 vcpu_id=8
kvmppc_xive_reset_ipi_on_cpu: cpu_index=2 vcpu_id=16
kvmppc_xive_reset_ipi_on_cpu: cpu_index=3 vcpu_id=24
kvmppc_xive_reset_ipi_on_cpu: cpu_index=4 vcpu_id=32
kvmppc_xive_reset_ipi_on_cpu: cpu_index=5 vcpu_id=40
kvmppc_xive_reset_ipi_on_cpu: cpu_index=6 vcpu_id=48
kvmppc_xive_reset_ipi_on_cpu: cpu_index=7 vcpu_id=56
> when the OS boots, H_INT_SET_SOURCE_CONFIG fails with EINVAL, which
> should mean that the IPI is not created at the host/KVM level.
>
[...]
> > +static int kvmppc_xive_reset_ipi(SpaprXive *xive, CPUState *cs, Error **errp)
> > +{
> > + unsigned long ipi = kvm_arch_vcpu_id(cs);
>
> ( I am wondering if this is the correct id to use ? )
>
Setting the ipi to the vcpu id seems to assume that the vcpu ids are
consecutive, which is definitely not the case when vsmt != threads
as explained above.
Passing cs->cpu_index would provide consecutive ids but I'm not
sure this is a correct fix. I gave a try : all the vCPUs get
online in the guest as expected but something goes wrong when
terminating QEMU:
[ 5557.599881] WARNING: CPU: 40 PID: 59101 at arch/powerpc/kvm/book3s_xive_native.c:259 xive_native_esb_fault+0xe4/0x240 [kvm]
[ 5557.599897] Modules linked in: xt_CHECKSUM ipt_MASQUERADE xt_conntrack ipt_REJECT nf_reject_ipv4 nft_compat nft_counter nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables nfnetlink tun bridge stp llc nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache kvm_hv kvm i2c_dev sunrpc ext4 mbcache jbd2 xts vmx_crypto ofpart ipmi_powernv ipmi_devintf powernv_flash ipmi_msghandler mtd ibmpowernv opal_prd at24 uio_pdrv_genirq uio ip_tables xfs libcrc32c sd_mod sg ast i2c_algo_bit drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm ahci libahci tg3 libata drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
[ 5557.600010] CPU: 40 PID: 59101 Comm: qemu-system-ppc Kdump: loaded Tainted: G W --------- - - 4.18.0-240.el8.ppc64le #1
[ 5557.600041] NIP: c00800000e949fac LR: c00000000044b164 CTR: c00800000e949ec8
[ 5557.600060] REGS: c000001f69617840 TRAP: 0700 Tainted: G W --------- - - (4.18.0-240.el8.ppc64le)
[ 5557.600089] MSR: 9000000000029033 <SF,HV,EE,ME,IR,DR,RI,LE> CR: 44044282 XER: 00000000
[ 5557.600110] CFAR: c00000000044b160 IRQMASK: 0
[ 5557.600110] GPR00: c00000000044b164 c000001f69617ac0 c00800000e96e000 c000001f69617c10
[ 5557.600110] GPR04: 05faa2b21e000080 0000000000000000 0000000000000005 ffffffffffffffff
[ 5557.600110] GPR08: 0000000000000000 0000000000000001 0000000000000000 0000000000000001
[ 5557.600110] GPR12: c00800000e949ec8 c000001ffffd3400 0000000000000000 0000000000000000
[ 5557.600110] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 5557.600110] GPR20: 0000000000000000 0000000000000000 c000001f5c065160 c000000001c76f90
[ 5557.600110] GPR24: c000001f06f20000 c000001f5c065100 0000000000000008 c000001f0eb98c78
[ 5557.600110] GPR28: c000001dcab40000 c000001dcab403d8 c000001f69617c10 0000000000000011
[ 5557.600255] NIP [c00800000e949fac] xive_native_esb_fault+0xe4/0x240 [kvm]
[ 5557.600274] LR [c00000000044b164] __do_fault+0x64/0x220
[ 5557.600298] Call Trace:
[ 5557.600302] [c000001f69617ac0] [0000000137a5dc20] 0x137a5dc20 (unreliable)
[ 5557.600320] [c000001f69617b50] [c00000000044b164] __do_fault+0x64/0x220
[ 5557.600337] [c000001f69617b90] [c000000000453838] do_fault+0x218/0x930
[ 5557.600355] [c000001f69617bf0] [c000000000456f50] __handle_mm_fault+0x350/0xdf0
[ 5557.600373] [c000001f69617cd0] [c000000000457b1c] handle_mm_fault+0x12c/0x310
[ 5557.600393] [c000001f69617d10] [c00000000007ef44] __do_page_fault+0x264/0xbb0
[ 5557.600411] [c000001f69617df0] [c00000000007f8c8] do_page_fault+0x38/0xd0
[ 5557.600429] [c000001f69617e30] [c00000000000a714] handle_page_fault+0x18/0x38
[ 5557.600438] Instruction dump:
[ 5557.600444] 40c2fff0 7c2004ac 2fa90000 409e0118 73e90001 41820080 e8bd0008 7c2004ac
[ 5557.600455] 7ca90074 39400000 915c0000 7929d182 <0b090000> 2fa50000 419e0080 e89e0018
[ 5557.600485] ---[ end trace 66c6ff034c53f64f ]---
[ 5557.600509] xive-kvm: xive_native_esb_fault: accessing invalid ESB page for source 8 !
So it looks like something needs to be done in the XIVE KVM device anyway.
[...]
> > static int kvmppc_xive_source_reset(XiveSource *xsrc, Error **errp)
> > {
> > SpaprXive *xive = SPAPR_XIVE(xsrc->xive);
> > int i;
> >
> > - for (i = 0; i < xsrc->nr_irqs; i++) {
> > + /*
> > + * Skip the vCPU IPIs. These are created/reset when the vCPUs are
> > + * connected in kvmppc_xive_cpu_connect()
> > + */
> > + for (i = SPAPR_XIRQ_BASE; i < xsrc->nr_irqs; i++) {
>
> This change skips the range [ 0x0 ... 0x1000 ] and relies on the presenter
> to create the vCPU IPIs at the KVM level. But spapr_irq_init() could have
> claimed more in :
>
> /* Enable the CPU IPIs */
> for (i = 0; i < nr_servers; ++i) {
> SpaprInterruptControllerClass *sicc
> = SPAPR_INTC_GET_CLASS(spapr->xive);
>
> if (sicc->claim_irq(SPAPR_INTC(spapr->xive), SPAPR_IRQ_IPI + i,
> false, errp) < 0) {
> return;
> }
> }
>
This doesn't reach the XIVE KVM device when running in dual mode because
it doesn't exist yet.
> I think this is what is happening with vsmt. However, I don't know how to
> fix it :/
>
> Thanks,
>
> C.
>
next prev parent reply other threads:[~2020-11-05 11:40 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-08-20 13:45 [PATCH v2 0/4] spapr/xive: Allocate vCPU IPIs from the vCPU contexts Cédric Le Goater
2020-08-20 13:45 ` [PATCH v2 1/4] spapr/xive: Modify kvm_cpu_is_enabled() interface Cédric Le Goater
2020-08-20 13:45 ` [PATCH v2 2/4] spapr/xive: Use kvmppc_xive_source_reset() in post_load Cédric Le Goater
2020-08-20 13:45 ` [PATCH v2 3/4] spapr/xive: Allocate IPIs independently from the other sources Cédric Le Goater
2020-11-05 8:37 ` Cédric Le Goater
2020-11-05 11:39 ` Greg Kurz [this message]
2020-08-20 13:45 ` [PATCH v2 4/4] spapr/xive: Allocate vCPU IPIs from the vCPU contexts Cédric Le Goater
2020-08-21 1:38 ` [PATCH v2 0/4] " David Gibson
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20201105123934.50ad1c39@bahia.lan \
--to=groug@kaod.org \
--cc=clg@kaod.org \
--cc=david@gibson.dropbear.id.au \
--cc=gromero@linux.ibm.com \
--cc=qemu-devel@nongnu.org \
--cc=qemu-ppc@nongnu.org \
--cc=sathnaga@linux.vnet.ibm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).