From: Salil Mehta via <qemu-devel@nongnu.org> To: Jonathan Cameron <jonathan.cameron@huawei.com> Cc: "qemu-devel@nongnu.org" <qemu-devel@nongnu.org>, "qemu-arm@nongnu.org" <qemu-arm@nongnu.org>, "maz@kernel.org" <maz@kernel.org>, "jean-philippe@linaro.org" <jean-philippe@linaro.org>, "lpieralisi@kernel.org" <lpieralisi@kernel.org>, "peter.maydell@linaro.org" <peter.maydell@linaro.org>, "richard.henderson@linaro.org" <richard.henderson@linaro.org>, "imammedo@redhat.com" <imammedo@redhat.com>, "andrew.jones@linux.dev" <andrew.jones@linux.dev>, "david@redhat.com" <david@redhat.com>, "philmd@linaro.org" <philmd@linaro.org>, "eric.auger@redhat.com" <eric.auger@redhat.com>, "oliver.upton@linux.dev" <oliver.upton@linux.dev>, "pbonzini@redhat.com" <pbonzini@redhat.com>, "mst@redhat.com" <mst@redhat.com>, "will@kernel.org" <will@kernel.org>, "gshan@redhat.com" <gshan@redhat.com>, "rafael@kernel.org" <rafael@kernel.org>, "alex.bennee@linaro.org" <alex.bennee@linaro.org>, "linux@armlinux.org.uk" <linux@armlinux.org.uk>, "darren@os.amperecomputing.com" <darren@os.amperecomputing.com>, "ilkka@os.amperecomputing.com" <ilkka@os.amperecomputing.com>, "vishnu@os.amperecomputing.com" <vishnu@os.amperecomputing.com>, "karl.heubaum@oracle.com" <karl.heubaum@oracle.com>, "miguel.luis@oracle.com" <miguel.luis@oracle.com>, "salil.mehta@opnsrc.net" <salil.mehta@opnsrc.net>, zhukeqian <zhukeqian1@huawei.com>, "wangxiongfeng (C)" <wangxiongfeng2@huawei.com>, "wangyanan (Y)" <wangyanan55@huawei.com>, "jiakernel2@gmail.com" <jiakernel2@gmail.com>, "maobibo@loongson.cn" <maobibo@loongson.cn>, "lixianglai@loongson.cn" <lixianglai@loongson.cn>, Linuxarm <linuxarm@huawei.com> Subject: RE: [PATCH V2 01/10] accel/kvm: Extract common KVM vCPU {creation,parking} code Date: Tue, 3 Oct 2023 12:27:36 +0000 [thread overview] Message-ID: <630c00201fd34c8fa307bed9c25bc334@huawei.com> (raw) In-Reply-To: <20231003125111.00002013@Huawei.com> > From: Jonathan Cameron <jonathan.cameron@huawei.com> > Sent: Tuesday, October 3, 2023 12:51 PM > To: Salil Mehta <salil.mehta@huawei.com> > Cc: qemu-devel@nongnu.org; qemu-arm@nongnu.org; maz@kernel.org; jean- > philippe@linaro.org; lpieralisi@kernel.org; peter.maydell@linaro.org; > richard.henderson@linaro.org; imammedo@redhat.com; andrew.jones@linux.dev; > david@redhat.com; philmd@linaro.org; eric.auger@redhat.com; > oliver.upton@linux.dev; pbonzini@redhat.com; mst@redhat.com; > will@kernel.org; gshan@redhat.com; rafael@kernel.org; > alex.bennee@linaro.org; linux@armlinux.org.uk; > darren@os.amperecomputing.com; ilkka@os.amperecomputing.com; > vishnu@os.amperecomputing.com; karl.heubaum@oracle.com; > miguel.luis@oracle.com; salil.mehta@opnsrc.net; zhukeqian > <zhukeqian1@huawei.com>; wangxiongfeng (C) <wangxiongfeng2@huawei.com>; > wangyanan (Y) <wangyanan55@huawei.com>; jiakernel2@gmail.com; > maobibo@loongson.cn; lixianglai@loongson.cn; Linuxarm <linuxarm@huawei.com> > Subject: Re: [PATCH V2 01/10] accel/kvm: Extract common KVM vCPU > {creation,parking} code > > On Tue, 3 Oct 2023 12:05:11 +0100 > Salil Mehta <salil.mehta@huawei.com> wrote: > > > Hi Jonathan, > > > > > From: Jonathan Cameron <jonathan.cameron@huawei.com> > > > Sent: Monday, October 2, 2023 4:53 PM > > > To: Salil Mehta <salil.mehta@huawei.com> > > > Cc: qemu-devel@nongnu.org; qemu-arm@nongnu.org; maz@kernel.org; jean- > > > philippe@linaro.org; lpieralisi@kernel.org; peter.maydell@linaro.org; > > > richard.henderson@linaro.org; imammedo@redhat.com; andrew.jones@linux.dev; > > > david@redhat.com; philmd@linaro.org; eric.auger@redhat.com; > > > oliver.upton@linux.dev; pbonzini@redhat.com; mst@redhat.com; > > > will@kernel.org; gshan@redhat.com; rafael@kernel.org; > > > alex.bennee@linaro.org; linux@armlinux.org.uk; > > > darren@os.amperecomputing.com; ilkka@os.amperecomputing.com; > > > vishnu@os.amperecomputing.com; karl.heubaum@oracle.com; > > > miguel.luis@oracle.com; salil.mehta@opnsrc.net; zhukeqian > > > <zhukeqian1@huawei.com>; wangxiongfeng (C) <wangxiongfeng2@huawei.com>; > > > wangyanan (Y) <wangyanan55@huawei.com>; jiakernel2@gmail.com; > > > maobibo@loongson.cn; lixianglai@loongson.cn; Linuxarm <linuxarm@huawei.com> > > > Subject: Re: [PATCH V2 01/10] accel/kvm: Extract common KVM vCPU > > > {creation,parking} code > > > > > > On Sat, 30 Sep 2023 01:19:24 +0100 > > > Salil Mehta <salil.mehta@huawei.com> wrote: > > > > > > > KVM vCPU creation is done once during the initialization of the VM when Qemu > > > > threads are spawned. This is common to all the architectures. > > > > > > > > Hot-unplug of vCPU results in destruction of the vCPU objects in QOM but > > > > the KVM vCPU objects in the Host KVM are not destroyed and their representative > > > > KVM vCPU objects/context in Qemu are parked. > > > > > > > > Refactor common logic so that some APIs could be reused by vCPU Hotplug code. > > > > > > > > Signed-off-by: Salil Mehta <salil.mehta@huawei.com> > > > > > > Hi Salil, > > > > > > A few trivial things inline, plus a question about why > > > cpu->cpu_index can now be used but kvm_arch_vcpu_id(cpu); > > > was previously needed. > > > > Good point. I used the API because it was returning > > 'unsigned long' and it was being used across the archs. > > I thought maybe the size of the index could vary across > > archs. For example, for PowerPC above API returns vcpu_id > > which presumably could have different data type than > > an 'integer'. > > > > But after Alex's comment, I was made to believe that this > > assumption might not be correct and CPU index is an > > 'integer' across archs and perhaps semantics of above > > API is not correct. > > > > But perhaps original code was functionally correct? > > I wasn't concerned with the type, but rather that the > value comes from other places than cpu->cpu_index > on some architectures. Sure, I meant there is a reason why type was chosen as 'unsigned long' and not an 'integer'. Perhaps the value can exceed the 'integer' size limits because of the way CPU index is being created on certain archs? If we try to put value from a larger container 'unsigned long' to a smaller container 'integer' things can go wrong. [...] > > > > > > > + vcpu->kvm_fd = cpu->kvm_fd; > > > > + QLIST_INSERT_HEAD(&kvm_state->kvm_parked_vcpus, vcpu, node); > > > > +} > > > > + > > > > +int kvm_create_vcpu(CPUState *cpu) > > > > +{ > > > > + int vcpu_id = cpu->cpu_index; > > > > > > See below. I'm not sure why it's safe not to use kvm_arch_vcpu_id() > > > Seems a few architectures have less than trivial implementations of > > > that function currently. > > > > I doubt this as well. Other architectures like PowerPC are returning > > different type? > > > It wasn't the type that bothered, me but rather that the source of > the data isn't always cpu->cpu_index so I have no idea if the values > are consistent. Got it. I meant 'unsigned long' return type in the kvm_arch_vcpu_id(). On some Architectures, the required container size for their vcpu-id could exceed an 'integer'. Sorry, for not making it clear. Thanks Salil.
WARNING: multiple messages have this Message-ID (diff)
From: Salil Mehta <salil.mehta@huawei.com> To: Jonathan Cameron <jonathan.cameron@huawei.com> Cc: "qemu-devel@nongnu.org" <qemu-devel@nongnu.org>, "qemu-arm@nongnu.org" <qemu-arm@nongnu.org>, "maz@kernel.org" <maz@kernel.org>, "jean-philippe@linaro.org" <jean-philippe@linaro.org>, "lpieralisi@kernel.org" <lpieralisi@kernel.org>, "peter.maydell@linaro.org" <peter.maydell@linaro.org>, "richard.henderson@linaro.org" <richard.henderson@linaro.org>, "imammedo@redhat.com" <imammedo@redhat.com>, "andrew.jones@linux.dev" <andrew.jones@linux.dev>, "david@redhat.com" <david@redhat.com>, "philmd@linaro.org" <philmd@linaro.org>, "eric.auger@redhat.com" <eric.auger@redhat.com>, "oliver.upton@linux.dev" <oliver.upton@linux.dev>, "pbonzini@redhat.com" <pbonzini@redhat.com>, "mst@redhat.com" <mst@redhat.com>, "will@kernel.org" <will@kernel.org>, "gshan@redhat.com" <gshan@redhat.com>, "rafael@kernel.org" <rafael@kernel.org>, "alex.bennee@linaro.org" <alex.bennee@linaro.org>, "linux@armlinux.org.uk" <linux@armlinux.org.uk>, "darren@os.amperecomputing.com" <darren@os.amperecomputing.com>, "ilkka@os.amperecomputing.com" <ilkka@os.amperecomputing.com>, "vishnu@os.amperecomputing.com" <vishnu@os.amperecomputing.com>, "karl.heubaum@oracle.com" <karl.heubaum@oracle.com>, "miguel.luis@oracle.com" <miguel.luis@oracle.com>, "salil.mehta@opnsrc.net" <salil.mehta@opnsrc.net>, zhukeqian <zhukeqian1@huawei.com>, "wangxiongfeng (C)" <wangxiongfeng2@huawei.com>, "wangyanan (Y)" <wangyanan55@huawei.com>, "jiakernel2@gmail.com" <jiakernel2@gmail.com>, "maobibo@loongson.cn" <maobibo@loongson.cn>, "lixianglai@loongson.cn" <lixianglai@loongson.cn>, Linuxarm <linuxarm@huawei.com> Subject: RE: [PATCH V2 01/10] accel/kvm: Extract common KVM vCPU {creation,parking} code Date: Tue, 3 Oct 2023 12:27:36 +0000 [thread overview] Message-ID: <630c00201fd34c8fa307bed9c25bc334@huawei.com> (raw) Message-ID: <20231003122736.WDof4EUzgnMaGX493R6pPG5O11eCtK9s72DxwnrflHU@z> (raw) In-Reply-To: <20231003125111.00002013@Huawei.com> > From: Jonathan Cameron <jonathan.cameron@huawei.com> > Sent: Tuesday, October 3, 2023 12:51 PM > To: Salil Mehta <salil.mehta@huawei.com> > Cc: qemu-devel@nongnu.org; qemu-arm@nongnu.org; maz@kernel.org; jean- > philippe@linaro.org; lpieralisi@kernel.org; peter.maydell@linaro.org; > richard.henderson@linaro.org; imammedo@redhat.com; andrew.jones@linux.dev; > david@redhat.com; philmd@linaro.org; eric.auger@redhat.com; > oliver.upton@linux.dev; pbonzini@redhat.com; mst@redhat.com; > will@kernel.org; gshan@redhat.com; rafael@kernel.org; > alex.bennee@linaro.org; linux@armlinux.org.uk; > darren@os.amperecomputing.com; ilkka@os.amperecomputing.com; > vishnu@os.amperecomputing.com; karl.heubaum@oracle.com; > miguel.luis@oracle.com; salil.mehta@opnsrc.net; zhukeqian > <zhukeqian1@huawei.com>; wangxiongfeng (C) <wangxiongfeng2@huawei.com>; > wangyanan (Y) <wangyanan55@huawei.com>; jiakernel2@gmail.com; > maobibo@loongson.cn; lixianglai@loongson.cn; Linuxarm <linuxarm@huawei.com> > Subject: Re: [PATCH V2 01/10] accel/kvm: Extract common KVM vCPU > {creation,parking} code > > On Tue, 3 Oct 2023 12:05:11 +0100 > Salil Mehta <salil.mehta@huawei.com> wrote: > > > Hi Jonathan, > > > > > From: Jonathan Cameron <jonathan.cameron@huawei.com> > > > Sent: Monday, October 2, 2023 4:53 PM > > > To: Salil Mehta <salil.mehta@huawei.com> > > > Cc: qemu-devel@nongnu.org; qemu-arm@nongnu.org; maz@kernel.org; jean- > > > philippe@linaro.org; lpieralisi@kernel.org; peter.maydell@linaro.org; > > > richard.henderson@linaro.org; imammedo@redhat.com; andrew.jones@linux.dev; > > > david@redhat.com; philmd@linaro.org; eric.auger@redhat.com; > > > oliver.upton@linux.dev; pbonzini@redhat.com; mst@redhat.com; > > > will@kernel.org; gshan@redhat.com; rafael@kernel.org; > > > alex.bennee@linaro.org; linux@armlinux.org.uk; > > > darren@os.amperecomputing.com; ilkka@os.amperecomputing.com; > > > vishnu@os.amperecomputing.com; karl.heubaum@oracle.com; > > > miguel.luis@oracle.com; salil.mehta@opnsrc.net; zhukeqian > > > <zhukeqian1@huawei.com>; wangxiongfeng (C) <wangxiongfeng2@huawei.com>; > > > wangyanan (Y) <wangyanan55@huawei.com>; jiakernel2@gmail.com; > > > maobibo@loongson.cn; lixianglai@loongson.cn; Linuxarm <linuxarm@huawei.com> > > > Subject: Re: [PATCH V2 01/10] accel/kvm: Extract common KVM vCPU > > > {creation,parking} code > > > > > > On Sat, 30 Sep 2023 01:19:24 +0100 > > > Salil Mehta <salil.mehta@huawei.com> wrote: > > > > > > > KVM vCPU creation is done once during the initialization of the VM when Qemu > > > > threads are spawned. This is common to all the architectures. > > > > > > > > Hot-unplug of vCPU results in destruction of the vCPU objects in QOM but > > > > the KVM vCPU objects in the Host KVM are not destroyed and their representative > > > > KVM vCPU objects/context in Qemu are parked. > > > > > > > > Refactor common logic so that some APIs could be reused by vCPU Hotplug code. > > > > > > > > Signed-off-by: Salil Mehta <salil.mehta@huawei.com> > > > > > > Hi Salil, > > > > > > A few trivial things inline, plus a question about why > > > cpu->cpu_index can now be used but kvm_arch_vcpu_id(cpu); > > > was previously needed. > > > > Good point. I used the API because it was returning > > 'unsigned long' and it was being used across the archs. > > I thought maybe the size of the index could vary across > > archs. For example, for PowerPC above API returns vcpu_id > > which presumably could have different data type than > > an 'integer'. > > > > But after Alex's comment, I was made to believe that this > > assumption might not be correct and CPU index is an > > 'integer' across archs and perhaps semantics of above > > API is not correct. > > > > But perhaps original code was functionally correct? > > I wasn't concerned with the type, but rather that the > value comes from other places than cpu->cpu_index > on some architectures. Sure, I meant there is a reason why type was chosen as 'unsigned long' and not an 'integer'. Perhaps the value can exceed the 'integer' size limits because of the way CPU index is being created on certain archs? If we try to put value from a larger container 'unsigned long' to a smaller container 'integer' things can go wrong. [...] > > > > > > > + vcpu->kvm_fd = cpu->kvm_fd; > > > > + QLIST_INSERT_HEAD(&kvm_state->kvm_parked_vcpus, vcpu, node); > > > > +} > > > > + > > > > +int kvm_create_vcpu(CPUState *cpu) > > > > +{ > > > > + int vcpu_id = cpu->cpu_index; > > > > > > See below. I'm not sure why it's safe not to use kvm_arch_vcpu_id() > > > Seems a few architectures have less than trivial implementations of > > > that function currently. > > > > I doubt this as well. Other architectures like PowerPC are returning > > different type? > > > It wasn't the type that bothered, me but rather that the source of > the data isn't always cpu->cpu_index so I have no idea if the values > are consistent. Got it. I meant 'unsigned long' return type in the kvm_arch_vcpu_id(). On some Architectures, the required container size for their vcpu-id could exceed an 'integer'. Sorry, for not making it clear. Thanks Salil.
next prev parent reply other threads:[~2023-10-03 12:28 UTC|newest] Thread overview: 81+ messages / expand[flat|nested] mbox.gz Atom feed top 2023-09-30 0:19 [PATCH V2 00/10] Add architecture agnostic code to support vCPU Hotplug Salil Mehta via 2023-09-30 0:19 ` [PATCH V2 01/10] accel/kvm: Extract common KVM vCPU {creation, parking} code Salil Mehta via 2023-10-02 15:53 ` [PATCH V2 01/10] accel/kvm: Extract common KVM vCPU {creation,parking} code Jonathan Cameron via 2023-10-02 15:53 ` Jonathan Cameron 2023-10-03 11:05 ` Salil Mehta via 2023-10-03 11:05 ` Salil Mehta 2023-10-03 11:51 ` Jonathan Cameron via 2023-10-03 11:51 ` Jonathan Cameron 2023-10-03 12:27 ` Salil Mehta via [this message] 2023-10-03 12:27 ` Salil Mehta 2023-10-02 23:17 ` Gavin Shan 2023-10-03 11:22 ` Salil Mehta via 2023-10-03 11:22 ` Salil Mehta 2023-09-30 0:19 ` [PATCH V2 02/10] hw/acpi: Move CPU ctrl-dev MMIO region len macro to common header file Salil Mehta via 2023-10-02 15:54 ` Jonathan Cameron via 2023-10-02 15:54 ` Jonathan Cameron 2023-10-03 11:24 ` Salil Mehta via 2023-10-03 11:24 ` Salil Mehta 2023-10-02 23:19 ` Gavin Shan 2023-10-03 11:24 ` Salil Mehta via 2023-10-03 11:24 ` Salil Mehta 2023-09-30 0:19 ` [PATCH V2 03/10] hw/acpi: Add ACPI CPU hotplug init stub Salil Mehta via 2023-10-02 16:00 ` Jonathan Cameron via 2023-10-02 16:00 ` Jonathan Cameron 2023-10-03 11:27 ` Salil Mehta via 2023-10-03 11:27 ` Salil Mehta 2023-10-02 23:25 ` Gavin Shan 2023-10-03 11:28 ` Salil Mehta via 2023-10-03 11:28 ` Salil Mehta 2023-09-30 0:19 ` [PATCH V2 04/10] hw/acpi: Init GED framework with cpu hotplug events Salil Mehta via 2023-10-02 16:07 ` Jonathan Cameron via 2023-10-02 16:07 ` Jonathan Cameron 2023-10-03 11:29 ` Salil Mehta via 2023-10-03 11:29 ` Salil Mehta 2023-10-02 23:28 ` Gavin Shan 2023-10-03 11:30 ` Salil Mehta via 2023-10-03 11:30 ` Salil Mehta 2023-09-30 0:19 ` [PATCH V2 05/10] hw/acpi: Update CPUs AML with cpu-(ctrl)dev change Salil Mehta via 2023-10-02 16:09 ` Jonathan Cameron via 2023-10-02 16:09 ` Jonathan Cameron 2023-10-03 11:31 ` Salil Mehta via 2023-10-03 11:31 ` Salil Mehta 2023-10-03 0:09 ` Gavin Shan 2023-10-03 11:33 ` Salil Mehta via 2023-10-03 11:33 ` Salil Mehta 2023-09-30 0:19 ` [PATCH V2 06/10] hw/acpi: Update GED _EVT method AML with cpu scan Salil Mehta via 2023-10-02 16:14 ` Jonathan Cameron via 2023-10-02 16:14 ` Jonathan Cameron 2023-10-03 11:43 ` Salil Mehta via 2023-10-03 11:43 ` Salil Mehta 2023-10-03 11:53 ` Jonathan Cameron via 2023-10-03 11:53 ` Jonathan Cameron 2023-10-03 12:13 ` Salil Mehta via 2023-10-03 12:13 ` Salil Mehta 2023-10-03 0:10 ` Gavin Shan 2023-10-03 11:43 ` Salil Mehta via 2023-10-03 11:43 ` Salil Mehta 2023-09-30 0:19 ` [PATCH V2 07/10] hw/acpi: Update ACPI GED framework to support vCPU Hotplug Salil Mehta via 2023-10-02 16:16 ` Jonathan Cameron via 2023-10-02 16:16 ` Jonathan Cameron 2023-10-03 11:44 ` Salil Mehta via 2023-10-03 11:44 ` Salil Mehta 2023-10-03 0:11 ` Gavin Shan 2023-10-03 11:45 ` Salil Mehta via 2023-10-03 11:45 ` Salil Mehta 2023-09-30 0:19 ` [PATCH V2 08/10] physmem: Add helper function to destroy CPU AddressSpace Salil Mehta via 2023-10-02 16:20 ` Jonathan Cameron via 2023-10-02 16:20 ` Jonathan Cameron 2023-10-03 11:46 ` Salil Mehta via 2023-10-03 11:46 ` Salil Mehta 2023-10-03 1:36 ` Gavin Shan 2023-10-03 11:54 ` Salil Mehta via 2023-10-03 11:54 ` Salil Mehta 2023-10-04 10:48 ` Salil Mehta via 2023-10-04 10:48 ` Salil Mehta 2023-09-30 0:19 ` [PATCH V2 09/10] gdbstub: Add helper function to unregister GDB register space Salil Mehta via 2023-10-03 3:16 ` Gavin Shan 2023-10-03 11:56 ` Salil Mehta via 2023-10-03 11:56 ` Salil Mehta 2023-09-30 0:19 ` [PATCH V2 10/10] target/arm/kvm: Write CPU state back to KVM on reset Salil Mehta via 2023-10-03 3:54 ` Gavin Shan
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=630c00201fd34c8fa307bed9c25bc334@huawei.com \ --to=qemu-devel@nongnu.org \ --cc=alex.bennee@linaro.org \ --cc=andrew.jones@linux.dev \ --cc=darren@os.amperecomputing.com \ --cc=david@redhat.com \ --cc=eric.auger@redhat.com \ --cc=gshan@redhat.com \ --cc=ilkka@os.amperecomputing.com \ --cc=imammedo@redhat.com \ --cc=jean-philippe@linaro.org \ --cc=jiakernel2@gmail.com \ --cc=jonathan.cameron@huawei.com \ --cc=karl.heubaum@oracle.com \ --cc=linux@armlinux.org.uk \ --cc=linuxarm@huawei.com \ --cc=lixianglai@loongson.cn \ --cc=lpieralisi@kernel.org \ --cc=maobibo@loongson.cn \ --cc=maz@kernel.org \ --cc=miguel.luis@oracle.com \ --cc=mst@redhat.com \ --cc=oliver.upton@linux.dev \ --cc=pbonzini@redhat.com \ --cc=peter.maydell@linaro.org \ --cc=philmd@linaro.org \ --cc=qemu-arm@nongnu.org \ --cc=rafael@kernel.org \ --cc=richard.henderson@linaro.org \ --cc=salil.mehta@huawei.com \ --cc=salil.mehta@opnsrc.net \ --cc=vishnu@os.amperecomputing.com \ --cc=wangxiongfeng2@huawei.com \ --cc=wangyanan55@huawei.com \ --cc=will@kernel.org \ --cc=zhukeqian1@huawei.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).