[PATCH] KVM: s390: vsie: retry SIE instruction on host intercepts

[PATCH] KVM: s390: vsie: retry SIE instruction on host intercepts

[Eric Farman]

It's possible that SIE exits for work that the host needs to perform
rather than something that is intended for the guest.

A Linux guest will ignore this intercept code since there is nothing
for it to do, but a more robust solution would rewind the PSW back to
the SIE instruction. This will transparently resume the guest once
the host completes its work, without the guest needing to process
what is effectively a NOP and re-issue SIE itself.

Signed-off-by: Eric Farman <farman@linux.ibm.com>
---
 arch/s390/kvm/vsie.c | 22 +++++++++++++++++++---
 1 file changed, 19 insertions(+), 3 deletions(-)

diff --git a/arch/s390/kvm/vsie.c b/arch/s390/kvm/vsie.c
index 8f0f944f81c2..63aae70a6790 100644
--- a/arch/s390/kvm/vsie.c
+++ b/arch/s390/kvm/vsie.c
@@ -1306,10 +1306,24 @@ static int vsie_run(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
 
 		if (rc == -EAGAIN)
 			rc = 0;
-		if (rc || scb_s->icptcode || signal_pending(current) ||
+
+		/*
+		 * Exit the loop if the guest needs to process the intercept
+		 */
+		if (rc || scb_s->icptcode)
+			break;
+
+		/*
+		 * Exit the loop if the host needs to process an intercept,
+		 * but rewind the PSW to re-enter SIE once that's completed
+		 * instead of passing a "no action" intercept to the guest.
+		 */
+		if (signal_pending(current) ||
 		    kvm_s390_vcpu_has_irq(vcpu, 0) ||
-		    kvm_s390_vcpu_sie_inhibited(vcpu))
+		    kvm_s390_vcpu_sie_inhibited(vcpu)) {
+			kvm_s390_rewind_psw(vcpu, 4);
 			break;
+		}
 		cond_resched();
 	}
 
@@ -1428,8 +1442,10 @@ int kvm_s390_handle_vsie(struct kvm_vcpu *vcpu)
 		return kvm_s390_inject_program_int(vcpu, PGM_SPECIFICATION);
 
 	if (signal_pending(current) || kvm_s390_vcpu_has_irq(vcpu, 0) ||
-	    kvm_s390_vcpu_sie_inhibited(vcpu))
+	    kvm_s390_vcpu_sie_inhibited(vcpu)) {
+		kvm_s390_rewind_psw(vcpu, 4);
 		return 0;
+	}
 
 	vsie_page = get_vsie_page(vcpu->kvm, scb_addr);
 	if (IS_ERR(vsie_page))
-- 
2.40.1

Re: [PATCH] KVM: s390: vsie: retry SIE instruction on host intercepts

[David Hildenbrand]

On 01.03.24 21:43, Eric Farman wrote:
> It's possible that SIE exits for work that the host needs to perform
> rather than something that is intended for the guest.
> 
> A Linux guest will ignore this intercept code since there is nothing
> for it to do, but a more robust solution would rewind the PSW back to
> the SIE instruction. This will transparently resume the guest once
> the host completes its work, without the guest needing to process
> what is effectively a NOP and re-issue SIE itself.

I recall that 0-intercepts are valid by the architecture. Further, I 
recall that there were some rather tricky corner cases where avoiding 
0-intercepts would not be that easy.

Now, it's been a while ago, and maybe I misremember. SoI'm trusting 
people with access to documentation can review this.

-- 
Cheers,

David / dhildenb

Re: [PATCH] KVM: s390: vsie: retry SIE instruction on host intercepts

[Christian Borntraeger]

Am 04.03.24 um 09:35 schrieb David Hildenbrand:
> On 01.03.24 21:43, Eric Farman wrote:
>> It's possible that SIE exits for work that the host needs to perform
>> rather than something that is intended for the guest.
>>
>> A Linux guest will ignore this intercept code since there is nothing
>> for it to do, but a more robust solution would rewind the PSW back to
>> the SIE instruction. This will transparently resume the guest once
>> the host completes its work, without the guest needing to process
>> what is effectively a NOP and re-issue SIE itself.
> 
> I recall that 0-intercepts are valid by the architecture. Further, I recall that there were some rather tricky corner cases where avoiding 0-intercepts would not be that easy.
> 
> Now, it's been a while ago, and maybe I misremember. SoI'm trusting people with access to documentation can review this.

Yes, 0-intercepts are allowed, and this also happens when LPAR has an exit.
So this patch is not necessary, the question is if this would be an valuable optimization?

Re: [PATCH] KVM: s390: vsie: retry SIE instruction on host intercepts

[Eric Farman]

On Mon, 2024-03-04 at 09:44 +0100, Christian Borntraeger wrote:
> 
> 
> Am 04.03.24 um 09:35 schrieb David Hildenbrand:
> > On 01.03.24 21:43, Eric Farman wrote:
> > > It's possible that SIE exits for work that the host needs to
> > > perform
> > > rather than something that is intended for the guest.
> > > 
> > > A Linux guest will ignore this intercept code since there is
> > > nothing
> > > for it to do, but a more robust solution would rewind the PSW
> > > back to
> > > the SIE instruction. This will transparently resume the guest
> > > once
> > > the host completes its work, without the guest needing to process
> > > what is effectively a NOP and re-issue SIE itself.
> > 
> > I recall that 0-intercepts are valid by the architecture. Further,
> > I recall that there were some rather tricky corner cases where
> > avoiding 0-intercepts would not be that easy.

Any chance you recall any details of those corner cases? I can try to
chase some of them down.

> > 
> > Now, it's been a while ago, and maybe I misremember. SoI'm trusting
> > people with access to documentation can review this.
> 
> Yes, 0-intercepts are allowed, and this also happens when LPAR has an
> exit.

From an offline conversation I'd had some months back:

"""
The arch does allow ICODE=0 to be stored, but it's supposed to happen
only upon a host interruption -- in which case the old PSW is supposed
to point back at the SIE, to resume guest execution if the host should
LPSW oldPSW.
"""

> So this patch is not necessary, the question is if this would be an
> valuable optimization?

It's a reasonable question. I don't think I have a reasonable way of
measuring the exit, though. :/

Re: [PATCH] KVM: s390: vsie: retry SIE instruction on host intercepts

[Christian Borntraeger]

Am 04.03.24 um 16:37 schrieb Eric Farman:
> On Mon, 2024-03-04 at 09:44 +0100, Christian Borntraeger wrote:
>>
>>
>> Am 04.03.24 um 09:35 schrieb David Hildenbrand:
>>> On 01.03.24 21:43, Eric Farman wrote:
>>>> It's possible that SIE exits for work that the host needs to
>>>> perform
>>>> rather than something that is intended for the guest.
>>>>
>>>> A Linux guest will ignore this intercept code since there is
>>>> nothing
>>>> for it to do, but a more robust solution would rewind the PSW
>>>> back to
>>>> the SIE instruction. This will transparently resume the guest
>>>> once
>>>> the host completes its work, without the guest needing to process
>>>> what is effectively a NOP and re-issue SIE itself.
>>>
>>> I recall that 0-intercepts are valid by the architecture. Further,
>>> I recall that there were some rather tricky corner cases where
>>> avoiding 0-intercepts would not be that easy.
> 
> Any chance you recall any details of those corner cases? I can try to
> chase some of them down.
> 
>>>
>>> Now, it's been a while ago, and maybe I misremember. SoI'm trusting
>>> people with access to documentation can review this.
>>
>> Yes, 0-intercepts are allowed, and this also happens when LPAR has an
>> exit.
> 
>  From an offline conversation I'd had some months back:
> 
> """
> The arch does allow ICODE=0 to be stored, but it's supposed to happen
> only upon a host interruption -- in which case the old PSW is supposed
> to point back at the SIE, to resume guest execution if the host should
> LPSW oldPSW.
> """

Just re-read the architecture again and I agree, the SIE instruction should
be nullified. So we should go forward with this somehow.

Eric, can you maybe add this to devel for CI coverage so that we see if there
are corner cases? Maybe also try to do some performance things (how many IPIs
can we get in guest2 when a guest3 is running and how many IPIs are possible
in a guest3).

Re: [PATCH] KVM: s390: vsie: retry SIE instruction on host intercepts

[Eric Farman]

On Mon, 2024-04-29 at 12:18 +0200, Christian Borntraeger wrote:
> Am 04.03.24 um 16:37 schrieb Eric Farman:
> > On Mon, 2024-03-04 at 09:44 +0100, Christian Borntraeger wrote:
> > > 
> > > 
> > > Am 04.03.24 um 09:35 schrieb David Hildenbrand:
> > > > On 01.03.24 21:43, Eric Farman wrote:
> > > > > It's possible that SIE exits for work that the host needs to
> > > > > perform
> > > > > rather than something that is intended for the guest.
> > > > > 
> > > > > A Linux guest will ignore this intercept code since there is
> > > > > nothing
> > > > > for it to do, but a more robust solution would rewind the PSW
> > > > > back to
> > > > > the SIE instruction. This will transparently resume the guest
> > > > > once
> > > > > the host completes its work, without the guest needing to
> > > > > process
> > > > > what is effectively a NOP and re-issue SIE itself.
> > > > 
> > > > I recall that 0-intercepts are valid by the architecture.
> > > > Further,
> > > > I recall that there were some rather tricky corner cases where
> > > > avoiding 0-intercepts would not be that easy.
> > 
> > Any chance you recall any details of those corner cases? I can try
> > to
> > chase some of them down.
> > 
> > > > 
> > > > Now, it's been a while ago, and maybe I misremember. SoI'm
> > > > trusting
> > > > people with access to documentation can review this.
> > > 
> > > Yes, 0-intercepts are allowed, and this also happens when LPAR
> > > has an
> > > exit.
> > 
> >  From an offline conversation I'd had some months back:
> > 
> > """
> > The arch does allow ICODE=0 to be stored, but it's supposed to
> > happen
> > only upon a host interruption -- in which case the old PSW is
> > supposed
> > to point back at the SIE, to resume guest execution if the host
> > should
> > LPSW oldPSW.
> > """
> 
> Just re-read the architecture again and I agree, the SIE instruction
> should
> be nullified. So we should go forward with this somehow.
> 
> Eric, can you maybe add this to devel for CI coverage so that we see
> if there
> are corner cases? 

Sure thing.

> Maybe also try to do some performance things (how many IPIs
> can we get in guest2 when a guest3 is running and how many IPIs are
> possible
> in a guest3).
> 

Fair enough. I'll see if I can come up with something and report back
here.

Re: [PATCH] KVM: s390: vsie: retry SIE instruction on host intercepts

[Janosch Frank]

On 4/29/24 12:18, Christian Borntraeger wrote:
> Am 04.03.24 um 16:37 schrieb Eric Farman:
>> On Mon, 2024-03-04 at 09:44 +0100, Christian Borntraeger wrote:
>>>
>>>
>>> Am 04.03.24 um 09:35 schrieb David Hildenbrand:
>>>> On 01.03.24 21:43, Eric Farman wrote:
>>>>> It's possible that SIE exits for work that the host needs to
>>>>> perform
>>>>> rather than something that is intended for the guest.
>>>>>
>>>>> A Linux guest will ignore this intercept code since there is
>>>>> nothing
>>>>> for it to do, but a more robust solution would rewind the PSW
>>>>> back to
>>>>> the SIE instruction. This will transparently resume the guest
>>>>> once
>>>>> the host completes its work, without the guest needing to process
>>>>> what is effectively a NOP and re-issue SIE itself.
>>>>
>>>> I recall that 0-intercepts are valid by the architecture. Further,
>>>> I recall that there were some rather tricky corner cases where
>>>> avoiding 0-intercepts would not be that easy.
>>
>> Any chance you recall any details of those corner cases? I can try to
>> chase some of them down.
>>
>>>>
>>>> Now, it's been a while ago, and maybe I misremember. SoI'm trusting
>>>> people with access to documentation can review this.
>>>
>>> Yes, 0-intercepts are allowed, and this also happens when LPAR has an
>>> exit.
>>
>>   From an offline conversation I'd had some months back:
>>
>> """
>> The arch does allow ICODE=0 to be stored, but it's supposed to happen
>> only upon a host interruption -- in which case the old PSW is supposed
>> to point back at the SIE, to resume guest execution if the host should
>> LPSW oldPSW.
>> """
> 
> Just re-read the architecture again and I agree, the SIE instruction should
> be nullified. So we should go forward with this somehow.
> 
> Eric, can you maybe add this to devel for CI coverage so that we see if there
> are corner cases? Maybe also try to do some performance things (how many IPIs
> can we get in guest2 when a guest3 is running and how many IPIs are possible
> in a guest3).
> 
> 

This patch has had contact with the CI for quite a while and I'm 
gathering patches.
Is this an ack now?

Re: [PATCH] KVM: s390: vsie: retry SIE instruction on host intercepts

[Christian Borntraeger]

Am 03.07.24 um 16:59 schrieb Janosch Frank:
g>
> This patch has had contact with the CI for quite a while and I'm gathering patches.
> Is this an ack now?
yes

Acked-by: Christian Borntraeger <borntraeger@linux.ibm.com