Linux-HyperV List
 help / color / mirror / Atom feed
From: Mukesh R <mrathor@linux.microsoft.com>
To: Michael Kelley <mhklinux@outlook.com>,
	"linux-hyperv@vger.kernel.org" <linux-hyperv@vger.kernel.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	Vitaly Kuznetsov <vkuznets@redhat.com>
Cc: "haiyangz@microsoft.com" <haiyangz@microsoft.com>,
	"wei.liu@kernel.org" <wei.liu@kernel.org>,
	"decui@microsoft.com" <decui@microsoft.com>,
	"kameroncarr@linux.microsoft.com"
	<kameroncarr@linux.microsoft.com>
Subject: Re: [PATCH V1 2/2] Drivers: hv: Fix bugs and issues around repeat hypercall
Date: Thu, 19 Nov 2026 17:31:26 -0800	[thread overview]
Message-ID: <e6bffec5-064c-aa37-6e24-11d90124805d@linux.microsoft.com> (raw)
In-Reply-To: <MN2PR02MB68639EE1AC00187FDA3D5A42D47B2@MN2PR02MB6863.namprd02.prod.outlook.com>

On 11/17/26 09:41, Michael Kelley wrote:
> From: Mukesh R <mrathor@linux.microsoft.com> Sent: Monday, November 16, 2026 6:29 PM
>>
>> On 11/15/26 15:48, Michael Kelley wrote:
>>> From: Mukesh R <mrathor@linux.microsoft.com> Sent: Friday, November 13, 2026 6:53 PM
>>>>
>>>> There are few issues/bugs around repeat hypercall:
>>>>    o If the hypercall fails, it could still have processed some entries.
>>>>      Fix places where it assumes none of the entries were processed and will
>>>>      try to undo entries already processed or print incorrect completions.
>>>>
>>>>    o Hypercall returns back with timeouts only for HW confidential VMs. In
>>>>      that case it should be retried instead of returning back. Returning back
>>>>      results in error conditions and take down of VMs in most cases.
>>>>
>>>>    o For non confidential VMs, the hypercall could be streamlined as it will
>>>>      never return back with timeouts.
>>>>
>>>>    o Rename HV_STATUS_TIME_OUT to HV_STATUS_TIMEOUT to match hypervisor
>>>>      public header file. There is no reason for it to be different.
>>>>
>>>>    o Minor formatting fix for checkpatch. Error printing missing in couple
>>>>      places when rep hypercall fails.
> 
> In my view, the changes to the call sites of hv_do_rep_hypercall() should
> be in a separate patch. These are just bugs in how the undo is handled
> in case of an error, and the fixes presumably want to backported.
> 
> Handling a TIMEOUT return should also be a separate patch. This is
> also a bug and the fix should be backported. The TIMEOUT is
> generated under a specific set of circumstances -- it's not a generic
> error. The patch should describe how and why those circumstances arise
> in the overall context of hypercall continuation. There's not any such
> documentation elsewhere, so the info should be captured in the
> patch commit message or code comments for future reference.
> 
> Finally, if the looping behavior in a normal VM is to be removed,
> that would be a third patch. Given that this changes long-standing
> behavior, some explanation of "why" is needed in the commit
> message. I don't think this change should be backported.

Well, turns out the timeouts need to be handled for all hypercalls,
not just repeat hypercalls.... so i think it is best to just split
the series. I'll send separate patches now. JFYI.

Thanks,
-Mukesh



>>>>
>>>> Signed-off-by: Mukesh R <mrathor@linux.microsoft.com>
>>>
>>> [snip]
>>>
>>>> diff --git a/include/asm-generic/mshyperv.h b/include/asm-generic/mshyperv.h
>>>> index 396bcb3db0bc..87c784b9daa6 100644
>>>> --- a/include/asm-generic/mshyperv.h
>>>> +++ b/include/asm-generic/mshyperv.h
>>>> @@ -28,6 +28,9 @@
>>>>
>>>>    #define VTPM_BASE_ADDRESS 0xfed40000
>>>>
>>>> +#define HV_REP_COUNT_MAX	\
>>>> +	(HV_HYPERCALL_REP_COMP_MASK >> HV_HYPERCALL_REP_COMP_OFFSET)
>>>
>>> Sashiko noted that this new #define is never used anywhere.
>>>
>>>> +
>>>>    enum hv_partition_type {
>>>>    	HV_PARTITION_TYPE_GUEST,
>>>>    	HV_PARTITION_TYPE_ROOT,
>>>> @@ -128,6 +131,11 @@ static inline bool hv_result_success(u64 status)
>>>>    	return hv_result(status) == HV_STATUS_SUCCESS;
>>>>    }
>>>>
>>>> +static inline bool hv_result_timeout(u64 status)
>>>> +{
>>>> +	return hv_result(status) == HV_STATUS_TIMEOUT;
>>>> +}
>>>> +
>>>>    static inline unsigned int hv_repcomp(u64 status)
>>>>    {
>>>>    	/* Bits [43:32] of status have 'Reps completed' data. */
>>>> @@ -135,45 +143,63 @@ static inline unsigned int hv_repcomp(u64 status)
>>>>    			 HV_HYPERCALL_REP_COMP_OFFSET;
>>>>    }
>>>>
>>>> -/*
>>>> - * Rep hypercalls. Callers of this functions are supposed to ensure that
>>>> - * rep_count, varhead_size, and rep_start comply with Hyper-V hypercall
>>>> - * definition.
>>>> - */
>>>> -static inline u64 hv_do_rep_hypercall_ex(u16 code, u16 rep_count,
>>>> -					 u16 varhead_size, u16 rep_start,
>>>> -					 void *input, void *output)
>>>> +static inline u64 hv_do_rep_hypercall_reg(u16 code, u16 rep_count,
>>>> +					  u16 varhead_size, u16 rep_start,
>>>> +					  void *input, void *output)
>>>>    {
>>>>    	u64 control = code;
>>>> -	u64 status;
>>>> -	u16 rep_comp;
>>>>
>>>>    	control |= (u64)varhead_size << HV_HYPERCALL_VARHEAD_OFFSET;
>>>>    	control |= (u64)rep_count << HV_HYPERCALL_REP_COMP_OFFSET;
>>>>    	control |= (u64)rep_start << HV_HYPERCALL_REP_START_OFFSET;
>>>>
>>>> -	do {
>>>
>>> Dropped the looping behavior here may be much more significant
>>> than just "streamlining" as described in the commit message. The looping
>>> behavior was included when rep hypercalls were originally added in
>>> commit 806c89273bab in kernel 4.14, long before any SNP/TDX/CCA
>>> functionality was added. So it was present for "normal" VMs doing
>>> hypercalls on x86/x64 the normal way. The looping ensured that
>>> rep hypercalls could not return with no errors but with the completed
>>> rep count less than the requested rep count.
>>>
>>> However, in my mind it *is* debatable whether that scenario could
>>> ever occur. The Hyper-V TLFS describes "Hypercall Continuation" [1]
>>> as essentially transparent to the caller. If a rep hypercall stops
>>> processing reps because of hitting the 50 microsecond time limit,
>>> it should restart and continue with the next set of reps without
>>> the caller having to loop. Commit 806c89273bab mentions
>>> hypercall continuation, but does not describe why the explicit
>>> looping was added or how that aligns with the TLFS statements.
>>> Vitaly -- it was a long time ago, but do you have any recollection?
>>>
>>> Have you gotten an explicit assurance from the Hyper-V team
>>> that the transparent hypercall continuation really does work,
>>> and the explicit looping has all along been unnecessary? This
>>
>> Correct, hence the patch. Waiting on confirmation that it is
>> same behavior all the way back to server 2016.
> 
> Sounds good. FWIW, old TLFS version 2.0A, which is specifically
> titled to cover Windows Server 2008 R2, has the paragraph
> about hypercall continuation. So that functionality was evidently
> present very early on. It is still curious that Vitaly found it
> necessary to include the looping behavior in the Linux guest
> code.
> 
>>
>> Basically, the hypervisor will just adjust the RIP back to the
>> vmcall upon internal timeouts, and reschedule the vcpu. But for
>> hw confidential vms, ie, tdx/snp/cca, it cannot adjust the RIP,
>> hence return back to the guest with TIMEOUT, so cvm has to do in
>> a loop.
>>
>> Now, for paravisor support, I was just told by a paravisor team
>> dev, that it will never return with TIMEOUTs. My view on that set
>> aside, I guess for paravisor we'll just call the _reg function
>> then.  Thanks for bringing attention to that...
> 
> Yes, that would be what I expected.
> 
>>
>>> assurance would be needed back to and including Windows
>>> Server 2016, which is now the oldest Hyper-V version supported
>>> by Linux. Also note that the rep hypercall code is architecture
>>> independent, so the transparent hypercall continuation would
>>> also need to work on arm64 when the hypercall is invoked via
>>> the standard path of using the HVC instruction (the TLFS doesn't
>>> cover arm64, so this is unclear).
>>
>> Both x86 and arm are same for this issue.
> 
> OK, good.
> 
>>
>>> If you have such assurance, then I'm good with the removal of
>>> the looping. But please make the removal a separate patch, with
>>> sufficient discussion in the commit message to justify why it's OK
>>> to do the removal (including that evidently the original commit
>>> misinterpreted how continuation worked). And instead of looping,
>>> perhaps do a sanity check that the completed rep count matches
>>> the requested rep count.
>>
>> Not sure if it's worth the overhead of sanity check, looks like
>> callers are checking completed and loop anyways... But I would
>> be OK with BUG(), since it would be a bug for that to happen. FWIW,
>> it's been running internally with couple months, and haven't seen
>> any issues.
>>
>>
>>> Without such assurance, I think the looping needs to stay. Callers
>>> are not prepared for a rep hypercall to return with no errors, but
>>> with the completed rep count less than the requested rep count.
>>>
>>> [1] https://learn.microsoft.com/en-us/virtualization/hyper-v-on-
>> windows/tlfs/hypercall-interface#hypercall-continuation
>>>
>>>> +	return hv_do_hypercall(control, input, output);
>>>> +}
>>>> +
>>>> +static inline u64 hv_do_rep_hypercall_cvm(u16 code, u16 rep_count,
>>>> +					  u16 varhead_size, u16 rep_start,
>>>> +					  void *input, void *output)
>>>> +{
>>>> +	u64 status, control = code;
>>>> +	u16 rep_comp = 0;
>>>> +
>>>> +	control |= (u64)varhead_size << HV_HYPERCALL_VARHEAD_OFFSET;
>>>> +	control |= (u64)rep_count << HV_HYPERCALL_REP_COMP_OFFSET;
>>>> +	control |= (u64)rep_start << HV_HYPERCALL_REP_START_OFFSET;
>>>> +
>>>> +	for (;;) {
>>>>    		status = hv_do_hypercall(control, input, output);
>>>> -		if (!hv_result_success(status))
>>>> +
>>>> +		if (!hv_result_timeout(status))
>>>>    			return status;
>>>>
>>>>    		rep_comp = hv_repcomp(status);
>>>> -
>>>>    		control &= ~HV_HYPERCALL_REP_START_MASK;
>>>>    		control |= (u64)rep_comp << HV_HYPERCALL_REP_START_OFFSET;
>>>>
>>>>    		touch_nmi_watchdog();
>>>> -	} while (rep_comp < rep_count);
>>>> +	}
>>>>
>>>>    	return status;
>>>
>>> And Sashiko noted that this "return status" is dead code. The only
>>> way to exit this function is via the "return status" up in the loop.
>>>
>>>>    }
>>>>
>>>> -/* For the typical case where rep_start is 0 */
>>>> +/*
>>>> + * Rep hypercalls for HW confidential VMs return back when timeouts occur,
>>>> + * whereas for regular VMs they return only on success or failure. In case
>>>> + * of successful completion, entire rep_count is done. But a hypercall could
>>>> + * fail with partial completions, hence callers must always check that
>>>> + * in case of failure. Lastly, a rep_count value of 0 is allowed for some
>>>> + * hypercalls.
>>>> + */
>>>>    static inline u64 hv_do_rep_hypercall(u16 code, u16 rep_count, u16 varhead_size,
>>>>    				      void *input, void *output)
>>>>    {
>>>> -	return hv_do_rep_hypercall_ex(code, rep_count, varhead_size, 0,
>>>> -				      input, output);
>>>> +	if (hv_isolation_type_hw())
>>>
>>> I don't think this is the right test, so let me explain. You don't describe the
>>> context for when HV_STATUS_TIMEOUT may be returned, and there's no
>>> TLFS documentation of this status, but I'm assuming the context is hypercall
>>> continuation. This isn't the typical "something isn't responding" timeout. I
>>> suspect it means the hypervisor hit the 50 us limit, and returned to the guest
>>> VM with some reps still to be done. In a normal VM when a hypercall is
>>> initiated by the hypercall page on x86/x64 or the HVC instruction on arm64,
>>> the transparent hypercall continuation mechanism makes a TIMEOUT status
>>> unnecessary. But with SNP, TDX, and CCA, hypercall invocation may be
>>> piggybacked on other transports, and transparent continuation isn't available.
>>> So Hyper-V returns HV_STATUS_TIMEOUT with the expectation that the
>>> invoking code will loop to do the remaining reps.
>>>
>>> The key factor is whether the hypercall is piggybacked on other transports.
>>> The tests in hv_isolation_type_hw() don't capture that. For example, if
>>> a paravisor is present, hv_isolation_type_snp() will return "true" even
>>> though hypercalls are invoked via the standard x86/x64 hypercall page.
>>> The hypercalls trap to the paravisor. If the paravisor is functioning properly,
>>> it should do transparent hypercall continuation just like the hypervisor would.
>>> But this is something to be confirmed with the hypervisor and paravisor teams.
>>>
>>> Piggybacking is done only when no paravisor is present, so the
>>> hv_do_rep_hypercall_cvm() path should be taken only in that circumstance.
>>> Your current code works, but it goes down the more expensive
>>> hv_do_rep_hypercall_cvm() path when it doesn't need to. (I'm not completely
>>> clear on how arm64 works with and without a paravisor, so my statement
>>> applies only to SNP and TDX. Need to sort out the CCA case with Kameron.)
>>>
>>> Given the non-trivial complexity in sorting all this out, I wonder if
>>> distinguishing the _cvm and _reg cases is really worth it. The current
>>> code for the _reg case is admittedly inefficient because it does work for
>>> the looping case before likely deciding that no looping is needed. But
>>> that could easily be fixed. When no looping is necessary, which is the
>>> vast majority of cases, the looping code could end up being only a
>>> couple of instructions slower than a separately coded non-looping
>>> version (I did some builds and looked at the generated code). Since
>>> there's overhead in deciding which to use, there may not be any
>>> performance win from having both versions.
>>
>> Well, to me it's correctness issue also. If the ABI says that it will
>> not return with TIMEOUTs for non-cvms, then adding loop just masks it
>> instead of opening a bug report against the hypervisor.
> 
> I see your point, but would not call it a "correctness issue".  A "correctness
> issue" is a case where the result is wrong, and there's no wrong result
> here. I've definitely fixed edge-case issues that might never occur in
> practice just in the interest of overall "correctness", but that's not this case.
> 
> Your point opens a different can-of-worms. The threat model for Linux
> guests on Hyper-V has historically been that the guest trusts the Hyper-V
> host to do the right thing. A buggy or malicious host can arbitrarily read
> or write guest memory, so there's really not much choice but for the
> guest to trust the host (of course, in a CoCo VM the threat model is
> different). As such, Linux code has never tried to fully validate what it
> gets back from Hyper-V. Various ad hoc validations have been added
> when it seemed helpful to do so, but it's not a priority or a directive.
> Doing such a validation here is not objectionable, but I wouldn't add
> a significant complexity to do so.
> 
> FWIW, here's pseudo-code for an alternate approach that doesn't
> require splitting into separate _reg and _cvm versions:
> 
> static inline u64 hv_do_rep_hypercall_ex(u16 code, u16 rep_count,
>                                            u16 varhead_size, u16 rep_start,
>                                            void *input, void *output)
> {
>          u64 status, control = code;
>          u16 rep_comp = 0;
> 
>          control |= (u64)varhead_size << HV_HYPERCALL_VARHEAD_OFFSET;
>          control |= (u64)rep_count << HV_HYPERCALL_REP_COMP_OFFSET;
>          control |= (u64)rep_start << HV_HYPERCALL_REP_START_OFFSET;
> 
>          for (;;) {
>                  status = hv_do_hypercall(control, input, output);
> 
>                  if (!hv_result_timeout(status))
>                          /* If result is success, could validate
>                             completed reps == rep_count - rep_start */
>                          return status;
> 
>                  if (! <hypercall is over alt transport> ) {
>                          /* Should not get TIMEOUT with normal calling seq */
>                          WARN();
>                          return HV_STATUS_INVALID_PARAMETER; /* or some such */
>                  }
> 
>                  rep_comp = hv_repcomp(status);
>                  control &= ~HV_HYPERCALL_REP_START_MASK;
>                  control |= (u64)rep_comp << HV_HYPERCALL_REP_START_OFFSET;
>          }
> }
> 
> In almost all cases, the result is not TIMEOUT, so the code returns
> immediately with no extra overhead except to check for TIMEOUT.
> If it *is* TIMEOUT, then go to the trouble to figure out whether
> TIMEOUT is valid. If the VM is using the normal hypercall invocation
> mechanism, flag that as a hypervisor bug and don't loop.
> 
>>
>> Hope that makes sense.
>>
>> On a side note, to solicit your thoughts, i am not sure i like the
>> touch_nmi_watchdog()... it appears the the WD is triggered in seconds,
>> and if a hypercall is that long, schedule() really should be done in
>> between. IOW, remove the touch, instead any very long running hypercalls
>> should call schedule(). Of course, that would be a different patch.
> 
> I agree. I would be fine with removing the touch_nmi_watchdog().
> 
> Michael


      parent reply	other threads:[~2026-11-20  1:31 UTC|newest]

Thread overview: 13+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-14  2:53 [PATCH V1 0/2] Hyper-V: Fix bugs and issues around repeat hypercall Mukesh R
2026-11-14  2:53 ` [PATCH V1 1/2] Drivers: hv: Add new function hv_isolation_type_hw() Mukesh R
2026-11-14  2:53 ` [PATCH V1 2/2] Drivers: hv: Fix bugs and issues around repeat hypercall Mukesh R
2026-11-14  3:18   ` sashiko-bot
2026-11-15 23:48   ` Michael Kelley
2026-11-16 16:53     ` Vitaly Kuznetsov
2026-11-17  2:29     ` Mukesh R
2026-11-17 17:41       ` Michael Kelley
2026-11-17 23:09         ` Mukesh R
2026-11-18 14:01           ` Vitaly Kuznetsov
2026-11-18 16:57             ` Michael Kelley
2026-11-19 10:28               ` Vitaly Kuznetsov
2026-11-20  1:31         ` Mukesh R [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=e6bffec5-064c-aa37-6e24-11d90124805d@linux.microsoft.com \
    --to=mrathor@linux.microsoft.com \
    --cc=decui@microsoft.com \
    --cc=haiyangz@microsoft.com \
    --cc=kameroncarr@linux.microsoft.com \
    --cc=linux-hyperv@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mhklinux@outlook.com \
    --cc=vkuznets@redhat.com \
    --cc=wei.liu@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox