From: Mukesh R <mrathor@linux.microsoft.com>
To: Michael Kelley <mhklinux@outlook.com>,
"linux-hyperv@vger.kernel.org" <linux-hyperv@vger.kernel.org>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
Vitaly Kuznetsov <vkuznets@redhat.com>
Cc: "haiyangz@microsoft.com" <haiyangz@microsoft.com>,
"wei.liu@kernel.org" <wei.liu@kernel.org>,
"decui@microsoft.com" <decui@microsoft.com>,
"kameroncarr@linux.microsoft.com"
<kameroncarr@linux.microsoft.com>
Subject: Re: [PATCH V1 2/2] Drivers: hv: Fix bugs and issues around repeat hypercall
Date: Mon, 16 Nov 2026 18:29:22 -0800 [thread overview]
Message-ID: <fd130973-25d6-e706-6822-793f12ba2015@linux.microsoft.com> (raw)
In-Reply-To: <MN2PR02MB6863E6C4F59864B7833681FCD47D2@MN2PR02MB6863.namprd02.prod.outlook.com>
On 11/15/26 15:48, Michael Kelley wrote:
> From: Mukesh R <mrathor@linux.microsoft.com> Sent: Friday, November 13, 2026 6:53 PM
>>
>> There are few issues/bugs around repeat hypercall:
>> o If the hypercall fails, it could still have processed some entries.
>> Fix places where it assumes none of the entries were processed and will
>> try to undo entries already processed or print incorrect completions.
>>
>> o Hypercall returns back with timeouts only for HW confidential VMs. In
>> that case it should be retried instead of returning back. Returning back
>> results in error conditions and take down of VMs in most cases.
>>
>> o For non confidential VMs, the hypercall could be streamlined as it will
>> never return back with timeouts.
>>
>> o Rename HV_STATUS_TIME_OUT to HV_STATUS_TIMEOUT to match hypervisor
>> public header file. There is no reason for it to be different.
>>
>> o Minor formatting fix for checkpatch. Error printing missing in couple
>> places when rep hypercall fails.
>>
>> Signed-off-by: Mukesh R <mrathor@linux.microsoft.com>
>
> [snip]
>
>> diff --git a/include/asm-generic/mshyperv.h b/include/asm-generic/mshyperv.h
>> index 396bcb3db0bc..87c784b9daa6 100644
>> --- a/include/asm-generic/mshyperv.h
>> +++ b/include/asm-generic/mshyperv.h
>> @@ -28,6 +28,9 @@
>>
>> #define VTPM_BASE_ADDRESS 0xfed40000
>>
>> +#define HV_REP_COUNT_MAX \
>> + (HV_HYPERCALL_REP_COMP_MASK >> HV_HYPERCALL_REP_COMP_OFFSET)
>
> Sashiko noted that this new #define is never used anywhere.
>
>> +
>> enum hv_partition_type {
>> HV_PARTITION_TYPE_GUEST,
>> HV_PARTITION_TYPE_ROOT,
>> @@ -128,6 +131,11 @@ static inline bool hv_result_success(u64 status)
>> return hv_result(status) == HV_STATUS_SUCCESS;
>> }
>>
>> +static inline bool hv_result_timeout(u64 status)
>> +{
>> + return hv_result(status) == HV_STATUS_TIMEOUT;
>> +}
>> +
>> static inline unsigned int hv_repcomp(u64 status)
>> {
>> /* Bits [43:32] of status have 'Reps completed' data. */
>> @@ -135,45 +143,63 @@ static inline unsigned int hv_repcomp(u64 status)
>> HV_HYPERCALL_REP_COMP_OFFSET;
>> }
>>
>> -/*
>> - * Rep hypercalls. Callers of this functions are supposed to ensure that
>> - * rep_count, varhead_size, and rep_start comply with Hyper-V hypercall
>> - * definition.
>> - */
>> -static inline u64 hv_do_rep_hypercall_ex(u16 code, u16 rep_count,
>> - u16 varhead_size, u16 rep_start,
>> - void *input, void *output)
>> +static inline u64 hv_do_rep_hypercall_reg(u16 code, u16 rep_count,
>> + u16 varhead_size, u16 rep_start,
>> + void *input, void *output)
>> {
>> u64 control = code;
>> - u64 status;
>> - u16 rep_comp;
>>
>> control |= (u64)varhead_size << HV_HYPERCALL_VARHEAD_OFFSET;
>> control |= (u64)rep_count << HV_HYPERCALL_REP_COMP_OFFSET;
>> control |= (u64)rep_start << HV_HYPERCALL_REP_START_OFFSET;
>>
>> - do {
>
> Dropped the looping behavior here may be much more significant
> than just "streamlining" as described in the commit message. The looping
> behavior was included when rep hypercalls were originally added in
> commit 806c89273bab in kernel 4.14, long before any SNP/TDX/CCA
> functionality was added. So it was present for "normal" VMs doing
> hypercalls on x86/x64 the normal way. The looping ensured that
> rep hypercalls could not return with no errors but with the completed
> rep count less than the requested rep count.
>
> However, in my mind it *is* debatable whether that scenario could
> ever occur. The Hyper-V TLFS describes "Hypercall Continuation" [1]
> as essentially transparent to the caller. If a rep hypercall stops
> processing reps because of hitting the 50 microsecond time limit,
> it should restart and continue with the next set of reps without
> the caller having to loop. Commit 806c89273bab mentions
> hypercall continuation, but does not describe why the explicit
> looping was added or how that aligns with the TLFS statements.
> Vitaly -- it was a long time ago, but do you have any recollection?
>
> Have you gotten an explicit assurance from the Hyper-V team
> that the transparent hypercall continuation really does work,
> and the explicit looping has all along been unnecessary? This
Correct, hence the patch. Waiting on confirmation that it is
same behavior all the way back to server 2016.
Basically, the hypervisor will just adjust the RIP back to the
vmcall upon internal timeouts, and reschedule the vcpu. But for
hw confidential vms, ie, tdx/snp/cca, it cannot adjust the RIP,
hence return back to the guest with TIMEOUT, so cvm has to do in
a loop.
Now, for paravisor support, I was just told by a paravisor team
dev, that it will never return with TIMEOUTs. My view on that set
aside, I guess for paravisor we'll just call the _reg function
then. Thanks for bringing attention to that...
> assurance would be needed back to and including Windows
> Server 2016, which is now the oldest Hyper-V version supported
> by Linux. Also note that the rep hypercall code is architecture
> independent, so the transparent hypercall continuation would
> also need to work on arm64 when the hypercall is invoked via
> the standard path of using the HVC instruction (the TLFS doesn't
> cover arm64, so this is unclear).
Both x86 and arm are same for this issue.
> If you have such assurance, then I'm good with the removal of
> the looping. But please make the removal a separate patch, with
> sufficient discussion in the commit message to justify why it's OK
> to do the removal (including that evidently the original commit
> misinterpreted how continuation worked). And instead of looping,
> perhaps do a sanity check that the completed rep count matches
> the requested rep count.
Not sure if it's worth the overhead of sanity check, looks like
callers are checking completed and loop anyways... But I would
be OK with BUG(), since it would be a bug for that to happen. FWIW,
it's been running internally with couple months, and haven't seen
any issues.
> Without such assurance, I think the looping needs to stay. Callers
> are not prepared for a rep hypercall to return with no errors, but
> with the completed rep count less than the requested rep count.
>
> [1] https://learn.microsoft.com/en-us/virtualization/hyper-v-on-windows/tlfs/hypercall-interface#hypercall-continuation
>
>> + return hv_do_hypercall(control, input, output);
>> +}
>> +
>> +static inline u64 hv_do_rep_hypercall_cvm(u16 code, u16 rep_count,
>> + u16 varhead_size, u16 rep_start,
>> + void *input, void *output)
>> +{
>> + u64 status, control = code;
>> + u16 rep_comp = 0;
>> +
>> + control |= (u64)varhead_size << HV_HYPERCALL_VARHEAD_OFFSET;
>> + control |= (u64)rep_count << HV_HYPERCALL_REP_COMP_OFFSET;
>> + control |= (u64)rep_start << HV_HYPERCALL_REP_START_OFFSET;
>> +
>> + for (;;) {
>> status = hv_do_hypercall(control, input, output);
>> - if (!hv_result_success(status))
>> +
>> + if (!hv_result_timeout(status))
>> return status;
>>
>> rep_comp = hv_repcomp(status);
>> -
>> control &= ~HV_HYPERCALL_REP_START_MASK;
>> control |= (u64)rep_comp << HV_HYPERCALL_REP_START_OFFSET;
>>
>> touch_nmi_watchdog();
>> - } while (rep_comp < rep_count);
>> + }
>>
>> return status;
>
> And Sashiko noted that this "return status" is dead code. The only
> way to exit this function is via the "return status" up in the loop.
>
>> }
>>
>> -/* For the typical case where rep_start is 0 */
>> +/*
>> + * Rep hypercalls for HW confidential VMs return back when timeouts occur,
>> + * whereas for regular VMs they return only on success or failure. In case
>> + * of successful completion, entire rep_count is done. But a hypercall could
>> + * fail with partial completions, hence callers must always check that
>> + * in case of failure. Lastly, a rep_count value of 0 is allowed for some
>> + * hypercalls.
>> + */
>> static inline u64 hv_do_rep_hypercall(u16 code, u16 rep_count, u16 varhead_size,
>> void *input, void *output)
>> {
>> - return hv_do_rep_hypercall_ex(code, rep_count, varhead_size, 0,
>> - input, output);
>> + if (hv_isolation_type_hw())
>
> I don't think this is the right test, so let me explain. You don't describe the
> context for when HV_STATUS_TIMEOUT may be returned, and there's no
> TLFS documentation of this status, but I'm assuming the context is hypercall
> continuation. This isn't the typical "something isn't responding" timeout. I
> suspect it means the hypervisor hit the 50 us limit, and returned to the guest
> VM with some reps still to be done. In a normal VM when a hypercall is
> initiated by the hypercall page on x86/x64 or the HVC instruction on arm64,
> the transparent hypercall continuation mechanism makes a TIMEOUT status
> unnecessary. But with SNP, TDX, and CCA, hypercall invocation may be
> piggybacked on other transports, and transparent continuation isn't available.
> So Hyper-V returns HV_STATUS_TIMEOUT with the expectation that the
> invoking code will loop to do the remaining reps.
>
> The key factor is whether the hypercall is piggybacked on other transports.
> The tests in hv_isolation_type_hw() don't capture that. For example, if
> a paravisor is present, hv_isolation_type_snp() will return "true" even
> though hypercalls are invoked via the standard x86/x64 hypercall page.
> The hypercalls trap to the paravisor. If the paravisor is functioning properly,
> it should do transparent hypercall continuation just like the hypervisor would.
> But this is something to be confirmed with the hypervisor and paravisor teams.
>
> Piggybacking is done only when no paravisor is present, so the
> hv_do_rep_hypercall_cvm() path should be taken only in that circumstance.
> Your current code works, but it goes down the more expensive
> hv_do_rep_hypercall_cvm() path when it doesn't need to. (I'm not completely
> clear on how arm64 works with and without a paravisor, so my statement
> applies only to SNP and TDX. Need to sort out the CCA case with Kameron.)
>
> Given the non-trivial complexity in sorting all this out, I wonder if
> distinguishing the _cvm and _reg cases is really worth it. The current
> code for the _reg case is admittedly inefficient because it does work for
> the looping case before likely deciding that no looping is needed. But
> that could easily be fixed. When no looping is necessary, which is the
> vast majority of cases, the looping code could end up being only a
> couple of instructions slower than a separately coded non-looping
> version (I did some builds and looked at the generated code). Since
> there's overhead in deciding which to use, there may not be any
> performance win from having both versions.
Well, to me it's correctness issue also. If the ABI says that it will
not return with TIMEOUTs for non-cvms, then adding loop just masks it
instead of opening a bug report against the hypervisor.
Hope that makes sense.
On a side note, to solicit your thoughts, i am not sure i like the
touch_nmi_watchdog()... it appears the the WD is triggered in seconds,
and if a hypercall is that long, schedule() really should be done in
between. IOW, remove the touch, instead any very long running hypercalls
should call schedule(). Of course, that would be a different patch.
Thanks,
-Mukesh
> Michael
>
>> + return hv_do_rep_hypercall_cvm(code, rep_count, varhead_size, 0,
>> + input, output);
>> + else
>> + return hv_do_rep_hypercall_reg(code, rep_count, varhead_size, 0,
>> + input, output);
>> }
>>
prev parent reply other threads:[~2026-11-17 2:29 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-14 2:53 [PATCH V1 0/2] Hyper-V: Fix bugs and issues around repeat hypercall Mukesh R
2026-11-14 2:53 ` [PATCH V1 1/2] Drivers: hv: Add new function hv_isolation_type_hw() Mukesh R
2026-11-14 2:53 ` [PATCH V1 2/2] Drivers: hv: Fix bugs and issues around repeat hypercall Mukesh R
2026-11-14 3:18 ` sashiko-bot
2026-11-15 23:48 ` Michael Kelley
2026-11-16 16:53 ` Vitaly Kuznetsov
2026-11-17 2:29 ` Mukesh R [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=fd130973-25d6-e706-6822-793f12ba2015@linux.microsoft.com \
--to=mrathor@linux.microsoft.com \
--cc=decui@microsoft.com \
--cc=haiyangz@microsoft.com \
--cc=kameroncarr@linux.microsoft.com \
--cc=linux-hyperv@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mhklinux@outlook.com \
--cc=vkuznets@redhat.com \
--cc=wei.liu@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox