Linux-HyperV List
 help / color / mirror / Atom feed
From: Long Li <longli@microsoft.com>
To: Naman Jain <namjain@linux.microsoft.com>,
	Sahil Chandna <sahilchandna@linux.microsoft.com>,
	KY Srinivasan <kys@microsoft.com>,
	Haiyang Zhang <haiyangz@microsoft.com>,
	"wei.liu@kernel.org" <wei.liu@kernel.org>,
	Dexuan Cui <DECUI@microsoft.com>,
	"lpieralisi@kernel.org" <lpieralisi@kernel.org>,
	"kwilczynski@kernel.org" <kwilczynski@kernel.org>,
	"mani@kernel.org" <mani@kernel.org>,
	"robh@kernel.org" <robh@kernel.org>,
	"bhelgaas@google.com" <bhelgaas@google.com>,
	"linux-hyperv@vger.kernel.org" <linux-hyperv@vger.kernel.org>,
	"linux-pci@vger.kernel.org" <linux-pci@vger.kernel.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Subject: RE: [PATCH] PCI: hv: Warn when wait_for_response() waits indefinitely
Date: Tue, 25 Aug 2026 17:00:47 +0000	[thread overview]
Message-ID: <SA1PR21MB66836C32FD0FABD94B45D7F6CEAF2@SA1PR21MB6683.namprd21.prod.outlook.com> (raw)
In-Reply-To: <736091cf-8d00-415e-a890-e2dfa6739325@linux.microsoft.com>

> 
> 
> On 8/25/2026 10:47 AM, Sahil Chandna wrote:
> > A guest can wait indefinitely in wait_for_response() for the host to
> > send either a rescind message or a packet completion. If the host does
> > not send either, the guest can remain blocked with no diagnostic
> > indicating a reason.
> > This was observed during a guest kernel upgrade in which the host-side
> > application handling the PCI channel faulted, causing the guest to
> > never receive the completion request.
> > Add a periodic warning in wait_for_response() when the wait exceeds a
> > timeout so that such a hang is visible in the guest's kernel log and
> > can be correlated with host-side state.
> >
> > Suggested-by: Hamza Mahfooz <hamzamahfooz@linux.microsoft.com>
> > Signed-off-by: Sahil Chandna <sahilchandna@linux.microsoft.com>
> > ---
> > This was sent earlier upstream [1]
> > [1]
> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore
> > .kernel.org%2Flinux-hyperv%2F20260612174010.2598695-1-
> hamzamahfooz%40l
> >
> inux.microsoft.com%2F&data=05%7C02%7Clongli%40microsoft.com%7C22b60
> f06
> >
> 81624fda801f08df02748dce%7C72f988bf86f141af91ab2d7cd011db47%7C1%7
> C0%7C
> >
> 639232371774209409%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOn
> RydWUsIlY
> >
> iOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7
> C0%
> >
> 7C%7C%7C&sdata=1uUKHOonR9CtiJfdofsRbWLbuKYHlWgBXdwByQWdk2A%3
> D&reserved
> > =0
> > ---
> >   drivers/pci/controller/pci-hyperv.c | 14 +++++++++++++-
> >   1 file changed, 13 insertions(+), 1 deletion(-)
> >
> > diff --git a/drivers/pci/controller/pci-hyperv.c
> > b/drivers/pci/controller/pci-hyperv.c
> > index cfc8fa403dad..c4fba0039164 100644
> > --- a/drivers/pci/controller/pci-hyperv.c
> > +++ b/drivers/pci/controller/pci-hyperv.c
> > @@ -1040,11 +1040,18 @@ static void put_pcichild(struct hv_pci_dev
> > *hpdev)
> >
> >   /*
> >    * There is no good way to get notified from
> > vmbus_onoffer_rescind(),
> > - * so let's use polling here, since this is not a hot path.
> > + * so let's use polling here, since this is not a hot path. If
> > + * wait_for_response() has been polling for
> > + PCI_RESPONSE_HANG_TIMEOUT_SEC
> > + * without either a rescind or completion, add a periodic warning.
> >    */
> > +#define PCI_RESPONSE_HANG_TIMEOUT_SEC 300
> > +
> >   static int wait_for_response(struct hv_device *hdev,
> >   			     struct completion *comp)
> >   {
> > +	unsigned long delay =
> secs_to_jiffies(PCI_RESPONSE_HANG_TIMEOUT_SEC);
> > +	u64 timeout = get_jiffies_64() + delay;
> > +
> >   	while (true) {
> >   		if (hdev->channel->rescind) {
> >   			dev_warn_once(&hdev->device, "The device is
> gone.\n"); @@ -1053,6
> > +1060,11 @@ static int wait_for_response(struct hv_device *hdev,
> >
> >   		if (wait_for_completion_timeout(comp, HZ / 10))
> >   			break;
> > +
> > +		if (time_after64(get_jiffies_64(), timeout)) {
> > +			dev_warn(&hdev->device, "PCI stuck waiting for
> response.\n");
> > +			timeout = get_jiffies_64() + delay;
> > +		}
> >   	}
> 
> There can be some enhancements in above patch to address these problems:
> 1. Logging forever every 5 minutes in case of no completion or rescind.
> 2. If we now print warning once, not knowing if completion ever arrived.
> 3. On solving pt. 1 and 2 by adding a print for completion, one should avoid
> adding a print by default for regular timely completions.
> 
> Basically something like this:
> 
> #define PCI_RESPONSE_WARN_TIMEOUT_SEC 300
> 
> static int wait_for_response(struct hv_device *hdev,
>                   struct completion *comp) {
>      unsigned long warn_at =
>          jiffies + secs_to_jiffies(PCI_RESPONSE_WARN_TIMEOUT_SEC);
>      bool warned = false;
> 
>      while (true) {
>          if (hdev->channel->rescind) {
>              dev_warn_once(&hdev->device, "The device is gone.\n");
>              return -ENODEV;
>          }
> 
>          if (wait_for_completion_timeout(comp, HZ / 10)) {
>              if (warned || time_after_eq(jiffies, warn_at))
>                  dev_warn(&hdev->device,
>                       "PCI response received after prolonged wait.\n");
>              return 0;
>          }
> 
>          if (!warned && time_after_eq(jiffies, warn_at)) {
>              dev_warn(&hdev->device,
>                   "PCI still waiting for response.\n");
>              warned = true;
>          }
>      }
> }
> 
> Regards,
> Naman

This looks better.

Thanks,
Long

  parent reply	other threads:[~2026-08-25 17:00 UTC|newest]

Thread overview: 11+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-08-25  5:17 [PATCH] PCI: hv: Warn when wait_for_response() waits indefinitely Sahil Chandna
2026-08-25  5:30 ` sashiko-bot
2026-08-25 12:03   ` Sahil Chandna
2026-08-25  6:46 ` Naman Jain
2026-08-25 12:00   ` Sahil Chandna
2026-08-25 17:00   ` Long Li [this message]
2026-08-27 16:07     ` Michael Kelley
2026-08-27 17:34       ` Saurabh Singh Sengar
2026-08-28  5:40         ` Naman Jain
2026-08-28 14:59           ` Hamza Mahfooz
2026-09-01 11:14             ` Sahil Chandna

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=SA1PR21MB66836C32FD0FABD94B45D7F6CEAF2@SA1PR21MB6683.namprd21.prod.outlook.com \
    --to=longli@microsoft.com \
    --cc=DECUI@microsoft.com \
    --cc=bhelgaas@google.com \
    --cc=haiyangz@microsoft.com \
    --cc=kwilczynski@kernel.org \
    --cc=kys@microsoft.com \
    --cc=linux-hyperv@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-pci@vger.kernel.org \
    --cc=lpieralisi@kernel.org \
    --cc=mani@kernel.org \
    --cc=namjain@linux.microsoft.com \
    --cc=robh@kernel.org \
    --cc=sahilchandna@linux.microsoft.com \
    --cc=wei.liu@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox