* Re: [PATCH v1 0/3] mm/memory_hotplug: Export generic_online_page()
From: David Hildenbrand @ 2019-09-23 12:34 UTC (permalink / raw)
To: Michal Hocko
Cc: linux-kernel, linux-mm, Souptick Joarder, linux-hyperv,
Andrew Morton, Dan Williams, Haiyang Zhang, K. Y. Srinivasan,
Oscar Salvador, Pavel Tatashin, Qian Cai, Sasha Levin,
Stephen Hemminger, Wei Yang
In-Reply-To: <20190923123008.GP6016@dhcp22.suse.cz>
On 23.09.19 14:30, Michal Hocko wrote:
> On Mon 23-09-19 14:20:05, David Hildenbrand wrote:
>> On 23.09.19 14:07, Michal Hocko wrote:
>>> On Mon 23-09-19 13:34:18, David Hildenbrand wrote:
>>>> On 23.09.19 13:15, Michal Hocko wrote:
>>> [...]
>>>>> I am wondering why those pages get onlined when they are, in fact,
>>>>> supposed to be offline.
>>>>>
>>>>
>>>> It's the current way of emulating sub-memory-block hotplug on top of the
>>>> memory bock device API we have. Hyper-V and XEN have been using that for
>>>> a long time.
>>>
>>> Do they really have to use the existing block interface when they in
>>> fact do not operate on the block granularity? Zone device memory already
>>> acts on sub section/block boundaries.
>>>
>>
>> Yes, we need memory blocks, especially for user space to properly online
>> them (as we discussed a while back, to decide on a zone) and for udev
>> events, to e.g., properly reload kexec when memory blocks get
>> added/removed/onlined/offlined.
>
> Just to make sure I really follow. We need a user interface to control
> where the memory gets onlined but it is the driver which determines
> which part of the block really gets onlined, right?
>
Yes, it's the drivers job to decide which part of the memory block can
actually get used, and when.
It's a little bit like the driver immediately allocates unbacked memory
again, to free that memory to the buddy once requested. (similar to how
ballooning works).
--
Thanks,
David / dhildenb
^ permalink raw reply
* Re: [PATCH 2/3] KVM: x86: hyper-v: set NoNonArchitecturalCoreSharing CPUID bit when SMT is impossible
From: Peter Zijlstra @ 2019-09-23 15:37 UTC (permalink / raw)
To: Vitaly Kuznetsov
Cc: kvm, linux-kernel, linux-hyperv, x86, Paolo Bonzini,
Radim Krčmář, Sean Christopherson, Jim Mattson,
Thomas Gleixner, Ingo Molnar, Borislav Petkov, H. Peter Anvin,
Michael Kelley, Roman Kagan
In-Reply-To: <20190916162258.6528-3-vkuznets@redhat.com>
On Mon, Sep 16, 2019 at 06:22:57PM +0200, Vitaly Kuznetsov wrote:
> Hyper-V 2019 doesn't expose MD_CLEAR CPUID bit to guests when it cannot
> guarantee that two virtual processors won't end up running on sibling SMT
> threads without knowing about it. This is done as an optimization as in
> this case there is nothing the guest can do to protect itself against MDS
> and issuing additional flush requests is just pointless. On bare metal the
> topology is known, however, when Hyper-V is running nested (e.g. on top of
> KVM) it needs an additional piece of information: a confirmation that the
> exposed topology (wrt vCPU placement on different SMT threads) is
> trustworthy.
>
> NoNonArchitecturalCoreSharing (CPUID 0x40000004 EAX bit 18) is described in
> TLFS as follows: "Indicates that a virtual processor will never share a
> physical core with another virtual processor, except for virtual processors
> that are reported as sibling SMT threads." From KVM we can give such
> guarantee in two cases:
> - SMT is unsupported or forcefully disabled (just 'disabled' doesn't work
> as it can become re-enabled during the lifetime of the guest).
> - vCPUs are properly pinned so the scheduler won't put them on sibling
> SMT threads (when they're not reported as such).
>
> This patch reports NoNonArchitecturalCoreSharing bit in to userspace in the
> first case. The second case is outside of KVM's domain of responsibility
> (as vCPU pinning is actually done by someone who manages KVM's userspace -
> e.g. libvirt pinning QEMU threads).
This is purely about guest<->guest MDS, right? Ie. not worse than actual
hardware.
^ permalink raw reply
* Re: [PATCH 2/3] KVM: x86: hyper-v: set NoNonArchitecturalCoreSharing CPUID bit when SMT is impossible
From: Paolo Bonzini @ 2019-09-23 16:48 UTC (permalink / raw)
To: Peter Zijlstra, Vitaly Kuznetsov
Cc: kvm, linux-kernel, linux-hyperv, x86, Radim Krčmář,
Sean Christopherson, Jim Mattson, Thomas Gleixner, Ingo Molnar,
Borislav Petkov, H. Peter Anvin, Michael Kelley, Roman Kagan
In-Reply-To: <20190923153713.GF2369@hirez.programming.kicks-ass.net>
On 23/09/19 17:37, Peter Zijlstra wrote:
>> This patch reports NoNonArchitecturalCoreSharing bit in to userspace in the
>> first case. The second case is outside of KVM's domain of responsibility
>> (as vCPU pinning is actually done by someone who manages KVM's userspace -
>> e.g. libvirt pinning QEMU threads).
> This is purely about guest<->guest MDS, right? Ie. not worse than actual
> hardware.
Even within the same guest. If vCPU 1 is on virtual core 1 and vCPU 2
is on virtual core 2, but they can share the same physical core, core
scheduling in the guest can do nothing about it.
Paolo
^ permalink raw reply
* [GIT PULL] Hyper-V commits for 5.4
From: Sasha Levin @ 2019-09-24 1:03 UTC (permalink / raw)
To: Linus Torvalds; +Cc: linux-kernel, linux-hyperv, kys, sthemmin, linux-kernel
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
The following changes since commit 5f9e832c137075045d15cd6899ab0505cfb2ca4b:
Linus 5.3-rc1 (2019-07-21 14:05:38 -0700)
are available in the Git repository at:
git://git.kernel.org/pub/scm/linux/kernel/git/hyperv/linux.git tags/hyperv-next-signed
for you to fetch changes up to d8bd2d442bb2688b428ac7164e5dc6d95d4fa65b:
Drivers: hv: vmbus: Resume after fixing up old primary channels (2019-09-06 14:52:44 -0400)
- ----------------------------------------------------------------
- - First round of vmbus hibernation support from Dexuan Cui.
- - Removal of dependencies on PAGE_SIZE by Maya Nakamura.
- - Moving the hyper-v tools/ code into the tools build system by Andy
Shevchenko.
- - hyper-v balloon cleanups by Dexuan Cui.
- ----------------------------------------------------------------
Andy Shevchenko (1):
Tools: hv: move to tools buildsystem
Dexuan Cui (11):
hv_balloon: Use a static page for the balloon_up send buffer
hv_balloon: Reorganize the probe function
Drivers: hv: vmbus: Break out synic enable and disable operations
Drivers: hv: vmbus: Suspend/resume the synic for hibernation
Drivers: hv: vmbus: Add a helper function is_sub_channel()
Drivers: hv: vmbus: Implement suspend/resume for VSC drivers for hibernation
Drivers: hv: vmbus: Ignore the offers when resuming from hibernation
Drivers: hv: vmbus: Suspend/resume the vmbus itself for hibernation
Drivers: hv: vmbus: Clean up hv_sock channels by force upon suspend
Drivers: hv: vmbus: Suspend after cleaning up hv_sock and sub channels
Drivers: hv: vmbus: Resume after fixing up old primary channels
Maya Nakamura (1):
HID: hv: Remove dependencies on PAGE_SIZE for ring buffer
drivers/hid/hid-hyperv.c | 4 +-
drivers/hv/channel_mgmt.c | 161 +++++++++++++++++++++++++---
drivers/hv/connection.c | 8 +-
drivers/hv/hv.c | 66 +++++++-----
drivers/hv/hv_balloon.c | 143 ++++++++++++-------------
drivers/hv/hyperv_vmbus.h | 30 ++++++
drivers/hv/vmbus_drv.c | 265 ++++++++++++++++++++++++++++++++++++++++++++++
include/linux/hyperv.h | 16 ++-
tools/hv/Build | 3 +
tools/hv/Makefile | 51 +++++++--
10 files changed, 613 insertions(+), 134 deletions(-)
create mode 100644 tools/hv/Build
-----BEGIN PGP SIGNATURE-----
iQIzBAEBCgAdFiEE4n5dijQDou9mhzu83qZv95d3LNwFAl2Jaw0ACgkQ3qZv95d3
LNx6xg//WxLzQxi8rnxO2vYFWL9/Hq0ZZfUTHVE3FuRYJdI/IZlFhR0Kw/5MOcT3
PmAFuLNbGFDbDTP21oI+eXVHbrp8mWbSfNRR9EN/BGCErB6WAVTJlrm8ImEe8aBj
yDU4PjMp7gHwSCcuyYh813Uj9jlPybISyroCBnJ2q2M7iSLSb/wlUR+3KB+p99eS
ScyAl0KS7f/hUI1MRUEJ/0i+buEGwG9uBGsEFcicVLqNwD1Mly1ulmyXY6oQm+QZ
P7/PO7k3h/nlCr5QI9rJVsZqy0pZcRp5RI9JIriPHyCQNxy2V7bpvl979XtrODv6
3F2UNBw4I73XUZ4GcQlGeM7kVAc6CuuyiKnbRqa58e4Bjnhzj+VDbk5Brf4LFyYl
ZG4rDuBTBU5cfKi2fjmasYjgfGtQdWdhTeJM+aqsT7ScyevH/8N7k1Vte9K1s5O8
xx1MszBQ27/KxMqKD8TiL2GKQU036FiQdEnnS8We1XD1ryCR6EA+WgWRQennsZa0
jUbGJOTI3SPcjxM8oHzuiJSIK6gKRq+zKY51EVeTNwEOCPnp+WwqVO9I8inVSN0m
WwJd84pxGXuqjjIe8FvmtUE3B0AF7bW8bPV+aNeDEtPOPMPzSW9HHKWS+rllKnsA
3Z4ktdCl1Al8dsKU9mJPuzAGn0Sxg3zKQ9VPTvZUQt2WYRWjKi4=
=y2xA
-----END PGP SIGNATURE-----
^ permalink raw reply
* RE: [Patch v4] storvsc: setup 1:1 mapping between hardware queue and CPU queue
From: Long Li @ 2019-09-24 1:14 UTC (permalink / raw)
To: Long Li, Sasha Levin, longli@linuxonhyperv.com
Cc: KY Srinivasan, Haiyang Zhang, Stephen Hemminger,
James E.J. Bottomley, Martin K. Petersen,
linux-hyperv@vger.kernel.org, linux-scsi@vger.kernel.org,
linux-kernel@vger.kernel.org
In-Reply-To: <CY4PR21MB0741A256EEF9D8DBAF50ED03CEBA0@CY4PR21MB0741.namprd21.prod.outlook.com>
>Subject: RE: [Patch v4] storvsc: setup 1:1 mapping between hardware queue
>and CPU queue
>
>>Subject: Re: [Patch v4] storvsc: setup 1:1 mapping between hardware
>>queue and CPU queue
>>
>>On Fri, Sep 06, 2019 at 10:24:20AM -0700, longli@linuxonhyperv.com wrote:
>>>From: Long Li <longli@microsoft.com>
>>>
>>>storvsc doesn't use a dedicated hardware queue for a given CPU queue.
>>>When issuing I/O, it selects returning CPU (hardware queue)
>>>dynamically based on vmbus channel usage across all channels.
>>>
>>>This patch advertises num_present_cpus() as number of hardware queues.
>>>This will have upper layer setup 1:1 mapping between hardware queue
>>>and CPU queue and avoid unnecessary locking when issuing I/O.
>>>
>>>Signed-off-by: Long Li <longli@microsoft.com>
Hi Martin,
I have addressed all comments on this patch. Can you merge it to SCSI?
If there is anything else I need to change, please let me know.
Thanks
Long
>>>---
>>>
>>>Changes:
>>>v2: rely on default upper layer function to map queues. (suggested by
>>>Ming Lei
>>><tom.leiming@gmail.com>)
>>>v3: use num_present_cpus() instead of num_online_cpus(). Hyper-v
>>>doesn't support hot-add CPUs. (suggested by Michael Kelley
>>><mikelley@microsoft.com>)
>>>v4: move change logs to after Signed-of-by
>>>
>>> drivers/scsi/storvsc_drv.c | 3 +--
>>> 1 file changed, 1 insertion(+), 2 deletions(-)
>>>
>>>diff --git a/drivers/scsi/storvsc_drv.c b/drivers/scsi/storvsc_drv.c
>>>index b89269120a2d..cf987712041a 100644
>>>--- a/drivers/scsi/storvsc_drv.c
>>>+++ b/drivers/scsi/storvsc_drv.c
>>>@@ -1836,8 +1836,7 @@ static int storvsc_probe(struct hv_device *device,
>>> /*
>>> * Set the number of HW queues we are supporting.
>>> */
>>>- if (stor_device->num_sc != 0)
>>>- host->nr_hw_queues = stor_device->num_sc + 1;
>>>+ host->nr_hw_queues = num_present_cpus();
>>
>>Just looking at the change notes for v3: why isn't this
>>num_active_cpus() then? One can still isolate CPUs on hyper-v, no?
>
>The isolated CPU can be made online at run time. For example, even
>maxcpus=x is put on the boot line, individual CPUs can still be made
>online/offline.
>
>>
>>--
>>Thanks,
>>Sasha
^ permalink raw reply
* Re: [Patch v4] storvsc: setup 1:1 mapping between hardware queue and CPU queue
From: Martin K. Petersen @ 2019-09-24 2:53 UTC (permalink / raw)
To: longli
Cc: K. Y. Srinivasan, Haiyang Zhang, Stephen Hemminger, Sasha Levin,
James E.J. Bottomley, Martin K. Petersen, linux-hyperv,
linux-scsi, linux-kernel, Long Li
In-Reply-To: <1567790660-48142-1-git-send-email-longli@linuxonhyperv.com>
Long,
> storvsc doesn't use a dedicated hardware queue for a given CPU
> queue. When issuing I/O, it selects returning CPU (hardware queue)
> dynamically based on vmbus channel usage across all channels.
Applied to 5.4/scsi-fixes. Thanks!
--
Martin K. Petersen Oracle Linux Engineering
^ permalink raw reply
* [PATCH v3] Drivers: hv: vmbus: Fix harmless building warnings without CONFIG_PM_SLEEP
From: Dexuan Cui @ 2019-09-25 18:43 UTC (permalink / raw)
To: KY Srinivasan, Haiyang Zhang, Stephen Hemminger,
sashal@kernel.org, linux-hyperv@vger.kernel.org,
linux-kernel@vger.kernel.org, Michael Kelley, arnd@arndb.de
Cc: Dexuan Cui
If CONFIG_PM_SLEEP is not set, we can comment out these functions to avoid
the below warnings:
drivers/hv/vmbus_drv.c:2208:12: warning: ‘vmbus_bus_resume’ defined but not used [-Wunused-function]
drivers/hv/vmbus_drv.c:2128:12: warning: ‘vmbus_bus_suspend’ defined but not used [-Wunused-function]
drivers/hv/vmbus_drv.c:937:12: warning: ‘vmbus_resume’ defined but not used [-Wunused-function]
drivers/hv/vmbus_drv.c:918:12: warning: ‘vmbus_suspend’ defined but not used [-Wunused-function]
Fixes: 271b2224d42f ("Drivers: hv: vmbus: Implement suspend/resume for VSC drivers for hibernation")
Fixes: f53335e3289f ("Drivers: hv: vmbus: Suspend/resume the vmbus itself for hibernation")
Reported-by: Arnd Bergmann <arnd@arndb.de>
Reviewed-by: Michael Kelley <mikelley@microsoft.com>
Signed-off-by: Dexuan Cui <decui@microsoft.com>
---
In v2:
test CONFIG_PM_SLEEP rather than CONFIG_PM. Thanks, Arnd!
In v3:
Add Michael's Reviewed-by.
No other change.
drivers/hv/vmbus_drv.c | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/drivers/hv/vmbus_drv.c b/drivers/hv/vmbus_drv.c
index 391f0b225c9a..53a60c81e220 100644
--- a/drivers/hv/vmbus_drv.c
+++ b/drivers/hv/vmbus_drv.c
@@ -912,6 +912,7 @@ static void vmbus_shutdown(struct device *child_device)
drv->shutdown(dev);
}
+#ifdef CONFIG_PM_SLEEP
/*
* vmbus_suspend - Suspend a vmbus device
*/
@@ -949,6 +950,7 @@ static int vmbus_resume(struct device *child_device)
return drv->resume(dev);
}
+#endif /* CONFIG_PM_SLEEP */
/*
* vmbus_device_release - Final callback release of the vmbus child device
@@ -1070,6 +1072,7 @@ void vmbus_on_msg_dpc(unsigned long data)
vmbus_signal_eom(msg, message_type);
}
+#ifdef CONFIG_PM_SLEEP
/*
* Fake RESCIND_CHANNEL messages to clean up hv_sock channels by force for
* hibernation, because hv_sock connections can not persist across hibernation.
@@ -1105,6 +1108,7 @@ static void vmbus_force_channel_rescinded(struct vmbus_channel *channel)
vmbus_connection.work_queue,
&ctx->work);
}
+#endif /* CONFIG_PM_SLEEP */
/*
* Direct callback for channels using other deferred processing
@@ -2125,6 +2129,7 @@ static int vmbus_acpi_add(struct acpi_device *device)
return ret_val;
}
+#ifdef CONFIG_PM_SLEEP
static int vmbus_bus_suspend(struct device *dev)
{
struct vmbus_channel *channel, *sc;
@@ -2247,6 +2252,7 @@ static int vmbus_bus_resume(struct device *dev)
return 0;
}
+#endif /* CONFIG_PM_SLEEP */
static const struct acpi_device_id vmbus_acpi_device_ids[] = {
{"VMBUS", 0},
--
2.19.1
^ permalink raw reply related
* RE: [PATCH] Input: hyperv-keyboard: Add the support of hibernation
From: Dexuan Cui @ 2019-09-25 19:49 UTC (permalink / raw)
To: Dexuan Cui, dmitry.torokhov@gmail.com
Cc: KY Srinivasan, Haiyang Zhang, Stephen Hemminger,
sashal@kernel.org, linux-hyperv@vger.kernel.org,
linux-input@vger.kernel.org, linux-kernel@vger.kernel.org,
Michael Kelley
In-Reply-To: <PU1P153MB016914A7C827CA35D7FEB66ABF8B0@PU1P153MB0169.APCP153.PROD.OUTLOOK.COM>
> From: linux-hyperv-owner@vger.kernel.org
> <linux-hyperv-owner@vger.kernel.org> On Behalf Of Dexuan Cui
> Sent: Friday, September 20, 2019 11:56 PM
> To: dmitry.torokhov@gmail.com
> Cc: KY Srinivasan <kys@microsoft.com>; Haiyang Zhang
> <haiyangz@microsoft.com>; Stephen Hemminger
> <sthemmin@microsoft.com>; sashal@kernel.org;
> linux-hyperv@vger.kernel.org; linux-input@vger.kernel.org;
> linux-kernel@vger.kernel.org; Michael Kelley <mikelley@microsoft.com>
> Subject: RE: [PATCH] Input: hyperv-keyboard: Add the support of hibernation
>
> > From: dmitry.torokhov@gmail.com <dmitry.torokhov@gmail.com>
> > Sent: Thursday, September 19, 2019 9:18 AM
> >
> > Hi Dexuan,
> >
> > On Wed, Sep 11, 2019 at 11:36:20PM +0000, Dexuan Cui wrote:
> > > We need hv_kbd_pm_notify() to make sure the pm_wakeup_hard_event()
> > call
> > > does not prevent the system from entering hibernation: the hibernation
> > > is a relatively long process, which can be aborted by the call
> > > pm_wakeup_hard_event(), which is invoked upon keyboard events.
> > >
> > > diff --git a/drivers/input/serio/hyperv-keyboard.c
> > b/drivers/input/serio/hyperv-keyboard.c
> > > index 88ae7c2..277dc4c 100644
> > > --- a/drivers/input/serio/hyperv-keyboard.c
> > > +++ b/drivers/input/serio/hyperv-keyboard.c
> > > @@ -10,6 +10,7 @@
> > > #include <linux/hyperv.h>
> > > #include <linux/serio.h>
> > > #include <linux/slab.h>
> > > +#include <linux/suspend.h>
> > >
> > > /*
> > > * Current version 1.0
> > > @@ -95,6 +96,9 @@ struct hv_kbd_dev {
> > > struct completion wait_event;
> > > spinlock_t lock; /* protects 'started' field */
> > > bool started;
> > > +
> > > + struct notifier_block pm_nb;
> > > + bool hibernation_in_progress;
> >
> > Why do you use notifier block instead of exposing proper PM methods if
> > you want to support hibernation?
> >
> > Dmitry
>
> Hi,
> In the patch I do implement hv_kbd_suspend() and hv_kbd_resume(), and
> add them into the hv_kbd_drv struct:
>
> @@ -416,6 +472,8 @@ static struct hv_driver hv_kbd_drv = {
> .id_table = id_table,
> .probe = hv_kbd_probe,
> .remove = hv_kbd_remove,
> + .suspend = hv_kbd_suspend,
> + .resume = hv_kbd_resume,
>
> The .suspend and .resume callbacks are inroduced by another patch (which
> uses the dev_pm_ops struct):
> 271b2224d42f ("Drivers: hv: vmbus: Implement suspend/resume for VSC
> drivers for hibernation")
> (which is on the Hyper-V tree's hyperv-next branch:
> https://git.kernel.org/pub/scm/linux/kernel/git/hyperv/linux.git/commit/?h=hyperv-next&id=271b2224d42f88870e6b060924ee374871c131fc
> )
>
> The only purpose of the notifier is to set the variable
> kbd_dev->hibernation_in_progress to true during the hibernation process.
>
> As I explained in the changelog, the hibernation is a long process (which
> can take 10+ seconds), during which the user may unintentionally touch
> the keyboard, causing key up/down events, which are still handled by
> hv_kbd_on_receive(), which calls pm_wakeup_hard_event(), which
> calls some other functions which increase the global counter
> "pm_abort_suspend", and hence pm_wakeup_pending() becomes true.
>
> pm_wakeup_pending() is tested in a lot of places in the suspend
> process and eventually an unintentional keystroke (or mouse movement,
> when it comes to the Hyper-V mouse driver drivers/hid/hid-hyperv.c)
> causes the whole hibernation process to be aborted. Usually this
> behavior is not expected by the user, I think.
>
> So, I use the notifier to set the flag variable and with it the driver can
> know when it should not call pm_wakeup_hard_event().
>
> -- Dexuan
Hi Dmitry,
Does my answer make sense? If yes, can I have an Acked-by from you?
Thanks,
-- Dexuan
^ permalink raw reply
* RE: [PATCH] HID: hyperv: Add the support of hibernation
From: Dexuan Cui @ 2019-09-25 19:53 UTC (permalink / raw)
To: Dexuan Cui, KY Srinivasan, Haiyang Zhang, Stephen Hemminger,
sashal@kernel.org, jikos@kernel.org,
benjamin.tissoires@redhat.com, linux-hyperv@vger.kernel.org,
linux-input@vger.kernel.org, linux-kernel@vger.kernel.org,
Michael Kelley
In-Reply-To: <1568244952-66716-1-git-send-email-decui@microsoft.com>
> From: Dexuan Cui <decui@microsoft.com>
> Sent: Wednesday, September 11, 2019 4:36 PM
>
> We need mousevsc_pm_notify() to make sure the pm_wakeup_hard_event()
> call
> does not prevent the system from entering hibernation: the hibernation
> is a relatively long process, which can be aborted by the call
> pm_wakeup_hard_event(), which is invoked upon mouse events.
>
> Signed-off-by: Dexuan Cui <decui@microsoft.com>
> ---
>
> This patch is basically a pure Hyper-V specific change and it has a
> build dependency on the commit 271b2224d42f ("Drivers: hv: vmbus:
> Implement
> suspend/resume for VSC drivers for hibernation"), which is on Sasha Levin's
> Hyper-V tree's hyperv-next branch [ ... snipped ...]
>
> I request this patch should go through Sasha's tree rather than the
> input subsystem's tree.
>
> Hi Jiri, Benjamin, can you please Ack?
Hi Jiri, Benjamin,
Can you please take a look at the patch?
Thanks,
-- Dexuan
^ permalink raw reply
* RE: [PATCH 0/4] Enhance pci-hyperv to support hibernation
From: Dexuan Cui @ 2019-09-25 19:58 UTC (permalink / raw)
To: Dexuan Cui, KY Srinivasan, Haiyang Zhang, Stephen Hemminger,
sashal@kernel.org, lorenzo.pieralisi@arm.com, bhelgaas@google.com,
linux-hyperv@vger.kernel.org, linux-pci@vger.kernel.org,
linux-kernel@vger.kernel.org, Michael Kelley
In-Reply-To: <1568245086-70601-1-git-send-email-decui@microsoft.com>
> From: Dexuan Cui <decui@microsoft.com>
> Sent: Wednesday, September 11, 2019 4:38 PM
> To: KY Srinivasan <kys@microsoft.com>; Haiyang Zhang
> <haiyangz@microsoft.com>; Stephen Hemminger
> <sthemmin@microsoft.com>; sashal@kernel.org; lorenzo.pieralisi@arm.com;
> bhelgaas@google.com; linux-hyperv@vger.kernel.org;
> linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; Michael Kelley
> <mikelley@microsoft.com>
> Cc: Dexuan Cui <decui@microsoft.com>
> Subject: [PATCH 0/4] Enhance pci-hyperv to support hibernation
>
> This patchset is basically a pure Hyper-V specific change and it has a
> build dependency on the commit 271b2224d42f ("Drivers: hv: vmbus:
> Implement
> suspend/resume for VSC drivers for hibernation"), which is on Sasha Levin's
> Hyper-V tree's hyperv-next branch: [... snipped ...]
>
> I request this patch should go through Sasha's tree rather than the
> pci tree.
>
> Dexuan Cui (4):
> PCI: hv: Reorganize the code in preparation of hibernation
> PCI: hv: Add the support of hibernation
> PCI: hv: Do not queue new work items on hibernation
> PCI: hv: Change pci_protocol_version to per-hbus
>
> drivers/pci/controller/pci-hyperv.c | 166
> ++++++++++++++++++++++++++++++------
> 1 file changed, 140 insertions(+), 26 deletions(-)
Hi Lorenzo, Bjorn, and all,
Can you please take a look at the patchset (4 patches in total)?
Thanks,
-- Dexuan
^ permalink raw reply
* RE: [PATCH] hv_balloon: Add the support of hibernation
From: Dexuan Cui @ 2019-09-25 20:03 UTC (permalink / raw)
To: Dexuan Cui, David Hildenbrand, KY Srinivasan, Haiyang Zhang,
Stephen Hemminger, sashal@kernel.org,
linux-hyperv@vger.kernel.org, linux-kernel@vger.kernel.org,
Michael Kelley
In-Reply-To: <PU1P153MB01699AB87526B16F7AB94045BFB20@PU1P153MB0169.APCP153.PROD.OUTLOOK.COM>
> From: linux-hyperv-owner@vger.kernel.org
> [... snipped ...]
> > Anyhow, just some comments from my side :) I can see how Windows Server
> > worked around that issue right now by just XOR'ing both features.
> >
> > David / dhildenb
>
> Thanks for sharing your thoughts!
>
> -- Dexuan
Hi David,
If my explanation sounds good to you, can I have an Acked-by from you?
Thanks,
-- Dexuan
^ permalink raw reply
* [PATCH v2] scsi: storvsc: Add the support of hibernation
From: Dexuan Cui @ 2019-09-25 20:11 UTC (permalink / raw)
To: KY Srinivasan, Haiyang Zhang, Stephen Hemminger,
sashal@kernel.org, jejb@linux.ibm.com, martin.petersen@oracle.com,
linux-hyperv@vger.kernel.org, linux-scsi@vger.kernel.org,
linux-kernel@vger.kernel.org, Michael Kelley
Cc: Dexuan Cui
When we're in storvsc_suspend(), we're sure the SCSI layer has quiesced the
scsi device by scsi_bus_suspend() -> ... -> scsi_device_quiesce(), so the
low level SCSI adapter driver only needs to suspend/resume its own state.
Signed-off-by: Dexuan Cui <decui@microsoft.com>
Acked-by: Martin K. Petersen <martin.petersen@oracle.com>
---
This patch is basically a pure Hyper-V specific change and it has a
build dependency on the commit 271b2224d42f ("Drivers: hv: vmbus: Implement
suspend/resume for VSC drivers for hibernation"), which is on Sasha Levin's
Hyper-V tree's hyperv-next branch:
https://git.kernel.org/pub/scm/linux/kernel/git/hyperv/linux.git/log/?h=hyperv-next
I request this patch should go through Sasha's tree rather than the
SCSI tree.
In v2:
Added Acked-by from Martin.
@Sasha: I think the patch can go through the hyper-v tree, since
we have the Ack from Martin. :-)
No other change.
drivers/scsi/storvsc_drv.c | 41 +++++++++++++++++++++++++++++++++++++++++
1 file changed, 41 insertions(+)
diff --git a/drivers/scsi/storvsc_drv.c b/drivers/scsi/storvsc_drv.c
index ed8b9ac..9fbf604 100644
--- a/drivers/scsi/storvsc_drv.c
+++ b/drivers/scsi/storvsc_drv.c
@@ -1727,6 +1727,13 @@ enum {
MODULE_DEVICE_TABLE(vmbus, id_table);
+static const struct { guid_t guid; } fc_guid = { HV_SYNTHFC_GUID };
+
+static bool hv_dev_is_fc(struct hv_device *hv_dev)
+{
+ return guid_equal(&fc_guid.guid, &hv_dev->dev_type);
+}
+
static int storvsc_probe(struct hv_device *device,
const struct hv_vmbus_device_id *dev_id)
{
@@ -1935,11 +1942,45 @@ static int storvsc_remove(struct hv_device *dev)
return 0;
}
+static int storvsc_suspend(struct hv_device *hv_dev)
+{
+ struct storvsc_device *stor_device = hv_get_drvdata(hv_dev);
+ struct Scsi_Host *host = stor_device->host;
+ struct hv_host_device *host_dev = shost_priv(host);
+
+ storvsc_wait_to_drain(stor_device);
+
+ drain_workqueue(host_dev->handle_error_wq);
+
+ vmbus_close(hv_dev->channel);
+
+ memset(stor_device->stor_chns, 0,
+ num_possible_cpus() * sizeof(void *));
+
+ kfree(stor_device->stor_chns);
+ stor_device->stor_chns = NULL;
+
+ cpumask_clear(&stor_device->alloced_cpus);
+
+ return 0;
+}
+
+static int storvsc_resume(struct hv_device *hv_dev)
+{
+ int ret;
+
+ ret = storvsc_connect_to_vsp(hv_dev, storvsc_ringbuffer_size,
+ hv_dev_is_fc(hv_dev));
+ return ret;
+}
+
static struct hv_driver storvsc_drv = {
.name = KBUILD_MODNAME,
.id_table = id_table,
.probe = storvsc_probe,
.remove = storvsc_remove,
+ .suspend = storvsc_suspend,
+ .resume = storvsc_resume,
.driver = {
.probe_type = PROBE_PREFER_ASYNCHRONOUS,
},
--
1.8.3.1
^ permalink raw reply related
* [PATCH v2] hv_sock: Add the support of hibernation
From: Dexuan Cui @ 2019-09-25 21:34 UTC (permalink / raw)
To: KY Srinivasan, Haiyang Zhang, Stephen Hemminger,
sashal@kernel.org, davem@davemloft.net,
linux-hyperv@vger.kernel.org, netdev@vger.kernel.org,
linux-kernel@vger.kernel.org, Michael Kelley
Cc: Dexuan Cui
Add the necessary dummy callbacks for hibernation.
Signed-off-by: Dexuan Cui <decui@microsoft.com>
Acked-by: David S. Miller <davem@davemloft.net>
---
In v2:
Added David's Acked-by.
Removed [net-next] from the Subject.
@Sasha, can you please pick this up into the hyper-v tree?
net/vmw_vsock/hyperv_transport.c | 20 ++++++++++++++++++++
1 file changed, 20 insertions(+)
diff --git a/net/vmw_vsock/hyperv_transport.c b/net/vmw_vsock/hyperv_transport.c
index f2084e3f7aa4..4c02e38aa728 100644
--- a/net/vmw_vsock/hyperv_transport.c
+++ b/net/vmw_vsock/hyperv_transport.c
@@ -930,6 +930,24 @@ static int hvs_remove(struct hv_device *hdev)
return 0;
}
+/* hv_sock connections can not persist across hibernation, and all the hv_sock
+ * channels are forced to be rescinded before hibernation: see
+ * vmbus_bus_suspend(). Here the dummy hvs_suspend() and hvs_resume()
+ * are only needed because hibernation requires that every vmbus device's
+ * driver should have a .suspend and .resume callback: see vmbus_suspend().
+ */
+static int hvs_suspend(struct hv_device *hv_dev)
+{
+ /* Dummy */
+ return 0;
+}
+
+static int hvs_resume(struct hv_device *dev)
+{
+ /* Dummy */
+ return 0;
+}
+
/* This isn't really used. See vmbus_match() and vmbus_probe() */
static const struct hv_vmbus_device_id id_table[] = {
{},
@@ -941,6 +959,8 @@ static struct hv_driver hvs_drv = {
.id_table = id_table,
.probe = hvs_probe,
.remove = hvs_remove,
+ .suspend = hvs_suspend,
+ .resume = hvs_resume,
};
static int __init hvs_init(void)
--
2.19.1
^ permalink raw reply related
* Re: [PATCH v2] scsi: storvsc: Add the support of hibernation
From: kbuild test robot @ 2019-09-25 21:55 UTC (permalink / raw)
To: Dexuan Cui
Cc: kbuild-all, KY Srinivasan, Haiyang Zhang, Stephen Hemminger,
sashal@kernel.org, jejb@linux.ibm.com, martin.petersen@oracle.com,
linux-hyperv@vger.kernel.org, linux-scsi@vger.kernel.org,
linux-kernel@vger.kernel.org, Michael Kelley, Dexuan Cui
In-Reply-To: <1569442244-16726-1-git-send-email-decui@microsoft.com>
[-- Attachment #1: Type: text/plain, Size: 2342 bytes --]
Hi Dexuan,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on mkp-scsi/for-next]
[cannot apply to v5.3 next-20190924]
[if your patch is applied to the wrong git tree, please drop us a note to help
improve the system. BTW, we also suggest to use '--base' option to specify the
base tree in git format-patch, please see https://stackoverflow.com/a/37406982]
url: https://github.com/0day-ci/linux/commits/Dexuan-Cui/scsi-storvsc-Add-the-support-of-hibernation/20190926-041702
base: https://kernel.googlesource.com/pub/scm/linux/kernel/git/mkp/scsi.git for-next
config: x86_64-rhel (attached as .config)
compiler: gcc-7 (Debian 7.4.0-13) 7.4.0
reproduce:
# save the attached .config to linux build tree
make ARCH=x86_64
If you fix the issue, kindly add following tag
Reported-by: kbuild test robot <lkp@intel.com>
All errors (new ones prefixed by >>):
>> drivers//scsi/storvsc_drv.c:1981:3: error: 'struct hv_driver' has no member named 'suspend'
.suspend = storvsc_suspend,
^~~~~~~
>> drivers//scsi/storvsc_drv.c:1981:13: error: initialization from incompatible pointer type [-Werror=incompatible-pointer-types]
.suspend = storvsc_suspend,
^~~~~~~~~~~~~~~
drivers//scsi/storvsc_drv.c:1981:13: note: (near initialization for 'storvsc_drv.shutdown')
>> drivers//scsi/storvsc_drv.c:1982:3: error: 'struct hv_driver' has no member named 'resume'; did you mean 'remove'?
.resume = storvsc_resume,
^~~~~~
remove
drivers//scsi/storvsc_drv.c:1982:12: warning: excess elements in struct initializer
.resume = storvsc_resume,
^~~~~~~~~~~~~~
drivers//scsi/storvsc_drv.c:1982:12: note: (near initialization for 'storvsc_drv')
cc1: some warnings being treated as errors
vim +1981 drivers//scsi/storvsc_drv.c
1975
1976 static struct hv_driver storvsc_drv = {
1977 .name = KBUILD_MODNAME,
1978 .id_table = id_table,
1979 .probe = storvsc_probe,
1980 .remove = storvsc_remove,
> 1981 .suspend = storvsc_suspend,
> 1982 .resume = storvsc_resume,
1983 .driver = {
1984 .probe_type = PROBE_PREFER_ASYNCHRONOUS,
1985 },
1986 };
1987
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
[-- Attachment #2: .config.gz --]
[-- Type: application/gzip, Size: 43550 bytes --]
^ permalink raw reply
* [PATCH v2][PATCH net] hv_netvsc: Add the support of hibernation
From: Dexuan Cui @ 2019-09-25 22:04 UTC (permalink / raw)
To: KY Srinivasan, Haiyang Zhang, Stephen Hemminger,
sashal@kernel.org, davem@davemloft.net,
linux-hyperv@vger.kernel.org, netdev@vger.kernel.org,
linux-kernel@vger.kernel.org, Michael Kelley
Cc: Dexuan Cui
The existing netvsc_detach() and netvsc_attach() APIs make it easy to
implement the suspend/resume callbacks.
Signed-off-by: Dexuan Cui <decui@microsoft.com>
---
This patch is basically a pure Hyper-V specific change. I request this
patch should go through Sasha's Hyper-V tree rather than the net tree.
Sasha's Hyper-V tree is here:
https://git.kernel.org/pub/scm/linux/kernel/git/hyperv/linux.git
Previously there was a dependency on the commit 271b2224d42f ("Drivers:
hv: vmbus: Implement suspend/resume for VSC drivers for hibernation"),
which was only on Sasha Levin's Hyper-V tree's hyperv-next branch:
https://git.kernel.org/pub/scm/linux/kernel/git/hyperv/linux.git/log/?h=hyperv-next
. Now the patch has been merged into Linus's master tree, but as of now,
the patch (271b2224d42f) has not appeared in the net.git tree, so IMO
it's better for this patch to go through the Hyper-V tree. The added
code in this patch is unlikely to cause a conflict.
In v2:
Removed the superfluous "cancel_work_sync(&nvdev->subchan_work)".
Changed the [PATCH net-next] to [PATCH net] in the Subject, because
IMO this is more of a bug fix rather than a new feaure.
No other change.
drivers/net/hyperv/hyperv_net.h | 3 ++
drivers/net/hyperv/netvsc_drv.c | 57 +++++++++++++++++++++++++++++++++
2 files changed, 60 insertions(+)
diff --git a/drivers/net/hyperv/hyperv_net.h b/drivers/net/hyperv/hyperv_net.h
index ecc9af050387..b8763ee4c0d0 100644
--- a/drivers/net/hyperv/hyperv_net.h
+++ b/drivers/net/hyperv/hyperv_net.h
@@ -952,6 +952,9 @@ struct net_device_context {
u32 vf_alloc;
/* Serial number of the VF to team with */
u32 vf_serial;
+
+ /* Used to temporarily save the config info across hibernation */
+ struct netvsc_device_info *saved_netvsc_dev_info;
};
/* Per channel data */
diff --git a/drivers/net/hyperv/netvsc_drv.c b/drivers/net/hyperv/netvsc_drv.c
index afdcc5664ea6..53a9451a58a7 100644
--- a/drivers/net/hyperv/netvsc_drv.c
+++ b/drivers/net/hyperv/netvsc_drv.c
@@ -2392,6 +2392,61 @@ static int netvsc_remove(struct hv_device *dev)
return 0;
}
+static int netvsc_suspend(struct hv_device *dev)
+{
+ struct net_device_context *ndev_ctx;
+ struct net_device *vf_netdev, *net;
+ struct netvsc_device *nvdev;
+ int ret;
+
+ net = hv_get_drvdata(dev);
+
+ ndev_ctx = netdev_priv(net);
+ cancel_delayed_work_sync(&ndev_ctx->dwork);
+
+ rtnl_lock();
+
+ nvdev = rtnl_dereference(ndev_ctx->nvdev);
+ if (nvdev == NULL) {
+ ret = -ENODEV;
+ goto out;
+ }
+
+ vf_netdev = rtnl_dereference(ndev_ctx->vf_netdev);
+ if (vf_netdev)
+ netvsc_unregister_vf(vf_netdev);
+
+ /* Save the current config info */
+ ndev_ctx->saved_netvsc_dev_info = netvsc_devinfo_get(nvdev);
+
+ ret = netvsc_detach(net, nvdev);
+out:
+ rtnl_unlock();
+
+ return ret;
+}
+
+static int netvsc_resume(struct hv_device *dev)
+{
+ struct net_device *net = hv_get_drvdata(dev);
+ struct net_device_context *net_device_ctx;
+ struct netvsc_device_info *device_info;
+ int ret;
+
+ rtnl_lock();
+
+ net_device_ctx = netdev_priv(net);
+ device_info = net_device_ctx->saved_netvsc_dev_info;
+
+ ret = netvsc_attach(net, device_info);
+
+ rtnl_unlock();
+
+ kfree(device_info);
+ net_device_ctx->saved_netvsc_dev_info = NULL;
+
+ return ret;
+}
static const struct hv_vmbus_device_id id_table[] = {
/* Network guid */
{ HV_NIC_GUID, },
@@ -2406,6 +2461,8 @@ static struct hv_driver netvsc_drv = {
.id_table = id_table,
.probe = netvsc_probe,
.remove = netvsc_remove,
+ .suspend = netvsc_suspend,
+ .resume = netvsc_resume,
.driver = {
.probe_type = PROBE_FORCE_SYNCHRONOUS,
},
--
2.19.1
^ permalink raw reply related
* RE: [PATCH v2] scsi: storvsc: Add the support of hibernation
From: Dexuan Cui @ 2019-09-25 22:15 UTC (permalink / raw)
To: kbuild test robot
Cc: kbuild-all@01.org, KY Srinivasan, Haiyang Zhang,
Stephen Hemminger, sashal@kernel.org, jejb@linux.ibm.com,
martin.petersen@oracle.com, linux-hyperv@vger.kernel.org,
linux-scsi@vger.kernel.org, linux-kernel@vger.kernel.org,
Michael Kelley
In-Reply-To: <201909260511.qkZEo9lk%lkp@intel.com>
> From: kbuild test robot <lkp@intel.com>
> Sent: Wednesday, September 25, 2019 2:55 PM
> To: Dexuan Cui <decui@microsoft.com>
> Cc: kbuild-all@01.org; KY Srinivasan <kys@microsoft.com>; Haiyang Zhang
> <haiyangz@microsoft.com>; Stephen Hemminger
> <sthemmin@microsoft.com>; sashal@kernel.org; jejb@linux.ibm.com;
> martin.petersen@oracle.com; linux-hyperv@vger.kernel.org;
> linux-scsi@vger.kernel.org; linux-kernel@vger.kernel.org; Michael Kelley
> <mikelley@microsoft.com>; Dexuan Cui <decui@microsoft.com>
> Subject: Re: [PATCH v2] scsi: storvsc: Add the support of hibernation
>
> Hi Dexuan,
>
> Thank you for the patch! Yet something to improve:
>
> [auto build test ERROR on mkp-scsi/for-next]
> [cannot apply to v5.3 next-20190924]
> [if your patch is applied to the wrong git tree, please drop us a note to help
> improve the system. BTW, we also suggest to use '--base' option to specify the
> base tree in git format-patch, please see
> [...snipped...]
> config: x86_64-rhel (attached as .config)
> compiler: gcc-7 (Debian 7.4.0-13) 7.4.0
> reproduce:
> # save the attached .config to linux build tree
> make ARCH=x86_64
>
> If you fix the issue, kindly add following tag
> Reported-by: kbuild test robot <lkp@intel.com>
>
> All errors (new ones prefixed by >>):
>
> >> drivers//scsi/storvsc_drv.c:1981:3: error: 'struct hv_driver' has no member
> named 'suspend'
> .suspend = storvsc_suspend,
> ^~~~~~~
This patch depends on the below patch in Linus's tree (today):
271b2224d42f ("Drivers: hv: vmbus: Implement suspend/resume for VSC drivers for hibernation")
So, we don't really have an issue here.
The mkp-scsi/for-next branch needs to do a merge with Linus's master branch.
Thanks,
-- Dexuan
^ permalink raw reply
* RE: [PATCH v2][PATCH net] hv_netvsc: Add the support of hibernation
From: Haiyang Zhang @ 2019-09-25 22:23 UTC (permalink / raw)
To: Dexuan Cui, KY Srinivasan, Stephen Hemminger, sashal@kernel.org,
davem@davemloft.net, linux-hyperv@vger.kernel.org,
netdev@vger.kernel.org, linux-kernel@vger.kernel.org,
Michael Kelley
In-Reply-To: <1569449034-29924-1-git-send-email-decui@microsoft.com>
> -----Original Message-----
> From: Dexuan Cui <decui@microsoft.com>
> Sent: Wednesday, September 25, 2019 6:04 PM
> To: KY Srinivasan <kys@microsoft.com>; Haiyang Zhang
> <haiyangz@microsoft.com>; Stephen Hemminger
> <sthemmin@microsoft.com>; sashal@kernel.org; davem@davemloft.net;
> linux-hyperv@vger.kernel.org; netdev@vger.kernel.org; linux-
> kernel@vger.kernel.org; Michael Kelley <mikelley@microsoft.com>
> Cc: Dexuan Cui <decui@microsoft.com>
> Subject: [PATCH v2][PATCH net] hv_netvsc: Add the support of hibernation
>
> The existing netvsc_detach() and netvsc_attach() APIs make it easy to
> implement the suspend/resume callbacks.
>
> Signed-off-by: Dexuan Cui <decui@microsoft.com>
Reviewed-by: Haiyang Zhang <haiyangz@microsoft.com>
^ permalink raw reply
* RE: [PATCH v2] PCI: PM: Move to D0 before calling pci_legacy_resume_early()
From: Dexuan Cui @ 2019-09-25 22:25 UTC (permalink / raw)
To: Dexuan Cui, lorenzo.pieralisi@arm.com, bhelgaas@google.com,
linux-pci@vger.kernel.org
Cc: linux-hyperv@vger.kernel.org, Stephen Hemminger,
jackm@mellanox.com, Haiyang Zhang,
driverdev-devel@linuxdriverproject.org,
linux-kernel@vger.kernel.org, Michael Kelley, Sasha Levin,
marcelo.cerri@canonical.com, olaf@aepfle.de, apw@canonical.com,
vkuznets, jasowang@redhat.com
In-Reply-To: <KU1P153MB0166994305CF5B9CFA612AA7BFB90@KU1P153MB0166.APCP153.PROD.OUTLOOK.COM>
> From: devel <driverdev-devel-bounces@linuxdriverproject.org> On Behalf Of
> Dexuan Cui
> Sent: Monday, September 2, 2019 5:35 PM
> To: lorenzo.pieralisi@arm.com; bhelgaas@google.com;
> linux-pci@vger.kernel.org
> [..snipped...]
> > ---
> >
> > changes in v2:
> > Updated the changelog with more details.
> >
> > drivers/pci/pci-driver.c | 7 ++++---
> > 1 file changed, 4 insertions(+), 3 deletions(-)
> >
> > diff --git a/drivers/pci/pci-driver.c b/drivers/pci/pci-driver.c
> > index 36dbe960306b..27dfc68db9e7 100644
> > --- a/drivers/pci/pci-driver.c
> > +++ b/drivers/pci/pci-driver.c
> > @@ -1074,15 +1074,16 @@ static int pci_pm_thaw_noirq(struct device
> *dev)
> > return error;
> > }
> >
> > - if (pci_has_legacy_pm_support(pci_dev))
> > - return pci_legacy_resume_early(dev);
> > -
> > /*
> > * pci_restore_state() requires the device to be in D0 (because of MSI
> > * restoration among other things), so force it into D0 in case the
> > * driver's "freeze" callbacks put it into a low-power state directly.
> > */
> > pci_set_power_state(pci_dev, PCI_D0);
> > +
> > + if (pci_has_legacy_pm_support(pci_dev))
> > + return pci_legacy_resume_early(dev);
> > +
> > pci_restore_state(pci_dev);
> >
> > if (drv && drv->pm && drv->pm->thaw_noirq)
> > --
>
> Hi, Lorenzo, Bjorn,
>
> Can you please take a look at the v2 ?
>
> -- Dexuan
Hi Lorenzo, Bjorn, and all,
It looks this patch has not been acked by anyone.
Should I resend it? There is no change since v2.
Thanks,
-- Dexuan
^ permalink raw reply
* RE: [PATCH v5 0/3] Enhance Hyper-V for hibernation
From: Dexuan Cui @ 2019-09-25 22:49 UTC (permalink / raw)
To: Dexuan Cui, arnd@arndb.de, bp@alien8.de,
daniel.lezcano@linaro.org, Haiyang Zhang, hpa@zytor.com,
KY Srinivasan, linux-hyperv@vger.kernel.org,
linux-kernel@vger.kernel.org, mingo@redhat.com, sashal@kernel.org,
Stephen Hemminger, Thomas Gleixner, x86@kernel.org,
Michael Kelley, Sasha Levin
Cc: linux-arch@vger.kernel.org
In-Reply-To: <1567723581-29088-1-git-send-email-decui@microsoft.com>
> From: Dexuan Cui <decui@microsoft.com>
> Sent: Thursday, September 5, 2019 3:47 PM
>
> This patchset (consisting of 3 patches) was part of the v4 patchset (consisting
> of 12 patches): https://lkml.org/lkml/2019/9/2/894
>
> I realized these 3 patches must go through the tip.git tree, because I have
> to rebase 2 of the 3 patches due to recent changes from others in the tip
> tree.
>
> All the 3 patches are now rebased to the tip tree's timers/core branch:
> https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/log/?h=timers/core
> , and all the 3 patches have Michael Kelley's Signed-off-by's.
>
> Please review.
>
> Thanks!
> Dexuan
>
> Dexuan Cui (3):
> x86/hyper-v: Suspend/resume the hypercall page for hibernation
> x86/hyper-v: Implement hv_is_hibernation_supported()
> clocksource/drivers: Suspend/resume Hyper-V clocksource for
> hibernation
>
> arch/x86/hyperv/hv_init.c | 40
> ++++++++++++++++++++++++++++++++++++++
> drivers/clocksource/hyperv_timer.c | 25 ++++++++++++++++++++++++
> include/asm-generic/mshyperv.h | 2 ++
> 3 files changed, 67 insertions(+)
Hi tglx and all,
Can you please take a look at this patchset (3 patches in total)?
IMO it's better for the 3 patches to go through the tip.git tree, but if you
(the x86 maintainers) have no objection, the patches can also go through
Saha Levin's Hyper-V tree, because the dependent patches have landed
in Linus's tree now.
Thanks,
-- Dexuan
^ permalink raw reply
* [PATCH][RESEND] clocksource/drivers: hyperv_timer: Fix CPU offlining by unbinding the timer
From: Dexuan Cui @ 2019-09-25 23:18 UTC (permalink / raw)
To: KY Srinivasan, Haiyang Zhang, Stephen Hemminger,
sashal@kernel.org, daniel.lezcano@linaro.org, tglx@linutronix.de,
linux-hyperv@vger.kernel.org, linux-kernel@vger.kernel.org,
Michael Kelley
Cc: Dexuan Cui
The commit fd1fea6834d0 says "No behavior is changed", but actually it
removes the clockevents_unbind_device() call from hv_synic_cleanup().
In the discussion earlier this month, I thought the unbind call is
unnecessary (see https://www.spinics.net/lists/arm-kernel/msg739888.html),
however, after more investigation, when a VM runs on Hyper-V, it turns out
the unbind call must be kept, otherwise CPU offling may not work, because
a per-cpu timer device is still needed, after hv_synic_cleanup() disables
the per-cpu Hyper-V timer device.
The issue is found in the hibernation test. These are the details:
1. CPU0 hangs in wait_for_ap_thread(), when trying to offline CPU1:
hibernation_snapshot
create_image
suspend_disable_secondary_cpus
freeze_secondary_cpus
_cpu_down(1, 1, CPUHP_OFFLINE)
cpuhp_kick_ap_work
cpuhp_kick_ap
__cpuhp_kick_ap
wait_for_ap_thread()
2. CPU0 hangs because CPU1 hangs this way: after CPU1 disables the per-cpu
Hyper-V timer device in hv_synic_cleanup(), CPU1 sets a timer... Please
read on to see how this can happen.
2.1 By "_cpu_down(1, 1, CPUHP_OFFLINE):", CPU0 first tries to move CPU1 to
the CPUHP_TEARDOWN_CPU state and this wakes up the cpuhp/1 thread on CPU1;
the thread is basically a loop of executing various callbacks defined in
the global array cpuhp_hp_states[]: see smpboot_thread_fn().
2.2 This is how a callback is called on CPU1:
smpboot_thread_fn
ht->thread_fn(td->cpu), i.e. cpuhp_thread_fun
cpuhp_invoke_callback
state = st->state
st->state--
cpuhp_get_step(state)->teardown.single()
2.3 At first, the state of CPU1 is CPUHP_ONLINE, which defines a
.teardown.single of NULL, so the execution of the code returns to the loop
in smpboot_thread_fn(), and then reruns cpuhp_invoke_callback() with a
smaller st->state.
2.4 The .teardown.single of every state between CPUHP_ONLINE and
CPUHP_TEARDOWN_CPU runs one by one.
2.5 When it comes to the CPUHP_AP_ONLINE_DYN range, hv_synic_cleanup()
runs: see vmbus_bus_init(). It calls hv_stimer_cleanup() ->
hv_ce_shutdown() to disable the per-cpu timer device, so timer interrupt
will no longer happen on CPU1.
2.6 Later, the .teardown.single of CPUHP_AP_SMPBOOT_THREADS, i.e.
smpboot_park_threads(), starts to run, trying to park all the other
hotplug_threads, e.g. ksoftirqd/1 and rcuc/1; here a timer can be set up
this way and the timer will never be fired since CPU1 doesn't have
an active timer device now, so CPU1 hangs and can not be offlined:
smpboot_park_threads
smpboot_park_thread
kthread_park
wait_task_inactive
schedule_hrtimeout(&to, HRTIMER_MODE_REL)
With this patch, when the per-cpu Hyper-V timer device is disabled, the
system switches to the Local APIC timer, and the hang issue can not
happen.
Fixes: fd1fea6834d0 ("clocksource/drivers: Make Hyper-V clocksource ISA agnostic")
Signed-off-by: Dexuan Cui <decui@microsoft.com>
---
The patch was firstly posted on Jul 27: https://lkml.org/lkml/2019/7/27/5
There is no change since then.
drivers/clocksource/hyperv_timer.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/drivers/clocksource/hyperv_timer.c b/drivers/clocksource/hyperv_timer.c
index ba2c79e6a0ee..17b96f9ed0c9 100644
--- a/drivers/clocksource/hyperv_timer.c
+++ b/drivers/clocksource/hyperv_timer.c
@@ -139,6 +139,7 @@ void hv_stimer_cleanup(unsigned int cpu)
/* Turn off clockevent device */
if (ms_hyperv.features & HV_MSR_SYNTIMER_AVAILABLE) {
ce = per_cpu_ptr(hv_clock_event, cpu);
+ clockevents_unbind_device(ce, cpu);
hv_ce_shutdown(ce);
}
}
--
2.19.1
^ permalink raw reply related
* Re: [PATCH v5 3/3] clocksource/drivers: Suspend/resume Hyper-V clocksource for hibernation
From: Daniel Lezcano @ 2019-09-25 23:21 UTC (permalink / raw)
To: Dexuan Cui, arnd@arndb.de, bp@alien8.de, Haiyang Zhang,
hpa@zytor.com, KY Srinivasan, linux-hyperv@vger.kernel.org,
linux-kernel@vger.kernel.org, mingo@redhat.com, sashal@kernel.org,
Stephen Hemminger, tglx@linutronix.de, x86@kernel.org,
Michael Kelley, Sasha Levin
Cc: linux-arch@vger.kernel.org
In-Reply-To: <1567723581-29088-4-git-send-email-decui@microsoft.com>
On 06/09/2019 00:47, Dexuan Cui wrote:
> This is needed for hibernation, e.g. when we resume the old kernel, we need
> to disable the "current" kernel's TSC page and then resume the old kernel's.
>
> Signed-off-by: Dexuan Cui <decui@microsoft.com>
> Reviewed-by: Michael Kelley <mikelley@microsoft.com>
I can take this patch if needed.
> ---
> drivers/clocksource/hyperv_timer.c | 25 +++++++++++++++++++++++++
> 1 file changed, 25 insertions(+)
>
> diff --git a/drivers/clocksource/hyperv_timer.c b/drivers/clocksource/hyperv_timer.c
> index 726a65e..07f4747 100644
> --- a/drivers/clocksource/hyperv_timer.c
> +++ b/drivers/clocksource/hyperv_timer.c
> @@ -237,12 +237,37 @@ static u64 read_hv_sched_clock_tsc(void)
> return read_hv_clock_tsc(NULL) - hv_sched_clock_offset;
> }
>
> +static void suspend_hv_clock_tsc(struct clocksource *arg)
> +{
> + u64 tsc_msr;
> +
> + /* Disable the TSC page */
> + hv_get_reference_tsc(tsc_msr);
> + tsc_msr &= ~BIT_ULL(0);
> + hv_set_reference_tsc(tsc_msr);
> +}
> +
> +
> +static void resume_hv_clock_tsc(struct clocksource *arg)
> +{
> + phys_addr_t phys_addr = virt_to_phys(&tsc_pg);
> + u64 tsc_msr;
> +
> + /* Re-enable the TSC page */
> + hv_get_reference_tsc(tsc_msr);
> + tsc_msr &= GENMASK_ULL(11, 0);
> + tsc_msr |= BIT_ULL(0) | (u64)phys_addr;
> + hv_set_reference_tsc(tsc_msr);
> +}
> +
> static struct clocksource hyperv_cs_tsc = {
> .name = "hyperv_clocksource_tsc_page",
> .rating = 400,
> .read = read_hv_clock_tsc,
> .mask = CLOCKSOURCE_MASK(64),
> .flags = CLOCK_SOURCE_IS_CONTINUOUS,
> + .suspend= suspend_hv_clock_tsc,
> + .resume = resume_hv_clock_ts,> };
>
> static u64 notrace read_hv_clock_msr(struct clocksource *arg)
>
--
<http://www.linaro.org/> Linaro.org │ Open source software for ARM SoCs
Follow Linaro: <http://www.facebook.com/pages/Linaro> Facebook |
<http://twitter.com/#!/linaroorg> Twitter |
<http://www.linaro.org/linaro-blog/> Blog
^ permalink raw reply
* RE: [PATCH v5 3/3] clocksource/drivers: Suspend/resume Hyper-V clocksource for hibernation
From: Dexuan Cui @ 2019-09-25 23:35 UTC (permalink / raw)
To: Daniel Lezcano, tglx@linutronix.de, arnd@arndb.de, bp@alien8.de,
Haiyang Zhang, hpa@zytor.com, KY Srinivasan,
linux-hyperv@vger.kernel.org, linux-kernel@vger.kernel.org,
mingo@redhat.com, sashal@kernel.org, Stephen Hemminger,
x86@kernel.org, Michael Kelley, Sasha Levin
Cc: linux-arch@vger.kernel.org
In-Reply-To: <8ba5e2fd-6a9f-b61b-685e-23a69cabe3a2@linaro.org>
> From: Daniel Lezcano <daniel.lezcano@linaro.org>
> Sent: Wednesday, September 25, 2019 4:21 PM
> To: Dexuan Cui <decui@microsoft.com>; arnd@arndb.de; bp@alien8.de;
> Haiyang Zhang <haiyangz@microsoft.com>; hpa@zytor.com; KY Srinivasan
> <kys@microsoft.com>; linux-hyperv@vger.kernel.org;
> linux-kernel@vger.kernel.org; mingo@redhat.com; sashal@kernel.org; Stephen
> Hemminger <sthemmin@microsoft.com>; tglx@linutronix.de; x86@kernel.org;
> Michael Kelley <mikelley@microsoft.com>; Sasha Levin
> <Alexander.Levin@microsoft.com>
> Cc: linux-arch@vger.kernel.org
> Subject: Re: [PATCH v5 3/3] clocksource/drivers: Suspend/resume Hyper-V
> clocksource for hibernation
>
> On 06/09/2019 00:47, Dexuan Cui wrote:
> > This is needed for hibernation, e.g. when we resume the old kernel, we need
> > to disable the "current" kernel's TSC page and then resume the old kernel's.
> >
> > Signed-off-by: Dexuan Cui <decui@microsoft.com>
> > Reviewed-by: Michael Kelley <mikelley@microsoft.com>
>
> I can take this patch if needed.
Thanks, Daniel! Usually tglx takes care of the patches, but it looks recently he
may be too busy to handle the 3 patches.
I guess you can take the patch, if tglx has no objection. :-)
If you take the patch, please take all the 3 patches.
Thanks,
-- Dexuan
^ permalink raw reply
* RE: [PATCH] vsock: Fix a lockdep warning in __vsock_release()
From: Dexuan Cui @ 2019-09-26 1:10 UTC (permalink / raw)
To: Stefano Garzarella
Cc: jhansen@vmware.com, davem@davemloft.net, stefanha@redhat.com,
netdev@vger.kernel.org, Stephen Hemminger, Sasha Levin,
sashal@kernel.org, Haiyang Zhang, KY Srinivasan, Michael Kelley,
linux-hyperv@vger.kernel.org, gregkh@linuxfoundation.org,
linux-kernel@vger.kernel.org
In-Reply-To: <20190822102529.q5ozdvh6kbymi6ni@steredhat>
> From: Stefano Garzarella <sgarzare@redhat.com>
> Sent: Thursday, August 22, 2019 3:25 AM
> > [...snipped...]
> > --- a/net/vmw_vsock/hyperv_transport.c
> > +++ b/net/vmw_vsock/hyperv_transport.c
> > @@ -559,7 +559,7 @@ static void hvs_release(struct vsock_sock *vsk)
> > struct sock *sk = sk_vsock(vsk);
> > bool remove_sock;
> >
> > - lock_sock(sk);
> > + lock_sock_nested(sk, SINGLE_DEPTH_NESTING);
>
> Should we update also other transports?
>
> Stefano
Hi Stefano,
Sorry for the late reply! I'll post a v2 shortly.
As I checked, hyperv socket and virtio socket need to be fixed.
The vmci socket code doesn't acquire the sock lock in the release
callback, so it doesn't need any fix.
Thanks,
-- Dexuan
^ permalink raw reply
* [PATCH net v2] vsock: Fix a lockdep warning in __vsock_release()
From: Dexuan Cui @ 2019-09-26 1:11 UTC (permalink / raw)
To: davem@davemloft.net, KY Srinivasan, Haiyang Zhang,
Stephen Hemminger, sashal@kernel.org, stefanha@redhat.com,
gregkh@linuxfoundation.org, arnd@arndb.de, deepa.kernel@gmail.com,
Dexuan Cui, ytht.net@gmail.com, tglx@linutronix.de,
netdev@vger.kernel.org, linux-kernel@vger.kernel.org,
linux-hyperv@vger.kernel.org, kvm@vger.kernel.org,
virtualization@lists.linux-foundation.org, Michael Kelley,
sgarzare@redhat.com, jhansen@vmware.com
Lockdep is unhappy if two locks from the same class are held.
Fix the below warning for hyperv and virtio sockets (vmci socket code
doesn't have the issue) by using lock_sock_nested() when __vsock_release()
is called recursively:
============================================
WARNING: possible recursive locking detected
5.3.0+ #1 Not tainted
--------------------------------------------
server/1795 is trying to acquire lock:
ffff8880c5158990 (sk_lock-AF_VSOCK){+.+.}, at: hvs_release+0x10/0x120 [hv_sock]
but task is already holding lock:
ffff8880c5158150 (sk_lock-AF_VSOCK){+.+.}, at: __vsock_release+0x2e/0xf0 [vsock]
other info that might help us debug this:
Possible unsafe locking scenario:
CPU0
----
lock(sk_lock-AF_VSOCK);
lock(sk_lock-AF_VSOCK);
*** DEADLOCK ***
May be due to missing lock nesting notation
2 locks held by server/1795:
#0: ffff8880c5d05ff8 (&sb->s_type->i_mutex_key#10){+.+.}, at: __sock_release+0x2d/0xa0
#1: ffff8880c5158150 (sk_lock-AF_VSOCK){+.+.}, at: __vsock_release+0x2e/0xf0 [vsock]
stack backtrace:
CPU: 5 PID: 1795 Comm: server Not tainted 5.3.0+ #1
Call Trace:
dump_stack+0x67/0x90
__lock_acquire.cold.67+0xd2/0x20b
lock_acquire+0xb5/0x1c0
lock_sock_nested+0x6d/0x90
hvs_release+0x10/0x120 [hv_sock]
__vsock_release+0x24/0xf0 [vsock]
__vsock_release+0xa0/0xf0 [vsock]
vsock_release+0x12/0x30 [vsock]
__sock_release+0x37/0xa0
sock_close+0x14/0x20
__fput+0xc1/0x250
task_work_run+0x98/0xc0
do_exit+0x344/0xc60
do_group_exit+0x47/0xb0
get_signal+0x15c/0xc50
do_signal+0x30/0x720
exit_to_usermode_loop+0x50/0xa0
do_syscall_64+0x24e/0x270
entry_SYSCALL_64_after_hwframe+0x49/0xbe
RIP: 0033:0x7f4184e85f31
Signed-off-by: Dexuan Cui <decui@microsoft.com>
---
NOTE: I only tested the code on Hyper-V. I can not test the code for
virtio socket, as I don't have a KVM host. :-( Sorry.
@Stefan, @Stefano: please review & test the patch for virtio socket,
and let me know if the patch breaks anything. Thanks!
Changes in v2:
Avoid the duplication of code in v1: https://lkml.org/lkml/2019/8/19/1361
Also fix virtio socket code.
net/vmw_vsock/af_vsock.c | 19 +++++++++++++++----
net/vmw_vsock/hyperv_transport.c | 2 +-
net/vmw_vsock/virtio_transport_common.c | 2 +-
3 files changed, 17 insertions(+), 6 deletions(-)
diff --git a/net/vmw_vsock/af_vsock.c b/net/vmw_vsock/af_vsock.c
index ab47bf3ab66e..dbae4373cbab 100644
--- a/net/vmw_vsock/af_vsock.c
+++ b/net/vmw_vsock/af_vsock.c
@@ -638,8 +638,10 @@ struct sock *__vsock_create(struct net *net,
}
EXPORT_SYMBOL_GPL(__vsock_create);
-static void __vsock_release(struct sock *sk)
+static void __vsock_release(struct sock *sk, int level)
{
+ WARN_ON(level != 1 && level != 2);
+
if (sk) {
struct sk_buff *skb;
struct sock *pending;
@@ -648,9 +650,18 @@ static void __vsock_release(struct sock *sk)
vsk = vsock_sk(sk);
pending = NULL; /* Compiler warning. */
+ /* The release call is supposed to use lock_sock_nested()
+ * rather than lock_sock(), if a sock lock should be acquired.
+ */
transport->release(vsk);
- lock_sock(sk);
+ /* When "level" is 2, use the nested version to avoid the
+ * warning "possible recursive locking detected".
+ */
+ if (level == 1)
+ lock_sock(sk);
+ else
+ lock_sock_nested(sk, SINGLE_DEPTH_NESTING);
sock_orphan(sk);
sk->sk_shutdown = SHUTDOWN_MASK;
@@ -659,7 +670,7 @@ static void __vsock_release(struct sock *sk)
/* Clean up any sockets that never were accepted. */
while ((pending = vsock_dequeue_accept(sk)) != NULL) {
- __vsock_release(pending);
+ __vsock_release(pending, 2);
sock_put(pending);
}
@@ -708,7 +719,7 @@ EXPORT_SYMBOL_GPL(vsock_stream_has_space);
static int vsock_release(struct socket *sock)
{
- __vsock_release(sock->sk);
+ __vsock_release(sock->sk, 1);
sock->sk = NULL;
sock->state = SS_FREE;
diff --git a/net/vmw_vsock/hyperv_transport.c b/net/vmw_vsock/hyperv_transport.c
index 261521d286d6..c443db7af8d4 100644
--- a/net/vmw_vsock/hyperv_transport.c
+++ b/net/vmw_vsock/hyperv_transport.c
@@ -559,7 +559,7 @@ static void hvs_release(struct vsock_sock *vsk)
struct sock *sk = sk_vsock(vsk);
bool remove_sock;
- lock_sock(sk);
+ lock_sock_nested(sk, SINGLE_DEPTH_NESTING);
remove_sock = hvs_close_lock_held(vsk);
release_sock(sk);
if (remove_sock)
diff --git a/net/vmw_vsock/virtio_transport_common.c b/net/vmw_vsock/virtio_transport_common.c
index 5bb70c692b1e..a666ef8fc54e 100644
--- a/net/vmw_vsock/virtio_transport_common.c
+++ b/net/vmw_vsock/virtio_transport_common.c
@@ -820,7 +820,7 @@ void virtio_transport_release(struct vsock_sock *vsk)
struct sock *sk = &vsk->sk;
bool remove_sock = true;
- lock_sock(sk);
+ lock_sock_nested(sk, SINGLE_DEPTH_NESTING);
if (sk->sk_type == SOCK_STREAM)
remove_sock = virtio_transport_close(vsk);
--
2.19.1
^ permalink raw reply related
* Re: [PATCH] hv_balloon: Add the support of hibernation
From: David Hildenbrand @ 2019-09-26 7:19 UTC (permalink / raw)
To: Dexuan Cui, KY Srinivasan, Haiyang Zhang, Stephen Hemminger,
sashal@kernel.org, linux-hyperv@vger.kernel.org,
linux-kernel@vger.kernel.org, Michael Kelley
In-Reply-To: <PU1P153MB0169B05143A68A56740669AFBF870@PU1P153MB0169.APCP153.PROD.OUTLOOK.COM>
On 25.09.19 22:03, Dexuan Cui wrote:
>> From: linux-hyperv-owner@vger.kernel.org
>> [... snipped ...]
>>> Anyhow, just some comments from my side :) I can see how Windows Server
>>> worked around that issue right now by just XOR'ing both features.
>>>
>>> David / dhildenb
>>
>> Thanks for sharing your thoughts!
>>
>> -- Dexuan
>
> Hi David,
> If my explanation sounds good to you, can I have an Acked-by from you?
>
I do ACK the approach but not the patch in it's current state. I don't
like the ifdefs - once you can get rid of the ifdefery - e.g., after the
prerequisite is upstream - you can add my
Acked-by: David Hildenbrand <david@redhat.com>
--
Thanks,
David / dhildenb
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox