Re: [PATCH v2 09/11] virtio: net: Add freeze, restore handlers to support S4

Re: [PATCH v2 09/11] virtio: net: Add freeze, restore handlers to support S4

[Amit Shah]

On (Sun) 02 Oct 2011 [11:33:26], Michael S. Tsirkin wrote:
> On Thu, Sep 29, 2011 at 09:19:40PM +0530, Amit Shah wrote:
> > Remove all the vqs on hibernation and re-create them after restoring
> > from a hibernated image.  This keeps networking working across
> > hibernation.
> > 
> > Signed-off-by: Amit Shah <amit.shah@redhat.com>
> > ---
> >  drivers/net/virtio_net.c |   28 ++++++++++++++++++++++++++++
> >  1 files changed, 28 insertions(+), 0 deletions(-)
> > 
> > diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
> > index dcd4b01..8b9ed43 100644
> > --- a/drivers/net/virtio_net.c
> > +++ b/drivers/net/virtio_net.c
> > @@ -1131,6 +1131,30 @@ static void __devexit virtnet_remove(struct virtio_device *vdev)
> >  	free_netdev(vi->dev);
> >  }
> >  
> > +#ifdef CONFIG_PM
> > +static int virtnet_freeze(struct virtio_device *vdev)
> > +{
> > +	struct virtnet_info *vi = vdev->priv;
> 
> I'm guessing we need to do something like netif_device_detach here,
> otherwise guest might be in the process of using the vq for transmit at
> this point.

Done.

> I think we also must make sure NAPI RX handler is not in progress.

How to do that?  napi_disable() / napi_enable() doesn't seem right
(and it doesn't work, too).  pci_disable_device() in the suspend
routine may work?

> We also might need to mask interrupts from the device
> to prevent TX or RX from getting rescheduled?

pci_disable_device() will help this too, right?

		Amit

Re: [PATCH v2 09/11] virtio: net: Add freeze, restore handlers to support S4

[Michael S. Tsirkin]

On Tue, Nov 15, 2011 at 05:59:36PM +0530, Amit Shah wrote:
> On (Sun) 02 Oct 2011 [11:33:26], Michael S. Tsirkin wrote:
> > On Thu, Sep 29, 2011 at 09:19:40PM +0530, Amit Shah wrote:
> > > Remove all the vqs on hibernation and re-create them after restoring
> > > from a hibernated image.  This keeps networking working across
> > > hibernation.
> > > 
> > > Signed-off-by: Amit Shah <amit.shah@redhat.com>
> > > ---
> > >  drivers/net/virtio_net.c |   28 ++++++++++++++++++++++++++++
> > >  1 files changed, 28 insertions(+), 0 deletions(-)
> > > 
> > > diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
> > > index dcd4b01..8b9ed43 100644
> > > --- a/drivers/net/virtio_net.c
> > > +++ b/drivers/net/virtio_net.c
> > > @@ -1131,6 +1131,30 @@ static void __devexit virtnet_remove(struct virtio_device *vdev)
> > >  	free_netdev(vi->dev);
> > >  }
> > >  
> > > +#ifdef CONFIG_PM
> > > +static int virtnet_freeze(struct virtio_device *vdev)
> > > +{
> > > +	struct virtnet_info *vi = vdev->priv;
> > 
> > I'm guessing we need to do something like netif_device_detach here,
> > otherwise guest might be in the process of using the vq for transmit at
> > this point.
> 
> Done.
> 
> > I think we also must make sure NAPI RX handler is not in progress.
> 
> How to do that?  napi_disable() / napi_enable() doesn't seem right
> (and it doesn't work, too).  pci_disable_device() in the suspend
> routine may work?
> 
> > We also might need to mask interrupts from the device
> > to prevent TX or RX from getting rescheduled?
> 
> pci_disable_device() will help this too, right?
> 
> 		Amit

No, why would it help?

-- 
MST

Re: [PATCH v2 09/11] virtio: net: Add freeze, restore handlers to support S4

[Amit Shah]

On (Tue) 15 Nov 2011 [14:51:27], Michael S. Tsirkin wrote:
> On Tue, Nov 15, 2011 at 05:59:36PM +0530, Amit Shah wrote:
> > On (Sun) 02 Oct 2011 [11:33:26], Michael S. Tsirkin wrote:
> > > On Thu, Sep 29, 2011 at 09:19:40PM +0530, Amit Shah wrote:
> > > > Remove all the vqs on hibernation and re-create them after restoring
> > > > from a hibernated image.  This keeps networking working across
> > > > hibernation.
> > > > 
> > > > Signed-off-by: Amit Shah <amit.shah@redhat.com>
> > > > ---
> > > >  drivers/net/virtio_net.c |   28 ++++++++++++++++++++++++++++
> > > >  1 files changed, 28 insertions(+), 0 deletions(-)
> > > > 
> > > > diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
> > > > index dcd4b01..8b9ed43 100644
> > > > --- a/drivers/net/virtio_net.c
> > > > +++ b/drivers/net/virtio_net.c
> > > > @@ -1131,6 +1131,30 @@ static void __devexit virtnet_remove(struct virtio_device *vdev)
> > > >  	free_netdev(vi->dev);
> > > >  }
> > > >  
> > > > +#ifdef CONFIG_PM
> > > > +static int virtnet_freeze(struct virtio_device *vdev)
> > > > +{
> > > > +	struct virtnet_info *vi = vdev->priv;
> > > 
> > > I'm guessing we need to do something like netif_device_detach here,
> > > otherwise guest might be in the process of using the vq for transmit at
> > > this point.
> > 
> > Done.
> > 
> > > I think we also must make sure NAPI RX handler is not in progress.
> > 
> > How to do that?  napi_disable() / napi_enable() doesn't seem right
> > (and it doesn't work, too).  pci_disable_device() in the suspend
> > routine may work?
> > 
> > > We also might need to mask interrupts from the device
> > > to prevent TX or RX from getting rescheduled?
> > 
> > pci_disable_device() will help this too, right?
> > 
> > 		Amit
> 
> No, why would it help?

IRQs will be disabled after the call to pci_disable_device(), isn't
that sufficient?

		Amit

Re: [PATCH v2 09/11] virtio: net: Add freeze, restore handlers to support S4

[Michael S. Tsirkin]

On Tue, Nov 15, 2011 at 07:33:46PM +0530, Amit Shah wrote:
> On (Tue) 15 Nov 2011 [14:51:27], Michael S. Tsirkin wrote:
> > On Tue, Nov 15, 2011 at 05:59:36PM +0530, Amit Shah wrote:
> > > On (Sun) 02 Oct 2011 [11:33:26], Michael S. Tsirkin wrote:
> > > > On Thu, Sep 29, 2011 at 09:19:40PM +0530, Amit Shah wrote:
> > > > > Remove all the vqs on hibernation and re-create them after restoring
> > > > > from a hibernated image.  This keeps networking working across
> > > > > hibernation.
> > > > > 
> > > > > Signed-off-by: Amit Shah <amit.shah@redhat.com>
> > > > > ---
> > > > >  drivers/net/virtio_net.c |   28 ++++++++++++++++++++++++++++
> > > > >  1 files changed, 28 insertions(+), 0 deletions(-)
> > > > > 
> > > > > diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
> > > > > index dcd4b01..8b9ed43 100644
> > > > > --- a/drivers/net/virtio_net.c
> > > > > +++ b/drivers/net/virtio_net.c
> > > > > @@ -1131,6 +1131,30 @@ static void __devexit virtnet_remove(struct virtio_device *vdev)
> > > > >  	free_netdev(vi->dev);
> > > > >  }
> > > > >  
> > > > > +#ifdef CONFIG_PM
> > > > > +static int virtnet_freeze(struct virtio_device *vdev)
> > > > > +{
> > > > > +	struct virtnet_info *vi = vdev->priv;
> > > > 
> > > > I'm guessing we need to do something like netif_device_detach here,
> > > > otherwise guest might be in the process of using the vq for transmit at
> > > > this point.
> > > 
> > > Done.
> > > 
> > > > I think we also must make sure NAPI RX handler is not in progress.
> > > 
> > > How to do that?  napi_disable() / napi_enable() doesn't seem right
> > > (and it doesn't work, too).  pci_disable_device() in the suspend
> > > routine may work?
> > > 
> > > > We also might need to mask interrupts from the device
> > > > to prevent TX or RX from getting rescheduled?
> > > 
> > > pci_disable_device() will help this too, right?
> > > 
> > > 		Amit
> > 
> > No, why would it help?
> 
> IRQs will be disabled after the call to pci_disable_device(),
> isn't that sufficient?
> 
> 		Amit

They will?
 * pci_disable_device - Disable PCI device after use
 * @dev: PCI device to be disabled
 *
 * Signal to the system that the PCI device is not in use by the system
 * anymore.  This only involves disabling PCI bus-mastering, if active.
 *
 * Note we don't actually disable the device until all callers of
 * pci_enable_device() have called pci_disable_device().


-- 
MST

Re: [PATCH v2 09/11] virtio: net: Add freeze, restore handlers to support S4

[Amit Shah]

On (Tue) 15 Nov 2011 [16:23:00], Michael S. Tsirkin wrote:
> On Tue, Nov 15, 2011 at 07:33:46PM +0530, Amit Shah wrote:
> > On (Tue) 15 Nov 2011 [14:51:27], Michael S. Tsirkin wrote:
> > > On Tue, Nov 15, 2011 at 05:59:36PM +0530, Amit Shah wrote:
> > > > On (Sun) 02 Oct 2011 [11:33:26], Michael S. Tsirkin wrote:
> > > > > On Thu, Sep 29, 2011 at 09:19:40PM +0530, Amit Shah wrote:
> > > > > > Remove all the vqs on hibernation and re-create them after restoring
> > > > > > from a hibernated image.  This keeps networking working across
> > > > > > hibernation.
> > > > > > 
> > > > > > Signed-off-by: Amit Shah <amit.shah@redhat.com>
> > > > > > ---
> > > > > >  drivers/net/virtio_net.c |   28 ++++++++++++++++++++++++++++
> > > > > >  1 files changed, 28 insertions(+), 0 deletions(-)
> > > > > > 
> > > > > > diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
> > > > > > index dcd4b01..8b9ed43 100644
> > > > > > --- a/drivers/net/virtio_net.c
> > > > > > +++ b/drivers/net/virtio_net.c
> > > > > > @@ -1131,6 +1131,30 @@ static void __devexit virtnet_remove(struct virtio_device *vdev)
> > > > > >  	free_netdev(vi->dev);
> > > > > >  }
> > > > > >  
> > > > > > +#ifdef CONFIG_PM
> > > > > > +static int virtnet_freeze(struct virtio_device *vdev)
> > > > > > +{
> > > > > > +	struct virtnet_info *vi = vdev->priv;
> > > > > 
> > > > > I'm guessing we need to do something like netif_device_detach here,
> > > > > otherwise guest might be in the process of using the vq for transmit at
> > > > > this point.
> > > > 
> > > > Done.
> > > > 
> > > > > I think we also must make sure NAPI RX handler is not in progress.
> > > > 
> > > > How to do that?  napi_disable() / napi_enable() doesn't seem right
> > > > (and it doesn't work, too).  pci_disable_device() in the suspend
> > > > routine may work?
> > > > 
> > > > > We also might need to mask interrupts from the device
> > > > > to prevent TX or RX from getting rescheduled?
> > > > 
> > > > pci_disable_device() will help this too, right?
> > > > 
> > > 
> > > No, why would it help?
> > 
> > IRQs will be disabled after the call to pci_disable_device(),
> > isn't that sufficient?
> > 
> 
> They will?
>  * pci_disable_device - Disable PCI device after use
>  * @dev: PCI device to be disabled
>  *
>  * Signal to the system that the PCI device is not in use by the system
>  * anymore.  This only involves disabling PCI bus-mastering, if active.
>  *
>  * Note we don't actually disable the device until all callers of
>  * pci_enable_device() have called pci_disable_device().

You mean multiple devices could have called pci_enable_device()?  Not
likely to happen, at least in case of our virtio devices... only we
claim ownership over them.  I don't think that'll change.

		Amit

Re: [PATCH v2 09/11] virtio: net: Add freeze, restore handlers to support S4

[Michael S. Tsirkin]

On Tue, Nov 15, 2011 at 08:01:49PM +0530, Amit Shah wrote:
> On (Tue) 15 Nov 2011 [16:23:00], Michael S. Tsirkin wrote:
> > On Tue, Nov 15, 2011 at 07:33:46PM +0530, Amit Shah wrote:
> > > On (Tue) 15 Nov 2011 [14:51:27], Michael S. Tsirkin wrote:
> > > > On Tue, Nov 15, 2011 at 05:59:36PM +0530, Amit Shah wrote:
> > > > > On (Sun) 02 Oct 2011 [11:33:26], Michael S. Tsirkin wrote:
> > > > > > On Thu, Sep 29, 2011 at 09:19:40PM +0530, Amit Shah wrote:
> > > > > > > Remove all the vqs on hibernation and re-create them after restoring
> > > > > > > from a hibernated image.  This keeps networking working across
> > > > > > > hibernation.
> > > > > > > 
> > > > > > > Signed-off-by: Amit Shah <amit.shah@redhat.com>
> > > > > > > ---
> > > > > > >  drivers/net/virtio_net.c |   28 ++++++++++++++++++++++++++++
> > > > > > >  1 files changed, 28 insertions(+), 0 deletions(-)
> > > > > > > 
> > > > > > > diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
> > > > > > > index dcd4b01..8b9ed43 100644
> > > > > > > --- a/drivers/net/virtio_net.c
> > > > > > > +++ b/drivers/net/virtio_net.c
> > > > > > > @@ -1131,6 +1131,30 @@ static void __devexit virtnet_remove(struct virtio_device *vdev)
> > > > > > >  	free_netdev(vi->dev);
> > > > > > >  }
> > > > > > >  
> > > > > > > +#ifdef CONFIG_PM
> > > > > > > +static int virtnet_freeze(struct virtio_device *vdev)
> > > > > > > +{
> > > > > > > +	struct virtnet_info *vi = vdev->priv;
> > > > > > 
> > > > > > I'm guessing we need to do something like netif_device_detach here,
> > > > > > otherwise guest might be in the process of using the vq for transmit at
> > > > > > this point.
> > > > > 
> > > > > Done.
> > > > > 
> > > > > > I think we also must make sure NAPI RX handler is not in progress.
> > > > > 
> > > > > How to do that?  napi_disable() / napi_enable() doesn't seem right
> > > > > (and it doesn't work, too).  pci_disable_device() in the suspend
> > > > > routine may work?
> > > > > 
> > > > > > We also might need to mask interrupts from the device
> > > > > > to prevent TX or RX from getting rescheduled?
> > > > > 
> > > > > pci_disable_device() will help this too, right?
> > > > > 
> > > > 
> > > > No, why would it help?
> > > 
> > > IRQs will be disabled after the call to pci_disable_device(),
> > > isn't that sufficient?
> > > 
> > 
> > They will?
> >  * pci_disable_device - Disable PCI device after use
> >  * @dev: PCI device to be disabled
> >  *
> >  * Signal to the system that the PCI device is not in use by the system
> >  * anymore.  This only involves disabling PCI bus-mastering, if active.
> >  *
> >  * Note we don't actually disable the device until all callers of
> >  * pci_enable_device() have called pci_disable_device().
> 
> You mean multiple devices could have called pci_enable_device()?  Not
> likely to happen, at least in case of our virtio devices... only we
> claim ownership over them.  I don't think that'll change.
> 
> 		Amit

I simply mean that pci_disable_device does not seem to disable
interrupts.

-- 
MST

Re: [PATCH v2 09/11] virtio: net: Add freeze, restore handlers to support S4

[Amit Shah]

On (Tue) 15 Nov 2011 [16:36:20], Michael S. Tsirkin wrote:
> On Tue, Nov 15, 2011 at 08:01:49PM +0530, Amit Shah wrote:
> > On (Tue) 15 Nov 2011 [16:23:00], Michael S. Tsirkin wrote:
> > > On Tue, Nov 15, 2011 at 07:33:46PM +0530, Amit Shah wrote:
> > > > On (Tue) 15 Nov 2011 [14:51:27], Michael S. Tsirkin wrote:
> > > > > On Tue, Nov 15, 2011 at 05:59:36PM +0530, Amit Shah wrote:
> > > > > > On (Sun) 02 Oct 2011 [11:33:26], Michael S. Tsirkin wrote:
> > > > > > > On Thu, Sep 29, 2011 at 09:19:40PM +0530, Amit Shah wrote:
> > > > > > > > Remove all the vqs on hibernation and re-create them after restoring
> > > > > > > > from a hibernated image.  This keeps networking working across
> > > > > > > > hibernation.
> > > > > > > > 
> > > > > > > > Signed-off-by: Amit Shah <amit.shah@redhat.com>
> > > > > > > > ---
> > > > > > > >  drivers/net/virtio_net.c |   28 ++++++++++++++++++++++++++++
> > > > > > > >  1 files changed, 28 insertions(+), 0 deletions(-)
> > > > > > > > 
> > > > > > > > diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
> > > > > > > > index dcd4b01..8b9ed43 100644
> > > > > > > > --- a/drivers/net/virtio_net.c
> > > > > > > > +++ b/drivers/net/virtio_net.c
> > > > > > > > @@ -1131,6 +1131,30 @@ static void __devexit virtnet_remove(struct virtio_device *vdev)
> > > > > > > >  	free_netdev(vi->dev);
> > > > > > > >  }
> > > > > > > >  
> > > > > > > > +#ifdef CONFIG_PM
> > > > > > > > +static int virtnet_freeze(struct virtio_device *vdev)
> > > > > > > > +{
> > > > > > > > +	struct virtnet_info *vi = vdev->priv;
> > > > > > > 
> > > > > > > I'm guessing we need to do something like netif_device_detach here,
> > > > > > > otherwise guest might be in the process of using the vq for transmit at
> > > > > > > this point.
> > > > > > 
> > > > > > Done.
> > > > > > 
> > > > > > > I think we also must make sure NAPI RX handler is not in progress.
> > > > > > 
> > > > > > How to do that?  napi_disable() / napi_enable() doesn't seem right
> > > > > > (and it doesn't work, too).  pci_disable_device() in the suspend
> > > > > > routine may work?
> > > > > > 
> > > > > > > We also might need to mask interrupts from the device
> > > > > > > to prevent TX or RX from getting rescheduled?
> > > > > > 
> > > > > > pci_disable_device() will help this too, right?
> > > > > > 
> > > > > 
> > > > > No, why would it help?
> > > > 
> > > > IRQs will be disabled after the call to pci_disable_device(),
> > > > isn't that sufficient?
> > > > 
> > > 
> > > They will?
> > >  * pci_disable_device - Disable PCI device after use
> > >  * @dev: PCI device to be disabled
> > >  *
> > >  * Signal to the system that the PCI device is not in use by the system
> > >  * anymore.  This only involves disabling PCI bus-mastering, if active.
> > >  *
> > >  * Note we don't actually disable the device until all callers of
> > >  * pci_enable_device() have called pci_disable_device().
> > 
> > You mean multiple devices could have called pci_enable_device()?  Not
> > likely to happen, at least in case of our virtio devices... only we
> > claim ownership over them.  I don't think that'll change.
> > 
> 
> I simply mean that pci_disable_device does not seem to disable
> interrupts.

As far as I know, all our irqs are allocated for vqs, and they'll be
freed when the vqs are freed too.

Are there any others that need special handling?

		Amit