[PATCH] xen: register timer interrupt with IRQF_TIMER

[PATCH] xen: register timer interrupt with IRQF_TIMER

[Ian Campbell]

Otherwise the timer is disabled by dpm_suspend_noirq() which in turn prevents
correct operation of stop_machine on multi-processor systems and breaks
suspend.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Cc: Jeremy Fitzhardinge <jeremy@goop.org>
Cc: xen-devel@lists.xensource.com
---
 arch/x86/xen/time.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/arch/x86/xen/time.c b/arch/x86/xen/time.c
index 6bbff94..9d1f853 100644
--- a/arch/x86/xen/time.c
+++ b/arch/x86/xen/time.c
@@ -434,7 +434,7 @@ void xen_setup_timer(int cpu)
 		name = "<timer kasprintf failed>";
 
 	irq = bind_virq_to_irqhandler(VIRQ_TIMER, cpu, xen_timer_interrupt,
-				      IRQF_DISABLED|IRQF_PERCPU|IRQF_NOBALANCING,
+				      IRQF_DISABLED|IRQF_PERCPU|IRQF_NOBALANCING|IRQF_TIMER,
 				      name, NULL);
 
 	evt = &per_cpu(xen_clock_events, cpu);
-- 
1.5.6.5

Re: [PATCH] xen: register timer interrupt with IRQF_TIMER

[Ian Campbell]

On Tue, 2009-11-24 at 10:16 +0000, Ian Campbell wrote:
> Otherwise the timer is disabled by dpm_suspend_noirq() which in turn prevents
> correct operation of stop_machine on multi-processor systems and breaks
> suspend.

This appears to have been broken since after 2.6.29 sometime (I see
failures with 2.6.30). I suspect this commit, or one related to it, made
IRQF_TIMER a hard requirement:

        commit 0a0c5168df270a50e3518e4f12bddb31f8f5f38f
        Author: Rafael J. Wysocki <rjw@sisk.pl>
        Date:   Mon Mar 16 22:33:49 2009 +0100
        
            PM: Introduce functions for suspending and resuming device interrupts
            
            Introduce helper functions allowing us to prevent device drivers from
            getting any interrupts (without disabling interrupts on the CPU)
            during suspend (or hibernation) and to make them start to receive
            interrupts again during the subsequent resume.  These functions make it
            possible to keep timer interrupts enabled while the "late" suspend and
            "early" resume callbacks provided by device drivers are being
            executed.  In turn, this allows device drivers' "late" suspend and
            "early" resume callbacks to sleep, execute ACPI callbacks etc.
            
            The functions introduced here will be used to rework the handling of
            interrupts during suspend (hibernation) and resume.  Namely,
            interrupts will only be disabled on the CPU right before suspending
            sysdevs, while device drivers will be prevented from receiving
            interrupts, with the help of the new helper function, before their
            "late" suspend callbacks run (and analogously during resume).

Ian.

Re: Re: [PATCH] xen: register timer interrupt with IRQF_TIMER

[Valtteri Kiviniemi]

Hi,

This is not still working, when I try to save a domain:

file:

root@xen:/# du -h db.save
64K     db.save

domU console.

Ignoring shutdown request: suspend

xend.log:

[2009-11-24 14:47:57 2131] DEBUG (XendCheckpoint:124) [xc_save]: 
/usr/lib/xen/bin/xc_save 54 1 0 0 0
[2009-11-24 14:47:57 2131] INFO (XendCheckpoint:424) xc_save: failed to 
get the suspend evtchn port

domU config:

kernel = "/boot/bzImage-domU"
builder = "linux"
memory = "1024"
name = "db"
vcpus = "8"
cpus = [ "0", "1", "2", "3", "4", "5", "6", "7" ]
tsc_native="1"
vif = [ "mac=00:50:56:13:27:d9, bridge=eth0" ]
disk = [ "phy:/dev/virtuals/db,xvda1,w" ]
root = "/dev/xvda1 ro"
extra = "console=hvc0"
on_poweroff = "destroy"
on_reboot = "restart"
on_crash = "restart"

Latest xen-unstable (compiled 1 hour ago) latest 2.6.31.6 kernel with 
this patch applied.

- Valtteri Kiviniemi

Ian Campbell kirjoitti:
> On Tue, 2009-11-24 at 10:16 +0000, Ian Campbell wrote:
>> Otherwise the timer is disabled by dpm_suspend_noirq() which in turn prevents
>> correct operation of stop_machine on multi-processor systems and breaks
>> suspend.
> 
> This appears to have been broken since after 2.6.29 sometime (I see
> failures with 2.6.30). I suspect this commit, or one related to it, made
> IRQF_TIMER a hard requirement:
> 
>         commit 0a0c5168df270a50e3518e4f12bddb31f8f5f38f
>         Author: Rafael J. Wysocki <rjw@sisk.pl>
>         Date:   Mon Mar 16 22:33:49 2009 +0100
>         
>             PM: Introduce functions for suspending and resuming device interrupts
>             
>             Introduce helper functions allowing us to prevent device drivers from
>             getting any interrupts (without disabling interrupts on the CPU)
>             during suspend (or hibernation) and to make them start to receive
>             interrupts again during the subsequent resume.  These functions make it
>             possible to keep timer interrupts enabled while the "late" suspend and
>             "early" resume callbacks provided by device drivers are being
>             executed.  In turn, this allows device drivers' "late" suspend and
>             "early" resume callbacks to sleep, execute ACPI callbacks etc.
>             
>             The functions introduced here will be used to rework the handling of
>             interrupts during suspend (hibernation) and resume.  Namely,
>             interrupts will only be disabled on the CPU right before suspending
>             sysdevs, while device drivers will be prevented from receiving
>             interrupts, with the help of the new helper function, before their
>             "late" suspend callbacks run (and analogously during resume).
> 
> Ian.
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: Re: [PATCH] xen: register timer interrupt with IRQF_TIMER

[Ian Campbell]

On Tue, 2009-11-24 at 12:51 +0000, Valtteri Kiviniemi wrote:
> Hi,
> 
> This is not still working, when I try to save a domain:
> 
> file:
> 
> root@xen:/# du -h db.save
> 64K     db.save
> 
> domU console.
> 
> Ignoring shutdown request: suspend

As I said before - enable CONFIG_PM_SLEEP, you won't get anywhere
without it.

Ian.

> 
> xend.log:
> 
> [2009-11-24 14:47:57 2131] DEBUG (XendCheckpoint:124) [xc_save]: 
> /usr/lib/xen/bin/xc_save 54 1 0 0 0
> [2009-11-24 14:47:57 2131] INFO (XendCheckpoint:424) xc_save: failed to 
> get the suspend evtchn port
> 
> domU config:
> 
> kernel = "/boot/bzImage-domU"
> builder = "linux"
> memory = "1024"
> name = "db"
> vcpus = "8"
> cpus = [ "0", "1", "2", "3", "4", "5", "6", "7" ]
> tsc_native="1"
> vif = [ "mac=00:50:56:13:27:d9, bridge=eth0" ]
> disk = [ "phy:/dev/virtuals/db,xvda1,w" ]
> root = "/dev/xvda1 ro"
> extra = "console=hvc0"
> on_poweroff = "destroy"
> on_reboot = "restart"
> on_crash = "restart"
> 
> Latest xen-unstable (compiled 1 hour ago) latest 2.6.31.6 kernel with 
> this patch applied.
> 
> - Valtteri Kiviniemi
> 
> Ian Campbell kirjoitti:
> > On Tue, 2009-11-24 at 10:16 +0000, Ian Campbell wrote:
> >> Otherwise the timer is disabled by dpm_suspend_noirq() which in turn prevents
> >> correct operation of stop_machine on multi-processor systems and breaks
> >> suspend.
> > 
> > This appears to have been broken since after 2.6.29 sometime (I see
> > failures with 2.6.30). I suspect this commit, or one related to it, made
> > IRQF_TIMER a hard requirement:
> > 
> >         commit 0a0c5168df270a50e3518e4f12bddb31f8f5f38f
> >         Author: Rafael J. Wysocki <rjw@sisk.pl>
> >         Date:   Mon Mar 16 22:33:49 2009 +0100
> >         
> >             PM: Introduce functions for suspending and resuming device interrupts
> >             
> >             Introduce helper functions allowing us to prevent device drivers from
> >             getting any interrupts (without disabling interrupts on the CPU)
> >             during suspend (or hibernation) and to make them start to receive
> >             interrupts again during the subsequent resume.  These functions make it
> >             possible to keep timer interrupts enabled while the "late" suspend and
> >             "early" resume callbacks provided by device drivers are being
> >             executed.  In turn, this allows device drivers' "late" suspend and
> >             "early" resume callbacks to sleep, execute ACPI callbacks etc.
> >             
> >             The functions introduced here will be used to rework the handling of
> >             interrupts during suspend (hibernation) and resume.  Namely,
> >             interrupts will only be disabled on the CPU right before suspending
> >             sysdevs, while device drivers will be prevented from receiving
> >             interrupts, with the help of the new helper function, before their
> >             "late" suspend callbacks run (and analogously during resume).
> > 
> > Ian.
> > 
> > 
> > 
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel

Re: Re: [PATCH] xen: register timer interrupt with IRQF_TIMER

[Valtteri Kiviniemi]

Hi,

Ok, sorry didn not notice that. my domU kernels are quite small, they 
dont contain anything that is not necessary and since power management 
options were not necessary on Xen 3.4 I did not enable them on unstable 
either. Gonna now compile new domU kernels and test again.

Thanks.

- Valtteri Kiviniemi

Ian Campbell kirjoitti:
> On Tue, 2009-11-24 at 12:51 +0000, Valtteri Kiviniemi wrote:
>> Hi,
>>
>> This is not still working, when I try to save a domain:
>>
>> file:
>>
>> root@xen:/# du -h db.save
>> 64K     db.save
>>
>> domU console.
>>
>> Ignoring shutdown request: suspend
> 
> As I said before - enable CONFIG_PM_SLEEP, you won't get anywhere
> without it.
> 
> Ian.
> 
>> xend.log:
>>
>> [2009-11-24 14:47:57 2131] DEBUG (XendCheckpoint:124) [xc_save]: 
>> /usr/lib/xen/bin/xc_save 54 1 0 0 0
>> [2009-11-24 14:47:57 2131] INFO (XendCheckpoint:424) xc_save: failed to 
>> get the suspend evtchn port
>>
>> domU config:
>>
>> kernel = "/boot/bzImage-domU"
>> builder = "linux"
>> memory = "1024"
>> name = "db"
>> vcpus = "8"
>> cpus = [ "0", "1", "2", "3", "4", "5", "6", "7" ]
>> tsc_native="1"
>> vif = [ "mac=00:50:56:13:27:d9, bridge=eth0" ]
>> disk = [ "phy:/dev/virtuals/db,xvda1,w" ]
>> root = "/dev/xvda1 ro"
>> extra = "console=hvc0"
>> on_poweroff = "destroy"
>> on_reboot = "restart"
>> on_crash = "restart"
>>
>> Latest xen-unstable (compiled 1 hour ago) latest 2.6.31.6 kernel with 
>> this patch applied.
>>
>> - Valtteri Kiviniemi
>>
>> Ian Campbell kirjoitti:
>>> On Tue, 2009-11-24 at 10:16 +0000, Ian Campbell wrote:
>>>> Otherwise the timer is disabled by dpm_suspend_noirq() which in turn prevents
>>>> correct operation of stop_machine on multi-processor systems and breaks
>>>> suspend.
>>> This appears to have been broken since after 2.6.29 sometime (I see
>>> failures with 2.6.30). I suspect this commit, or one related to it, made
>>> IRQF_TIMER a hard requirement:
>>>
>>>         commit 0a0c5168df270a50e3518e4f12bddb31f8f5f38f
>>>         Author: Rafael J. Wysocki <rjw@sisk.pl>
>>>         Date:   Mon Mar 16 22:33:49 2009 +0100
>>>         
>>>             PM: Introduce functions for suspending and resuming device interrupts
>>>             
>>>             Introduce helper functions allowing us to prevent device drivers from
>>>             getting any interrupts (without disabling interrupts on the CPU)
>>>             during suspend (or hibernation) and to make them start to receive
>>>             interrupts again during the subsequent resume.  These functions make it
>>>             possible to keep timer interrupts enabled while the "late" suspend and
>>>             "early" resume callbacks provided by device drivers are being
>>>             executed.  In turn, this allows device drivers' "late" suspend and
>>>             "early" resume callbacks to sleep, execute ACPI callbacks etc.
>>>             
>>>             The functions introduced here will be used to rework the handling of
>>>             interrupts during suspend (hibernation) and resume.  Namely,
>>>             interrupts will only be disabled on the CPU right before suspending
>>>             sysdevs, while device drivers will be prevented from receiving
>>>             interrupts, with the help of the new helper function, before their
>>>             "late" suspend callbacks run (and analogously during resume).
>>>
>>> Ian.
>>>
>>>
>>>
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@lists.xensource.com
>>> http://lists.xensource.com/xen-devel
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: Re: [PATCH] xen: register timer interrupt with IRQF_TIMER

[Valtteri Kiviniemi]

Hi,

I can confirm that save works, but restore does not work. It will 
restore the domU, but the domU wont respond to anything. In the console 
I can see

root@xen:/# xm console db
Using V2 grant tables.

And then its stuck, cant login.

xend.log from restore:

[2009-11-24 16:20:11 14312] DEBUG (balloon:181) Balloon: 16069048 KiB 
free; need 1048576; done.
[2009-11-24 16:20:11 14312] DEBUG (XendCheckpoint:305) [xc_restore]: 
/usr/lib/xen/bin/xc_restore 4 2 1 2 0 0 0 0
[2009-11-24 16:20:11 14312] INFO (XendCheckpoint:424) xc_domain_restore 
start: p2m_size = 40000
[2009-11-24 16:20:11 14312] INFO (XendCheckpoint:424) Reloading memory 
pages:   0%
[2009-11-24 16:20:48 14312] INFO (XendCheckpoint:424) ERROR Internal 
error: Error when reading batch size
[2009-11-24 16:20:48 14312] INFO (XendCheckpoint:424) ERROR Internal 
error: error when buffering batch, finishing
[2009-11-24 16:20:48 14312] INFO (XendCheckpoint:424)
[2009-11-24 16:20:48 14312] INFO (XendCheckpoint:4100%
[2009-11-24 16:20:48 14312] INFO (XendCheckpoint:424) Memory reloaded (0 
pages)
[2009-11-24 16:20:48 14312] INFO (XendCheckpoint:424) read VCPU 0
[2009-11-24 16:20:48 14312] INFO (XendCheckpoint:424) read VCPU 1
[2009-11-24 16:20:48 14312] INFO (XendCheckpoint:424) read VCPU 2
[2009-11-24 16:20:48 14312] INFO (XendCheckpoint:424) read VCPU 3
[2009-11-24 16:20:48 14312] INFO (XendCheckpoint:424) read VCPU 4
[2009-11-24 16:20:48 14312] INFO (XendCheckpoint:424) read VCPU 5
[2009-11-24 16:20:48 14312] INFO (XendCheckpoint:424) read VCPU 6
[2009-11-24 16:20:48 14312] INFO (XendCheckpoint:424) read VCPU 7
[2009-11-24 16:20:48 14312] INFO (XendCheckpoint:424) Completed 
checkpoint load
[2009-11-24 16:20:48 14312] INFO (XendCheckpoint:424) Domain ready to be 
built.
[2009-11-24 16:20:48 14312] INFO (XendCheckpoint:424) Restore exit with rc=0
[2009-11-24 16:20:48 14312] DEBUG (XendCheckpoint:395) store-mfn 4037254
[2009-11-24 16:20:48 14312] DEBUG (XendCheckpoint:395) console-mfn 4037253
[2009-11-24 16:20:48 14312] DEBUG (XendDomainInfo:2883) 
XendDomainInfo.completeRestore



- Valtteri Kiviniemi

Valtteri Kiviniemi kirjoitti:
> Hi,
> 
> Ok, sorry didn not notice that. my domU kernels are quite small, they 
> dont contain anything that is not necessary and since power management 
> options were not necessary on Xen 3.4 I did not enable them on unstable 
> either. Gonna now compile new domU kernels and test again.
> 
> Thanks.
> 
> - Valtteri Kiviniemi
> 
> Ian Campbell kirjoitti:
>> On Tue, 2009-11-24 at 12:51 +0000, Valtteri Kiviniemi wrote:
>>> Hi,
>>>
>>> This is not still working, when I try to save a domain:
>>>
>>> file:
>>>
>>> root@xen:/# du -h db.save
>>> 64K     db.save
>>>
>>> domU console.
>>>
>>> Ignoring shutdown request: suspend
>>
>> As I said before - enable CONFIG_PM_SLEEP, you won't get anywhere
>> without it.
>>
>> Ian.
>>
>>> xend.log:
>>>
>>> [2009-11-24 14:47:57 2131] DEBUG (XendCheckpoint:124) [xc_save]: 
>>> /usr/lib/xen/bin/xc_save 54 1 0 0 0
>>> [2009-11-24 14:47:57 2131] INFO (XendCheckpoint:424) xc_save: failed 
>>> to get the suspend evtchn port
>>>
>>> domU config:
>>>
>>> kernel = "/boot/bzImage-domU"
>>> builder = "linux"
>>> memory = "1024"
>>> name = "db"
>>> vcpus = "8"
>>> cpus = [ "0", "1", "2", "3", "4", "5", "6", "7" ]
>>> tsc_native="1"
>>> vif = [ "mac=00:50:56:13:27:d9, bridge=eth0" ]
>>> disk = [ "phy:/dev/virtuals/db,xvda1,w" ]
>>> root = "/dev/xvda1 ro"
>>> extra = "console=hvc0"
>>> on_poweroff = "destroy"
>>> on_reboot = "restart"
>>> on_crash = "restart"
>>>
>>> Latest xen-unstable (compiled 1 hour ago) latest 2.6.31.6 kernel with 
>>> this patch applied.
>>>
>>> - Valtteri Kiviniemi
>>>
>>> Ian Campbell kirjoitti:
>>>> On Tue, 2009-11-24 at 10:16 +0000, Ian Campbell wrote:
>>>>> Otherwise the timer is disabled by dpm_suspend_noirq() which in 
>>>>> turn prevents
>>>>> correct operation of stop_machine on multi-processor systems and 
>>>>> breaks
>>>>> suspend.
>>>> This appears to have been broken since after 2.6.29 sometime (I see
>>>> failures with 2.6.30). I suspect this commit, or one related to it, 
>>>> made
>>>> IRQF_TIMER a hard requirement:
>>>>
>>>>         commit 0a0c5168df270a50e3518e4f12bddb31f8f5f38f
>>>>         Author: Rafael J. Wysocki <rjw@sisk.pl>
>>>>         Date:   Mon Mar 16 22:33:49 2009 +0100
>>>>                     PM: Introduce functions for suspending and 
>>>> resuming device interrupts
>>>>                         Introduce helper functions allowing us to 
>>>> prevent device drivers from
>>>>             getting any interrupts (without disabling interrupts on 
>>>> the CPU)
>>>>             during suspend (or hibernation) and to make them start 
>>>> to receive
>>>>             interrupts again during the subsequent resume.  These 
>>>> functions make it
>>>>             possible to keep timer interrupts enabled while the 
>>>> "late" suspend and
>>>>             "early" resume callbacks provided by device drivers are 
>>>> being
>>>>             executed.  In turn, this allows device drivers' "late" 
>>>> suspend and
>>>>             "early" resume callbacks to sleep, execute ACPI 
>>>> callbacks etc.
>>>>                         The functions introduced here will be used 
>>>> to rework the handling of
>>>>             interrupts during suspend (hibernation) and resume.  
>>>> Namely,
>>>>             interrupts will only be disabled on the CPU right before 
>>>> suspending
>>>>             sysdevs, while device drivers will be prevented from 
>>>> receiving
>>>>             interrupts, with the help of the new helper function, 
>>>> before their
>>>>             "late" suspend callbacks run (and analogously during 
>>>> resume).
>>>>
>>>> Ian.
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Xen-devel mailing list
>>>> Xen-devel@lists.xensource.com
>>>> http://lists.xensource.com/xen-devel
>>
>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>