* debugging spinlocks
@ 2012-01-30 16:27 Ben Guthro
2012-01-30 16:34 ` Andrew Cooper
2012-01-30 16:40 ` Jan Beulich
0 siblings, 2 replies; 5+ messages in thread
From: Ben Guthro @ 2012-01-30 16:27 UTC (permalink / raw)
To: Konrad Rzeszutek; +Cc: Xen Devel
Konrad, et al.
I'm having an issue with my guests that, when I am running wireless
drivers in dom0, and the PV-on-HVM guest tries to access the network -
I get a hung dom0.
When I turn on
Kernel Hacking->
Spinlock and rw-lock debugging: basic checks
The hang goes away, but does not print out anything terribly useful in
the serial log.
The config file changes made by this menuconfig setting can be found below:
While I know from other conversations, that Konrad is aware of my
setup, for other's benefit - this is a 64bit dom0 3.2.2 kernel, with a
few of his branches pulled in.
Any thoughts, or suggestions on how best to debug these deadlocks
would be greatly appreciated.
-Ben Guthro
diff --git a/configs/config.ubuntu.amd64 b/configs/config.ubuntu.amd64
index 205b5c6..f7a233f 100644
--- a/configs/config.ubuntu.amd64
+++ b/configs/config.ubuntu.amd64
@@ -257,27 +257,27 @@ CONFIG_PADATA=y
# CONFIG_INLINE_SPIN_LOCK_BH is not set
# CONFIG_INLINE_SPIN_LOCK_IRQ is not set
# CONFIG_INLINE_SPIN_LOCK_IRQSAVE is not set
-CONFIG_INLINE_SPIN_UNLOCK=y
+# CONFIG_INLINE_SPIN_UNLOCK is not set
# CONFIG_INLINE_SPIN_UNLOCK_BH is not set
-CONFIG_INLINE_SPIN_UNLOCK_IRQ=y
+# CONFIG_INLINE_SPIN_UNLOCK_IRQ is not set
# CONFIG_INLINE_SPIN_UNLOCK_IRQRESTORE is not set
# CONFIG_INLINE_READ_TRYLOCK is not set
# CONFIG_INLINE_READ_LOCK is not set
# CONFIG_INLINE_READ_LOCK_BH is not set
# CONFIG_INLINE_READ_LOCK_IRQ is not set
# CONFIG_INLINE_READ_LOCK_IRQSAVE is not set
-CONFIG_INLINE_READ_UNLOCK=y
+# CONFIG_INLINE_READ_UNLOCK is not set
# CONFIG_INLINE_READ_UNLOCK_BH is not set
-CONFIG_INLINE_READ_UNLOCK_IRQ=y
+# CONFIG_INLINE_READ_UNLOCK_IRQ is not set
# CONFIG_INLINE_READ_UNLOCK_IRQRESTORE is not set
# CONFIG_INLINE_WRITE_TRYLOCK is not set
# CONFIG_INLINE_WRITE_LOCK is not set
# CONFIG_INLINE_WRITE_LOCK_BH is not set
# CONFIG_INLINE_WRITE_LOCK_IRQ is not set
# CONFIG_INLINE_WRITE_LOCK_IRQSAVE is not set
-CONFIG_INLINE_WRITE_UNLOCK=y
+# CONFIG_INLINE_WRITE_UNLOCK is not set
# CONFIG_INLINE_WRITE_UNLOCK_BH is not set
-CONFIG_INLINE_WRITE_UNLOCK_IRQ=y
+# CONFIG_INLINE_WRITE_UNLOCK_IRQ is not set
# CONFIG_INLINE_WRITE_UNLOCK_IRQRESTORE is not set
CONFIG_MUTEX_SPIN_ON_OWNER=y
CONFIG_FREEZER=y
@@ -398,7 +398,7 @@ CONFIG_HAVE_MEMBLOCK_NODE_MAP=y
CONFIG_ARCH_DISCARD_MEMBLOCK=y
# CONFIG_MEMORY_HOTPLUG is not set
CONFIG_PAGEFLAGS_EXTENDED=y
-CONFIG_SPLIT_PTLOCK_CPUS=4
+CONFIG_SPLIT_PTLOCK_CPUS=999999
# CONFIG_COMPACTION is not set
CONFIG_MIGRATION=y
CONFIG_PHYS_ADDR_T_64BIT=y
@@ -4255,7 +4255,7 @@ CONFIG_TIMER_STATS=y
# CONFIG_DEBUG_KMEMLEAK is not set
# CONFIG_DEBUG_RT_MUTEXES is not set
# CONFIG_RT_MUTEX_TESTER is not set
-# CONFIG_DEBUG_SPINLOCK is not set
+CONFIG_DEBUG_SPINLOCK=y
# CONFIG_DEBUG_MUTEXES is not set
# CONFIG_DEBUG_LOCK_ALLOC is not set
# CONFIG_PROVE_LOCKING is not set
^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: debugging spinlocks
2012-01-30 16:27 debugging spinlocks Ben Guthro
@ 2012-01-30 16:34 ` Andrew Cooper
2012-01-30 17:32 ` Ben Guthro
2012-01-30 16:40 ` Jan Beulich
1 sibling, 1 reply; 5+ messages in thread
From: Andrew Cooper @ 2012-01-30 16:34 UTC (permalink / raw)
To: Ben Guthro; +Cc: Konrad Rzeszutek, Xen Devel
On 30/01/12 16:27, Ben Guthro wrote:
> Konrad, et al.
>
> I'm having an issue with my guests that, when I am running wireless
> drivers in dom0, and the PV-on-HVM guest tries to access the network -
> I get a hung dom0.
>
> When I turn on
> Kernel Hacking->
> Spinlock and rw-lock debugging: basic checks
>
> The hang goes away, but does not print out anything terribly useful in
> the serial log.
>
>
> The config file changes made by this menuconfig setting can be found below:
>
>
> While I know from other conversations, that Konrad is aware of my
> setup, for other's benefit - this is a 64bit dom0 3.2.2 kernel, with a
> few of his branches pulled in.
>
> Any thoughts, or suggestions on how best to debug these deadlocks
> would be greatly appreciated.
It looks like
http://lists.xen.org/archives/html/xen-devel/2012-01/msg01917.html might
apply to you.
Does that patch help?
~Andrew
>
> -Ben Guthro
>
>
> diff --git a/configs/config.ubuntu.amd64 b/configs/config.ubuntu.amd64
> index 205b5c6..f7a233f 100644
> --- a/configs/config.ubuntu.amd64
> +++ b/configs/config.ubuntu.amd64
> @@ -257,27 +257,27 @@ CONFIG_PADATA=y
> # CONFIG_INLINE_SPIN_LOCK_BH is not set
> # CONFIG_INLINE_SPIN_LOCK_IRQ is not set
> # CONFIG_INLINE_SPIN_LOCK_IRQSAVE is not set
> -CONFIG_INLINE_SPIN_UNLOCK=y
> +# CONFIG_INLINE_SPIN_UNLOCK is not set
> # CONFIG_INLINE_SPIN_UNLOCK_BH is not set
> -CONFIG_INLINE_SPIN_UNLOCK_IRQ=y
> +# CONFIG_INLINE_SPIN_UNLOCK_IRQ is not set
> # CONFIG_INLINE_SPIN_UNLOCK_IRQRESTORE is not set
> # CONFIG_INLINE_READ_TRYLOCK is not set
> # CONFIG_INLINE_READ_LOCK is not set
> # CONFIG_INLINE_READ_LOCK_BH is not set
> # CONFIG_INLINE_READ_LOCK_IRQ is not set
> # CONFIG_INLINE_READ_LOCK_IRQSAVE is not set
> -CONFIG_INLINE_READ_UNLOCK=y
> +# CONFIG_INLINE_READ_UNLOCK is not set
> # CONFIG_INLINE_READ_UNLOCK_BH is not set
> -CONFIG_INLINE_READ_UNLOCK_IRQ=y
> +# CONFIG_INLINE_READ_UNLOCK_IRQ is not set
> # CONFIG_INLINE_READ_UNLOCK_IRQRESTORE is not set
> # CONFIG_INLINE_WRITE_TRYLOCK is not set
> # CONFIG_INLINE_WRITE_LOCK is not set
> # CONFIG_INLINE_WRITE_LOCK_BH is not set
> # CONFIG_INLINE_WRITE_LOCK_IRQ is not set
> # CONFIG_INLINE_WRITE_LOCK_IRQSAVE is not set
> -CONFIG_INLINE_WRITE_UNLOCK=y
> +# CONFIG_INLINE_WRITE_UNLOCK is not set
> # CONFIG_INLINE_WRITE_UNLOCK_BH is not set
> -CONFIG_INLINE_WRITE_UNLOCK_IRQ=y
> +# CONFIG_INLINE_WRITE_UNLOCK_IRQ is not set
> # CONFIG_INLINE_WRITE_UNLOCK_IRQRESTORE is not set
> CONFIG_MUTEX_SPIN_ON_OWNER=y
> CONFIG_FREEZER=y
> @@ -398,7 +398,7 @@ CONFIG_HAVE_MEMBLOCK_NODE_MAP=y
> CONFIG_ARCH_DISCARD_MEMBLOCK=y
> # CONFIG_MEMORY_HOTPLUG is not set
> CONFIG_PAGEFLAGS_EXTENDED=y
> -CONFIG_SPLIT_PTLOCK_CPUS=4
> +CONFIG_SPLIT_PTLOCK_CPUS=999999
> # CONFIG_COMPACTION is not set
> CONFIG_MIGRATION=y
> CONFIG_PHYS_ADDR_T_64BIT=y
> @@ -4255,7 +4255,7 @@ CONFIG_TIMER_STATS=y
> # CONFIG_DEBUG_KMEMLEAK is not set
> # CONFIG_DEBUG_RT_MUTEXES is not set
> # CONFIG_RT_MUTEX_TESTER is not set
> -# CONFIG_DEBUG_SPINLOCK is not set
> +CONFIG_DEBUG_SPINLOCK=y
> # CONFIG_DEBUG_MUTEXES is not set
> # CONFIG_DEBUG_LOCK_ALLOC is not set
> # CONFIG_PROVE_LOCKING is not set
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
--
Andrew Cooper - Dom0 Kernel Engineer, Citrix XenServer
T: +44 (0)1223 225 900, http://www.citrix.com
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: debugging spinlocks
2012-01-30 16:34 ` Andrew Cooper
@ 2012-01-30 17:32 ` Ben Guthro
2012-01-31 11:00 ` David Vrabel
0 siblings, 1 reply; 5+ messages in thread
From: Ben Guthro @ 2012-01-30 17:32 UTC (permalink / raw)
To: Andrew Cooper; +Cc: Konrad Rzeszutek, Xen Devel
Yes, it helped a lot!
This seems to have resolved the hang - thanks for pointing it my way!
So - if I'm reading the git log correctly, I should just have to
cherry-pick 7a7546b377bdaa25ac77f33d9433c59f259b9688 from mainline?
It seems like this may be something to CC the -stable queue on, IMHO.
On Mon, Jan 30, 2012 at 11:34 AM, Andrew Cooper
<andrew.cooper3@citrix.com> wrote:
> On 30/01/12 16:27, Ben Guthro wrote:
>> Konrad, et al.
>>
>> I'm having an issue with my guests that, when I am running wireless
>> drivers in dom0, and the PV-on-HVM guest tries to access the network -
>> I get a hung dom0.
>>
>> When I turn on
>> Kernel Hacking->
>> Spinlock and rw-lock debugging: basic checks
>>
>> The hang goes away, but does not print out anything terribly useful in
>> the serial log.
>>
>>
>> The config file changes made by this menuconfig setting can be found below:
>>
>>
>> While I know from other conversations, that Konrad is aware of my
>> setup, for other's benefit - this is a 64bit dom0 3.2.2 kernel, with a
>> few of his branches pulled in.
>>
>> Any thoughts, or suggestions on how best to debug these deadlocks
>> would be greatly appreciated.
>
> It looks like
> http://lists.xen.org/archives/html/xen-devel/2012-01/msg01917.html might
> apply to you.
>
> Does that patch help?
>
> ~Andrew
>
>>
>> -Ben Guthro
>>
>>
>> diff --git a/configs/config.ubuntu.amd64 b/configs/config.ubuntu.amd64
>> index 205b5c6..f7a233f 100644
>> --- a/configs/config.ubuntu.amd64
>> +++ b/configs/config.ubuntu.amd64
>> @@ -257,27 +257,27 @@ CONFIG_PADATA=y
>> # CONFIG_INLINE_SPIN_LOCK_BH is not set
>> # CONFIG_INLINE_SPIN_LOCK_IRQ is not set
>> # CONFIG_INLINE_SPIN_LOCK_IRQSAVE is not set
>> -CONFIG_INLINE_SPIN_UNLOCK=y
>> +# CONFIG_INLINE_SPIN_UNLOCK is not set
>> # CONFIG_INLINE_SPIN_UNLOCK_BH is not set
>> -CONFIG_INLINE_SPIN_UNLOCK_IRQ=y
>> +# CONFIG_INLINE_SPIN_UNLOCK_IRQ is not set
>> # CONFIG_INLINE_SPIN_UNLOCK_IRQRESTORE is not set
>> # CONFIG_INLINE_READ_TRYLOCK is not set
>> # CONFIG_INLINE_READ_LOCK is not set
>> # CONFIG_INLINE_READ_LOCK_BH is not set
>> # CONFIG_INLINE_READ_LOCK_IRQ is not set
>> # CONFIG_INLINE_READ_LOCK_IRQSAVE is not set
>> -CONFIG_INLINE_READ_UNLOCK=y
>> +# CONFIG_INLINE_READ_UNLOCK is not set
>> # CONFIG_INLINE_READ_UNLOCK_BH is not set
>> -CONFIG_INLINE_READ_UNLOCK_IRQ=y
>> +# CONFIG_INLINE_READ_UNLOCK_IRQ is not set
>> # CONFIG_INLINE_READ_UNLOCK_IRQRESTORE is not set
>> # CONFIG_INLINE_WRITE_TRYLOCK is not set
>> # CONFIG_INLINE_WRITE_LOCK is not set
>> # CONFIG_INLINE_WRITE_LOCK_BH is not set
>> # CONFIG_INLINE_WRITE_LOCK_IRQ is not set
>> # CONFIG_INLINE_WRITE_LOCK_IRQSAVE is not set
>> -CONFIG_INLINE_WRITE_UNLOCK=y
>> +# CONFIG_INLINE_WRITE_UNLOCK is not set
>> # CONFIG_INLINE_WRITE_UNLOCK_BH is not set
>> -CONFIG_INLINE_WRITE_UNLOCK_IRQ=y
>> +# CONFIG_INLINE_WRITE_UNLOCK_IRQ is not set
>> # CONFIG_INLINE_WRITE_UNLOCK_IRQRESTORE is not set
>> CONFIG_MUTEX_SPIN_ON_OWNER=y
>> CONFIG_FREEZER=y
>> @@ -398,7 +398,7 @@ CONFIG_HAVE_MEMBLOCK_NODE_MAP=y
>> CONFIG_ARCH_DISCARD_MEMBLOCK=y
>> # CONFIG_MEMORY_HOTPLUG is not set
>> CONFIG_PAGEFLAGS_EXTENDED=y
>> -CONFIG_SPLIT_PTLOCK_CPUS=4
>> +CONFIG_SPLIT_PTLOCK_CPUS=999999
>> # CONFIG_COMPACTION is not set
>> CONFIG_MIGRATION=y
>> CONFIG_PHYS_ADDR_T_64BIT=y
>> @@ -4255,7 +4255,7 @@ CONFIG_TIMER_STATS=y
>> # CONFIG_DEBUG_KMEMLEAK is not set
>> # CONFIG_DEBUG_RT_MUTEXES is not set
>> # CONFIG_RT_MUTEX_TESTER is not set
>> -# CONFIG_DEBUG_SPINLOCK is not set
>> +CONFIG_DEBUG_SPINLOCK=y
>> # CONFIG_DEBUG_MUTEXES is not set
>> # CONFIG_DEBUG_LOCK_ALLOC is not set
>> # CONFIG_PROVE_LOCKING is not set
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>
> --
> Andrew Cooper - Dom0 Kernel Engineer, Citrix XenServer
> T: +44 (0)1223 225 900, http://www.citrix.com
>
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: debugging spinlocks
2012-01-30 17:32 ` Ben Guthro
@ 2012-01-31 11:00 ` David Vrabel
0 siblings, 0 replies; 5+ messages in thread
From: David Vrabel @ 2012-01-31 11:00 UTC (permalink / raw)
To: Ben Guthro; +Cc: Andrew Cooper, Xen Devel, Konrad Rzeszutek
On 30/01/12 17:32, Ben Guthro wrote:
> Yes, it helped a lot!
> This seems to have resolved the hang - thanks for pointing it my way!
>
> So - if I'm reading the git log correctly, I should just have to
> cherry-pick 7a7546b377bdaa25ac77f33d9433c59f259b9688 from mainline?
Yes.
> It seems like this may be something to CC the -stable queue on, IMHO.
It's already queued.
David
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: debugging spinlocks
2012-01-30 16:27 debugging spinlocks Ben Guthro
2012-01-30 16:34 ` Andrew Cooper
@ 2012-01-30 16:40 ` Jan Beulich
1 sibling, 0 replies; 5+ messages in thread
From: Jan Beulich @ 2012-01-30 16:40 UTC (permalink / raw)
To: Ben Guthro; +Cc: Konrad Rzeszutek, Xen Devel
>>> On 30.01.12 at 17:27, Ben Guthro <ben@guthro.net> wrote:
> Any thoughts, or suggestions on how best to debug these deadlocks
> would be greatly appreciated.
For Dom0 deadlocks all I generally need are the 'e' and '0' debugkeys
(with, for the latter, stack printing depth being set sufficiently high).
If that turns out insufficient, custom debugging changes to hypervisor
or kernel are the usual next (and final) step.
Jan
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2012-01-31 11:00 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2012-01-30 16:27 debugging spinlocks Ben Guthro
2012-01-30 16:34 ` Andrew Cooper
2012-01-30 17:32 ` Ben Guthro
2012-01-31 11:00 ` David Vrabel
2012-01-30 16:40 ` Jan Beulich
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.