Re: [PATCH 0/2] fix "perf top" soft lockups under Xen

Re: [PATCH 0/2] fix "perf top" soft lockups under Xen

[Lin Ming]

(Forgot mail subject, add it)

On Sun, Apr 15, 2012 at 2:09 PM, Lin Ming <mlin@ss.pku.edu.cn> wrote:
> Hi all,
>
> These 2 patches try to fix the "perf top" soft lockups under Xen
> reported by Steven at: https://lkml.org/lkml/2012/2/9/506
>
> I tested it with 3.4-rc2 and "perf top" works well now.
>
> Steven,
> Could you please help to test it too?
>
> The soft lockup code path is:
>
> __irq_work_queue
>  arch_irq_work_raise
>    apic->send_IPI_self(IRQ_WORK_VECTOR);
>      apic_send_IPI_self
>        __default_send_IPI_shortcut
>          __xapic_wait_icr_idle
>
> static inline void __xapic_wait_icr_idle(void)
> {
>        while (native_apic_mem_read(APIC_ICR) & APIC_ICR_BUSY)
>                cpu_relax();
> }
>
> The lockup happens at above while looop.
>
> The cause is that Xen has not implemented the APIC IPI interface yet.
> Xen has IPI interface: xen_send_IPI_one, but it's only used in
> xen_smp_send_reschedule, xen_smp_send_call_function_ipi and
> xen_smp_send_call_function_single_ipi, etc.
>
> So we need to implement Xen's APIC IPI interface as Ben's patch does.
> And implement Xen's IRQ_WORK_VECTOR handler.
>
> Ben Guthro (1):
>      xen: implement apic ipi interface
>
> Lin Ming (1):
>      xen: implement IRQ_WORK_VECTOR handler
>
>  arch/x86/include/asm/xen/events.h |    1 +
>  arch/x86/xen/enlighten.c          |    7 ++
>  arch/x86/xen/smp.c                |  111 +++++++++++++++++++++++++++++++++++-
>  arch/x86/xen/smp.h                |   12 ++++
>  4 files changed, 127 insertions(+), 4 deletions(-)
>
> Any comment is appreciated.
> Lin Ming

Re: [PATCH 0/2] fix "perf top" soft lockups under Xen

[Steven Noonan]

On Sun, Apr 15, 2012 at 02:14:57PM +0800, Lin Ming wrote:
> (Forgot mail subject, add it)
> 
> On Sun, Apr 15, 2012 at 2:09 PM, Lin Ming <mlin@ss.pku.edu.cn> wrote:
> > Hi all,
> >
> > These 2 patches try to fix the "perf top" soft lockups under Xen
> > reported by Steven at: https://lkml.org/lkml/2012/2/9/506
> >
> > I tested it with 3.4-rc2 and "perf top" works well now.
> >
> > Steven,
> > Could you please help to test it too?
> >
> > The soft lockup code path is:
> >
> > __irq_work_queue
> >  arch_irq_work_raise
> >    apic->send_IPI_self(IRQ_WORK_VECTOR);
> >      apic_send_IPI_self
> >        __default_send_IPI_shortcut
> >          __xapic_wait_icr_idle
> >
> > static inline void __xapic_wait_icr_idle(void)
> > {
> >        while (native_apic_mem_read(APIC_ICR) & APIC_ICR_BUSY)
> >                cpu_relax();
> > }
> >
> > The lockup happens at above while looop.
> >
> > The cause is that Xen has not implemented the APIC IPI interface yet.
> > Xen has IPI interface: xen_send_IPI_one, but it's only used in
> > xen_smp_send_reschedule, xen_smp_send_call_function_ipi and
> > xen_smp_send_call_function_single_ipi, etc.
> >
> > So we need to implement Xen's APIC IPI interface as Ben's patch does.
> > And implement Xen's IRQ_WORK_VECTOR handler.
> >
> > Ben Guthro (1):
> >      xen: implement apic ipi interface
> >
> > Lin Ming (1):
> >      xen: implement IRQ_WORK_VECTOR handler
> >
> >  arch/x86/include/asm/xen/events.h |    1 +
> >  arch/x86/xen/enlighten.c          |    7 ++
> >  arch/x86/xen/smp.c                |  111 +++++++++++++++++++++++++++++++++++-
> >  arch/x86/xen/smp.h                |   12 ++++
> >  4 files changed, 127 insertions(+), 4 deletions(-)
> >
> > Any comment is appreciated.
> > Lin Ming

OK, finally got around to testing this. Sorry about the delay, been
rather busy at work.

These two patches definitely fix the lockups, but 'perf test' doesn't
pass every test:

	 1: vmlinux symtab matches kallsyms:
	--- start ---
	Looking at the vmlinux_path (6 entries long)
	dso__load_sym: cannot get elf header.
	Using /lib/modules/3.3.2-00002-gf57130d/build/vmlinux for symbols
	Maps only in vmlinux:
	 ffffffff81ae61b1-ffffffff81b9d7b7 0 [kernel].init.text
	 ffffffff81b9d7b8-ffffffff9fffffff 0 [kernel].exit.text
	Maps in vmlinux with a different name in kallsyms:
	Maps only in kallsyms:
	---- end ----
	vmlinux symtab matches kallsyms: Ok
	 2: detect open syscall event:
	--- start ---
	---- end ----
	detect open syscall event: Ok
	 3: detect open syscall event on all cpus:
	--- start ---
	---- end ----
	detect open syscall event on all cpus: Ok
	 4: read samples using the mmap interface:
	--- start ---
	---- end ----
	read samples using the mmap interface: Ok
	 5: parse events tests:
	--- start ---
	---- end ----
	parse events tests: Ok
	 6: Validate PERF_RECORD_* events & perf_sample fields:
	--- start ---
	perf_evlist__open: Bad file descriptor
	---- end ----
	Validate PERF_RECORD_* events & perf_sample fields: FAILED!

I couldn't figure out what exactly was failing in that test with a
quick inspection... Someone else want to take a stab at it? If someone
wants to give me patches that add extra output, I could run those too.

Anyway:

Tested-by: Steven Noonan <steven@uplinklabs.net>

- Steven

Re: [PATCH 0/2] fix "perf top" soft lockups under Xen

[Lin Ming]

On Thu, Apr 19, 2012 at 2:47 PM, Steven Noonan <steven@uplinklabs.net> wrote:

[snip]

>
> OK, finally got around to testing this. Sorry about the delay, been
> rather busy at work.
>
> These two patches definitely fix the lockups, but 'perf test' doesn't
> pass every test:

Thanks for the test!

>
>         1: vmlinux symtab matches kallsyms:
>        --- start ---
>        Looking at the vmlinux_path (6 entries long)
>        dso__load_sym: cannot get elf header.
>        Using /lib/modules/3.3.2-00002-gf57130d/build/vmlinux for symbols
>        Maps only in vmlinux:
>         ffffffff81ae61b1-ffffffff81b9d7b7 0 [kernel].init.text
>         ffffffff81b9d7b8-ffffffff9fffffff 0 [kernel].exit.text
>        Maps in vmlinux with a different name in kallsyms:
>        Maps only in kallsyms:
>        ---- end ----
>        vmlinux symtab matches kallsyms: Ok
>         2: detect open syscall event:
>        --- start ---
>        ---- end ----
>        detect open syscall event: Ok
>         3: detect open syscall event on all cpus:
>        --- start ---
>        ---- end ----
>        detect open syscall event on all cpus: Ok
>         4: read samples using the mmap interface:
>        --- start ---
>        ---- end ----
>        read samples using the mmap interface: Ok
>         5: parse events tests:
>        --- start ---
>        ---- end ----
>        parse events tests: Ok
>         6: Validate PERF_RECORD_* events & perf_sample fields:
>        --- start ---
>        perf_evlist__open: Bad file descriptor
>        ---- end ----
>        Validate PERF_RECORD_* events & perf_sample fields: FAILED!

This should be another unrelated issue.
Let me try to see if I can reproduce it.

>
> I couldn't figure out what exactly was failing in that test with a
> quick inspection... Someone else want to take a stab at it? If someone
> wants to give me patches that add extra output, I could run those too.
>
> Anyway:
>
> Tested-by: Steven Noonan <steven@uplinklabs.net>

Thanks.

Lin Ming

>
> - Steven

Re: [PATCH 0/2] fix "perf top" soft lockups under Xen

[Lin Ming]

On Thu, Apr 19, 2012 at 2:47 PM, Steven Noonan <steven@uplinklabs.net> wrote:

[snip]

>
> OK, finally got around to testing this. Sorry about the delay, been
> rather busy at work.
>
> These two patches definitely fix the lockups, but 'perf test' doesn't
> pass every test:

Thanks for the test!

>
>         1: vmlinux symtab matches kallsyms:
>        --- start ---
>        Looking at the vmlinux_path (6 entries long)
>        dso__load_sym: cannot get elf header.
>        Using /lib/modules/3.3.2-00002-gf57130d/build/vmlinux for symbols
>        Maps only in vmlinux:
>         ffffffff81ae61b1-ffffffff81b9d7b7 0 [kernel].init.text
>         ffffffff81b9d7b8-ffffffff9fffffff 0 [kernel].exit.text
>        Maps in vmlinux with a different name in kallsyms:
>        Maps only in kallsyms:
>        ---- end ----
>        vmlinux symtab matches kallsyms: Ok
>         2: detect open syscall event:
>        --- start ---
>        ---- end ----
>        detect open syscall event: Ok
>         3: detect open syscall event on all cpus:
>        --- start ---
>        ---- end ----
>        detect open syscall event on all cpus: Ok
>         4: read samples using the mmap interface:
>        --- start ---
>        ---- end ----
>        read samples using the mmap interface: Ok
>         5: parse events tests:
>        --- start ---
>        ---- end ----
>        parse events tests: Ok
>         6: Validate PERF_RECORD_* events & perf_sample fields:
>        --- start ---
>        perf_evlist__open: Bad file descriptor
>        ---- end ----
>        Validate PERF_RECORD_* events & perf_sample fields: FAILED!

This should be another unrelated issue.
Let me try to see if I can reproduce it.

>
> I couldn't figure out what exactly was failing in that test with a
> quick inspection... Someone else want to take a stab at it? If someone
> wants to give me patches that add extra output, I could run those too.
>
> Anyway:
>
> Tested-by: Steven Noonan <steven@uplinklabs.net>

Thanks.

Lin Ming

>
> - Steven

Re: [PATCH 0/2] fix "perf top" soft lockups under Xen

[Steven Noonan]

On Sun, Apr 15, 2012 at 02:14:57PM +0800, Lin Ming wrote:
> (Forgot mail subject, add it)
> 
> On Sun, Apr 15, 2012 at 2:09 PM, Lin Ming <mlin@ss.pku.edu.cn> wrote:
> > Hi all,
> >
> > These 2 patches try to fix the "perf top" soft lockups under Xen
> > reported by Steven at: https://lkml.org/lkml/2012/2/9/506
> >
> > I tested it with 3.4-rc2 and "perf top" works well now.
> >
> > Steven,
> > Could you please help to test it too?
> >
> > The soft lockup code path is:
> >
> > __irq_work_queue
> >  arch_irq_work_raise
> >    apic->send_IPI_self(IRQ_WORK_VECTOR);
> >      apic_send_IPI_self
> >        __default_send_IPI_shortcut
> >          __xapic_wait_icr_idle
> >
> > static inline void __xapic_wait_icr_idle(void)
> > {
> >        while (native_apic_mem_read(APIC_ICR) & APIC_ICR_BUSY)
> >                cpu_relax();
> > }
> >
> > The lockup happens at above while looop.
> >
> > The cause is that Xen has not implemented the APIC IPI interface yet.
> > Xen has IPI interface: xen_send_IPI_one, but it's only used in
> > xen_smp_send_reschedule, xen_smp_send_call_function_ipi and
> > xen_smp_send_call_function_single_ipi, etc.
> >
> > So we need to implement Xen's APIC IPI interface as Ben's patch does.
> > And implement Xen's IRQ_WORK_VECTOR handler.
> >
> > Ben Guthro (1):
> >      xen: implement apic ipi interface
> >
> > Lin Ming (1):
> >      xen: implement IRQ_WORK_VECTOR handler
> >
> >  arch/x86/include/asm/xen/events.h |    1 +
> >  arch/x86/xen/enlighten.c          |    7 ++
> >  arch/x86/xen/smp.c                |  111 +++++++++++++++++++++++++++++++++++-
> >  arch/x86/xen/smp.h                |   12 ++++
> >  4 files changed, 127 insertions(+), 4 deletions(-)
> >
> > Any comment is appreciated.
> > Lin Ming

OK, finally got around to testing this. Sorry about the delay, been
rather busy at work.

These two patches definitely fix the lockups, but 'perf test' doesn't
pass every test:

	 1: vmlinux symtab matches kallsyms:
	--- start ---
	Looking at the vmlinux_path (6 entries long)
	dso__load_sym: cannot get elf header.
	Using /lib/modules/3.3.2-00002-gf57130d/build/vmlinux for symbols
	Maps only in vmlinux:
	 ffffffff81ae61b1-ffffffff81b9d7b7 0 [kernel].init.text
	 ffffffff81b9d7b8-ffffffff9fffffff 0 [kernel].exit.text
	Maps in vmlinux with a different name in kallsyms:
	Maps only in kallsyms:
	---- end ----
	vmlinux symtab matches kallsyms: Ok
	 2: detect open syscall event:
	--- start ---
	---- end ----
	detect open syscall event: Ok
	 3: detect open syscall event on all cpus:
	--- start ---
	---- end ----
	detect open syscall event on all cpus: Ok
	 4: read samples using the mmap interface:
	--- start ---
	---- end ----
	read samples using the mmap interface: Ok
	 5: parse events tests:
	--- start ---
	---- end ----
	parse events tests: Ok
	 6: Validate PERF_RECORD_* events & perf_sample fields:
	--- start ---
	perf_evlist__open: Bad file descriptor
	---- end ----
	Validate PERF_RECORD_* events & perf_sample fields: FAILED!

I couldn't figure out what exactly was failing in that test with a
quick inspection... Someone else want to take a stab at it? If someone
wants to give me patches that add extra output, I could run those too.

Anyway:

Tested-by: Steven Noonan <steven@uplinklabs.net>

- Steven

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [PATCH 0/2] fix "perf top" soft lockups under Xen

[Lin Ming]

(Forgot mail subject, add it)

On Sun, Apr 15, 2012 at 2:09 PM, Lin Ming <mlin@ss.pku.edu.cn> wrote:
> Hi all,
>
> These 2 patches try to fix the "perf top" soft lockups under Xen
> reported by Steven at: https://lkml.org/lkml/2012/2/9/506
>
> I tested it with 3.4-rc2 and "perf top" works well now.
>
> Steven,
> Could you please help to test it too?
>
> The soft lockup code path is:
>
> __irq_work_queue
>  arch_irq_work_raise
>    apic->send_IPI_self(IRQ_WORK_VECTOR);
>      apic_send_IPI_self
>        __default_send_IPI_shortcut
>          __xapic_wait_icr_idle
>
> static inline void __xapic_wait_icr_idle(void)
> {
>        while (native_apic_mem_read(APIC_ICR) & APIC_ICR_BUSY)
>                cpu_relax();
> }
>
> The lockup happens at above while looop.
>
> The cause is that Xen has not implemented the APIC IPI interface yet.
> Xen has IPI interface: xen_send_IPI_one, but it's only used in
> xen_smp_send_reschedule, xen_smp_send_call_function_ipi and
> xen_smp_send_call_function_single_ipi, etc.
>
> So we need to implement Xen's APIC IPI interface as Ben's patch does.
> And implement Xen's IRQ_WORK_VECTOR handler.
>
> Ben Guthro (1):
>      xen: implement apic ipi interface
>
> Lin Ming (1):
>      xen: implement IRQ_WORK_VECTOR handler
>
>  arch/x86/include/asm/xen/events.h |    1 +
>  arch/x86/xen/enlighten.c          |    7 ++
>  arch/x86/xen/smp.c                |  111 +++++++++++++++++++++++++++++++++++-
>  arch/x86/xen/smp.h                |   12 ++++
>  4 files changed, 127 insertions(+), 4 deletions(-)
>
> Any comment is appreciated.
> Lin Ming