[BUG][4.5.1]xl cpupool-create segfault (with config file parameter)

[BUG][4.5.1]xl cpupool-create segfault (with config file parameter)

[Benjamin Beier]

Since xen version 4.5.1 there is a bug with the xl cpupool-create command.
It always throws a segmentation fault when used with a config file as 
parameter.
The content of the config file does not really matter (valid or invalid).
It also fails with the example config file in /etc/xen/cpupool or with 
the dryrun flag.
You never get any output apart from "segmentation fault".
If you create exactly the same cpupool without using a config file it 
works absolutely fine.
Tested on multiple systems and it seems to be reproducible for everyone.
STrace shows that the command fails right after reading the content of 
the config file.

Regards,
Benjamin

Re: [BUG][4.5.1]xl cpupool-create segfault (with config file parameter)

[Ian Campbell]

On Wed, 2015-08-12 at 13:03 +0200, Benjamin Beier wrote:
> Since xen version 4.5.1 there is a bug with the xl cpupool-create 
> command.
> It always throws a segmentation fault when used with a config file as 
> parameter.
> The content of the config file does not really matter (valid or invalid).
> It also fails with the example config file in /etc/xen/cpupool or with 
> the dryrun flag.
> You never get any output apart from "segmentation fault".
> If you create exactly the same cpupool without using a config file it 
> works absolutely fine.
> Tested on multiple systems and it seems to be reproducible for everyone.
> STrace shows that the command fails right after reading the content of 
> the config file.

Please can you capture a backtrace by running it under gdb. Valgrind might
also have something interesting to say.

FWIW this doesn't happen on the current development branch:
    # cat foo
    Testing
    # xl cpupool-create foo
    foo:1: config parsing error near `Testing': lexical error
    Failed to parse config file: Invalid argument

But I don't see any obviously related looking fixes in the commit log.

Was 4.5.0 ok?

Ian.

Re: [BUG][4.5.1]xl cpupool-create segfault (with config file parameter)

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 3083 bytes --]

On Wed, 2015-08-12 at 12:28 +0100, Ian Campbell wrote:
> On Wed, 2015-08-12 at 13:03 +0200, Benjamin Beier wrote:

> > You never get any output apart from "segmentation fault".
> > If you create exactly the same cpupool without using a config file it 
> > works absolutely fine.
> > Tested on multiple systems and it seems to be reproducible for everyone.
> > STrace shows that the command fails right after reading the content of 
> > the config file.
> 
> Please can you capture a backtrace by running it under gdb. Valgrind might
> also have something interesting to say.
> 
Yep, a gdb backtrace would be really helpful.

I've 'played' with cpupools quite a bit in the last few months (during
the whole Xen 4.6 release cycle), and I don't remember seeing anything
like this... :-/

> FWIW this doesn't happen on the current development branch:
>     # cat foo
>     Testing
>     # xl cpupool-create foo
>     foo:1: config parsing error near `Testing': lexical error
>     Failed to parse config file: Invalid argument
> 
Just tried with the example cpupool config file, from /etc/xen/cpupool,
and it works. I'm also on staging (i.e., 4-6.-rc1):

  root@Zhaman:~# cat /etc/xen/cpupool  
  #============================================================================
  # Configuration setup for 'xm cpupool-create' or 'xl cpupool-create'.
  # This script sets the parameters used when a cpupool is created using
  # 'xm cpupool-create' or 'xl cpupool-create'.
  # You use a separate script for each cpupool you want to create, or 
  # you can set the parameters for the cpupool on the xm command line.
  #============================================================================

  # the name of the new cpupool
  name = "Example-Cpupool"

  # the scheduler to use: valid are e.g. credit, credit2 and rtds
  sched = "credit"

  # list of cpus to use
  cpus = ["2", "3"]

  root@Zhaman:~# xl cpupool-create /etc/xen/cpupool
  cpu 2 illegal or not free
  root@Zhaman:~# xl cpupool-cpu-remove Pool-0 2,3
  root@Zhaman:~# xl cpupool-create /etc/xen/cpupool
  Using config file "/etc/xen/cpupool"
  cpupool name:   Example-Cpupool
  scheduler:      credit
  number of cpus: 2
  root@Zhaman:~# xl cpupool-list 
  Name               CPUs   Sched     Active   Domain count
  Pool-0              14    credit       y          1
  Example-Cpupool      2    credit       y          0

> But I don't see any obviously related looking fixes in the commit log.
> 
Since 4.5, there has been changes touching both xl and libxl bits of
cpupools. E.g.:

  commit a86eecbbf5155aa5b4ec02b6c5e41baf1a7f49de
  xl: enable using ranges of pCPUs when creating cpupools

However, as I said, I don't remember facing (and fixing) such error.

> Was 4.5.0 ok?
> 
> Ian.

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [BUG][4.5.1]xl cpupool-create segfault (with config file parameter)

[Benjamin Beier]

On 08/12/2015 01:28 PM, Ian Campbell wrote:
> On Wed, 2015-08-12 at 13:03 +0200, Benjamin Beier wrote:
>> Since xen version 4.5.1 there is a bug with the xl cpupool-create
>> command.
>> It always throws a segmentation fault when used with a config file as
>> parameter.
>> The content of the config file does not really matter (valid or invalid).
>> It also fails with the example config file in /etc/xen/cpupool or with
>> the dryrun flag.
>> You never get any output apart from "segmentation fault".
>> If you create exactly the same cpupool without using a config file it
>> works absolutely fine.
>> Tested on multiple systems and it seems to be reproducible for everyone.
>> STrace shows that the command fails right after reading the content of
>> the config file.
>
> Please can you capture a backtrace by running it under gdb. Valgrind might
> also have something interesting to say.
>
> FWIW this doesn't happen on the current development branch:
>      # cat foo
>      Testing
>      # xl cpupool-create foo
>      foo:1: config parsing error near `Testing': lexical error
>      Failed to parse config file: Invalid argument
>
> But I don't see any obviously related looking fixes in the commit log.
>
> Was 4.5.0 ok?

I am not 100% sure about that, because I only had a test system running 
with it.
As you guys didn't notice any problems I should mention that my servers 
are running Gentoo Linux.
Maybe this is a Gentoo specific problem and not relevant at all for 
upstream xen.
Or it is an upstream problem that only shows up on the custom-compiled 
gentoo systems.

# cat testing
foo

# xl cpupool-create testing
Segmentation fault

# strace xl cpupool-create testing
(...)
open("testing", O_RDONLY)               = 7
fstat(7, {st_mode=S_IFREG|0644, st_size=4, ...}) = 0
fstat(7, {st_mode=S_IFREG|0644, st_size=4, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) 
= 0x776ec3228000
read(7, "foo\n", 4096)                  = 4
close(7)                                = 0
munmap(0x776ec3228000, 4096)            = 0
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0} ---
+++ killed by SIGSEGV (core dumped) +++
Segmentation fault

Program received signal SIGSEGV, Segmentation fault.
0x0000000000418db5 in main_cpupoolcreate (argc=<optimized out>, 
argv=0x7fffffffda50) at xl_cmdimpl.c:7102
7102    xl_cmdimpl.c: No such file or directory.
(gdb) bt
#0  0x0000000000418db5 in main_cpupoolcreate (argc=<optimized out>, 
argv=0x7fffffffda50) at xl_cmdimpl.c:7102
#1  0x0000000000407c79 in main (argc=2, argv=0x7fffffffda50) at xl.c:361

Just tell me if this is a gentoo-only thing and I will open a bug report 
there.

Regards,
Benjamin

Re: [BUG][4.5.1]xl cpupool-create segfault (with config file parameter)

[Ian Campbell]

On Thu, 2015-08-13 at 13:13 +0200, Benjamin Beier wrote:
> 
> Program received signal SIGSEGV, Segmentation fault.
> 0x0000000000418db5 in main_cpupoolcreate (argc=, 
> argv=0x7fffffffda50) at xl_cmdimpl.c:7102
> 7102    xl_cmdimpl.c: No such file or directory.
> (gdb) bt
> #0  0x0000000000418db5 in main_cpupoolcreate (argc=<optimized out>, 
> argv=0x7fffffffda50) at xl_cmdimpl.c:7102
> #1  0x0000000000407c79 in main (argc=2, argv=0x7fffffffda50) at xl.c:361

In "git show RELEASE-4.5.1:tools/libxl/xl_cmdimpl.c" line 7102 is a blank
line. The next line is:
     config = xlu_cfg_init(stderr, config_src);
which might be the culprit but also might not be.

Are you running pristine 4.5.1 or are the (perhaps gentoo added) patches?
Can you figure out what line 7102 of that might be.

> Just tell me if this is a gentoo-only thing and I will open a bug report 
> there.

It's not clear yet, since it might be a backport of a buggy patch, but it's
starting to look a bit that way. Once xl_cmdimpl.c:7102 and the list of
patches is clarified I think we'll know one way or another.

Ian

Re: [BUG][4.5.1]xl cpupool-create segfault (with config file parameter)

[Benjamin Beier]

On 08/13/2015 01:29 PM, Ian Campbell wrote:
> On Thu, 2015-08-13 at 13:13 +0200, Benjamin Beier wrote:
>>
>> Program received signal SIGSEGV, Segmentation fault.
>> 0x0000000000418db5 in main_cpupoolcreate (argc=,
>> argv=0x7fffffffda50) at xl_cmdimpl.c:7102
>> 7102    xl_cmdimpl.c: No such file or directory.
>> (gdb) bt
>> #0  0x0000000000418db5 in main_cpupoolcreate (argc=<optimized out>,
>> argv=0x7fffffffda50) at xl_cmdimpl.c:7102
>> #1  0x0000000000407c79 in main (argc=2, argv=0x7fffffffda50) at xl.c:361
>
> In "git show RELEASE-4.5.1:tools/libxl/xl_cmdimpl.c" line 7102 is a blank
> line. The next line is:
>       config = xlu_cfg_init(stderr, config_src);
> which might be the culprit but also might not be.
>
> Are you running pristine 4.5.1 or are the (perhaps gentoo added) patches?
> Can you figure out what line 7102 of that might be.
>
>> Just tell me if this is a gentoo-only thing and I will open a bug report
>> there.
>
> It's not clear yet, since it might be a backport of a buggy patch, but it's
> starting to look a bit that way. Once xl_cmdimpl.c:7102 and the list of
> patches is clarified I think we'll know one way or another.
>
> Ian

I am not an expert for ebuilds and the xen-tools ebuild is rather 
complicated. I filed a gentoo bug report and linked to this upstream 
report here. Hopefully one of the gentoo ebuild maintainers has further 
information for you.

https://bugs.gentoo.org/show_bug.cgi?id=557458

Regards,
Benjamin

Re: [BUG][4.5.1]xl cpupool-create segfault (with config file parameter)

[Jim Fehlig]

Benjamin Beier wrote:
> On 08/12/2015 01:28 PM, Ian Campbell wrote:
>> On Wed, 2015-08-12 at 13:03 +0200, Benjamin Beier wrote:
>>> Since xen version 4.5.1 there is a bug with the xl cpupool-create
>>> command.
>>> It always throws a segmentation fault when used with a config file as
>>> parameter.
>>> The content of the config file does not really matter (valid or
>>> invalid).
>>> It also fails with the example config file in /etc/xen/cpupool or with
>>> the dryrun flag.
>>> You never get any output apart from "segmentation fault".
>>> If you create exactly the same cpupool without using a config file it
>>> works absolutely fine.
>>> Tested on multiple systems and it seems to be reproducible for
>>> everyone.
>>> STrace shows that the command fails right after reading the content of
>>> the config file.
>>
>> Please can you capture a backtrace by running it under gdb. Valgrind
>> might
>> also have something interesting to say.
>>
>> FWIW this doesn't happen on the current development branch:
>>      # cat foo
>>      Testing
>>      # xl cpupool-create foo
>>      foo:1: config parsing error near `Testing': lexical error
>>      Failed to parse config file: Invalid argument
>>
>> But I don't see any obviously related looking fixes in the commit log.
>>
>> Was 4.5.0 ok?
>
> I am not 100% sure about that, because I only had a test system
> running with it.
> As you guys didn't notice any problems I should mention that my
> servers are running Gentoo Linux.
> Maybe this is a Gentoo specific problem and not relevant at all for
> upstream xen.
> Or it is an upstream problem that only shows up on the custom-compiled
> gentoo systems.
>
> # cat testing
> foo
>
> # xl cpupool-create testing
> Segmentation fault
>
> # strace xl cpupool-create testing
> (...)
> open("testing", O_RDONLY)               = 7
> fstat(7, {st_mode=S_IFREG|0644, st_size=4, ...}) = 0
> fstat(7, {st_mode=S_IFREG|0644, st_size=4, ...}) = 0
> mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
> 0) = 0x776ec3228000
> read(7, "foo\n", 4096)                  = 4
> close(7)                                = 0
> munmap(0x776ec3228000, 4096)            = 0
> --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0} ---
> +++ killed by SIGSEGV (core dumped) +++
> Segmentation fault
>
> Program received signal SIGSEGV, Segmentation fault.
> 0x0000000000418db5 in main_cpupoolcreate (argc=<optimized out>,
> argv=0x7fffffffda50) at xl_cmdimpl.c:7102
> 7102    xl_cmdimpl.c: No such file or directory.
> (gdb) bt
> #0  0x0000000000418db5 in main_cpupoolcreate (argc=<optimized out>,
> argv=0x7fffffffda50) at xl_cmdimpl.c:7102
> #1  0x0000000000407c79 in main (argc=2, argv=0x7fffffffda50) at xl.c:361

Sorry for jumping in late, but I encountered this issue yesterday with
4.5.1. Commit 705c9e12 fixes it and should probably be backported to the
4.5 branch.

Regards,
Jim

Re: [BUG][4.5.1]xl cpupool-create segfault (with config file parameter)

[Ian Campbell]

On Wed, 2015-08-26 at 16:25 -0600, Jim Fehlig wrote:
> Sorry for jumping in late, but I encountered this issue yesterday with
> 4.5.1. Commit 705c9e12 fixes it and should probably be backported to the
> 4.5 branch.

Adding Ian J (tools stable maintainer) to To:
> 
> Regards,
> Jim
> 

Re: [BUG][4.5.1]xl cpupool-create segfault (with config file parameter)

[Ian Jackson]

Ian Campbell writes ("Re: [Xen-devel] [BUG][4.5.1]xl cpupool-create segfault (with config file parameter)"):
> On Wed, 2015-08-26 at 16:25 -0600, Jim Fehlig wrote:
> > Sorry for jumping in late, but I encountered this issue yesterday with
> > 4.5.1. Commit 705c9e12 fixes it and should probably be backported to the
> > 4.5 branch.
> 
> Adding Ian J (tools stable maintainer) to To:

Queued, thanks.

Ian.