FreeBSD crushtool crashing while checking a crushmap

FreeBSD crushtool crashing while checking a crushmap

[Willem Jan Withagen]

Hi,

While running run-cli-tests one of the tests dumps while checking if the
previous command has build the correct crushmap.
I've verified the actual crushmap against one build on CentOS, and they
are byte for byte equal.

Before I rebased (from beginning of januari) all these tests completed
just fine.

During checking I get the following dump:

# src/crushtool -i
src/tes/cli/crushtool/check-overlapped-rules.crushmap.ref --check
Assertion failed: (this->_map.find(inter_val) == this->_map.end()),
function gap_insert, file
/usr/local/include/boost/icl/interval_base_map.hpp, line 555.
*** Caught signal (Abort trap) **
 in thread 803e15000
 ceph version Development (no_version)
 1: 0x86f865 <_ZN4ceph9BackTraceC2Ei+0x35> at
/usr/srcs/Ceph/work/ceph/src/crushtool
 2: 0x86e5f9 <_ZL19handle_fatal_signali+0xa9> at
/usr/srcs/Ceph/work/ceph/src/crushtool
 3: 0x801811c7d <pthread_sigmask+0x50d> at /lib/libthr.so.3
 4: 0x8018112b2 <pthread_getspecific+0xe22> at /lib/libthr.so.3
2016-02-06 22:55:09.075189 803e15000 -1

Deleted remainder of the output....

And I appreciate any pointers in helping me debugging this.

--WjW

Re: FreeBSD crushtool crashing while checking a crushmap

[kefu chai]

[-- Attachment #1: Type: text/plain, Size: 1665 bytes --]

On Sun, Feb 7, 2016 at 6:12 AM, Willem Jan Withagen <wjw@digiware.nl> wrote:
> Hi,
>
> While running run-cli-tests one of the tests dumps while checking if the
> previous command has build the correct crushmap.
> I've verified the actual crushmap against one build on CentOS, and they
> are byte for byte equal.
>
> Before I rebased (from beginning of januari) all these tests completed
> just fine.
>
> During checking I get the following dump:
>
> # src/crushtool -i
> src/tes/cli/crushtool/check-overlapped-rules.crushmap.ref --check
> Assertion failed: (this->_map.find(inter_val) == this->_map.end()),
> function gap_insert, file
> /usr/local/include/boost/icl/interval_base_map.hpp, line 555.
> *** Caught signal (Abort trap) **
>  in thread 803e15000
>  ceph version Development (no_version)
>  1: 0x86f865 <_ZN4ceph9BackTraceC2Ei+0x35> at
> /usr/srcs/Ceph/work/ceph/src/crushtool
>  2: 0x86e5f9 <_ZL19handle_fatal_signali+0xa9> at
> /usr/srcs/Ceph/work/ceph/src/crushtool
>  3: 0x801811c7d <pthread_sigmask+0x50d> at /lib/libthr.so.3
>  4: 0x8018112b2 <pthread_getspecific+0xe22> at /lib/libthr.so.3
> 2016-02-06 22:55:09.075189 803e15000 -1
>
> Deleted remainder of the output....
>
> And I appreciate any pointers in helping me debugging this.

Willem, this is a newly added feature for detecting overlapped crush
rules. i just created a minimal reproducer, hopefull with which one is
able to have the same crash. it

>
> --WjW
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Regards
Kefu Chai

[-- Attachment #2: test.cc --]
[-- Type: application/octet-stream, Size: 1523 bytes --]

#include <boost/icl/interval_map.hpp>
#include <vector>
#include <string>
#include <utility>

using namespace std;

enum class Type {
  replicated,
  erasure,
};

struct Rule {
  string name;
  int ruleset;
  Type type;
  int min_size;
  int max_size;
};

void check_overlapped_rules(const vector<Rule>& rules)
{  
  namespace icl = boost::icl;
  typedef std::set<string> RuleNames;
  typedef icl::interval_map<int, RuleNames> Rules;
  // <ruleset, type> => interval_map<size, {names}>
  typedef std::map<std::pair<int, Type>, Rules> RuleSets;
  using interval = icl::interval<int>;

  // mimic the logic of crush_find_rule(), but it only return the first matched
  // one, but I am collecting all of them by the overlapped sizes.
  RuleSets rulesets;
  for (auto rule : rules) {
    Rules& rules = rulesets[{rule.ruleset,
                             rule.type}];
    rules += std::make_pair(interval::closed(rule.min_size,
                                             rule.max_size),
                            RuleNames{rule.name});
  }
}

int main()
{
  vector<Rule> rules = {Rule{"0", 0, Type::replicated, 1, 3},
                        Rule{"1", 0, Type::replicated, 1, 1},
                        Rule{"2", 0, Type::replicated, 1, 2},
                        Rule{"3", 0, Type::replicated, 2, 3},
                        Rule{"4", 0, Type::replicated, 4, 5},
                        Rule{"5", 0, Type::erasure, 1, 10},
                        Rule{"6", 1, Type::erasure, 1, 10}};
  check_overlapped_rules(rules);
}

Re: FreeBSD crushtool crashing while checking a crushmap

[kefu chai]

[-- Attachment #1: Type: text/plain, Size: 3485 bytes --]

sorry, just sent an out-dated reply.

On Sun, Feb 7, 2016 at 6:12 AM, Willem Jan Withagen <wjw@digiware.nl> wrote:
> Hi,
>
> While running run-cli-tests one of the tests dumps while checking if the
> previous command has build the correct crushmap.
> I've verified the actual crushmap against one build on CentOS, and they
> are byte for byte equal.
>
> Before I rebased (from beginning of januari) all these tests completed
> just fine.
>
> During checking I get the following dump:
>
> # src/crushtool -i
> src/tes/cli/crushtool/check-overlapped-rules.crushmap.ref --check
> Assertion failed: (this->_map.find(inter_val) == this->_map.end()),
> function gap_insert, file
> /usr/local/include/boost/icl/interval_base_map.hpp, line 555.
> *** Caught signal (Abort trap) **
>  in thread 803e15000
>  ceph version Development (no_version)
>  1: 0x86f865 <_ZN4ceph9BackTraceC2Ei+0x35> at
> /usr/srcs/Ceph/work/ceph/src/crushtool
>  2: 0x86e5f9 <_ZL19handle_fatal_signali+0xa9> at
> /usr/srcs/Ceph/work/ceph/src/crushtool
>  3: 0x801811c7d <pthread_sigmask+0x50d> at /lib/libthr.so.3
>  4: 0x8018112b2 <pthread_getspecific+0xe22> at /lib/libthr.so.3
> 2016-02-06 22:55:09.075189 803e15000 -1
>
> Deleted remainder of the output....
>
> And I appreciate any pointers in helping me debugging this.

hey Willem, thanks for looking into this. turns out this is a known issue of
boost 1.55. just posted a pull request to workaround it at
https://github.com/ceph/ceph/pull/7560.

also did i create a minimal reproducer, it can help with reproducing this
issue and verify the fix.

you can compile it using:

$ clang++ -std=c++11 -O0 -g test.cc -lm -o test

-- 
Regards
Kefu Chai

On Mon, Feb 8, 2016 at 1:52 PM, kefu chai <tchaikov@gmail.com> wrote:
> On Sun, Feb 7, 2016 at 6:12 AM, Willem Jan Withagen <wjw@digiware.nl> wrote:
>> Hi,
>>
>> While running run-cli-tests one of the tests dumps while checking if the
>> previous command has build the correct crushmap.
>> I've verified the actual crushmap against one build on CentOS, and they
>> are byte for byte equal.
>>
>> Before I rebased (from beginning of januari) all these tests completed
>> just fine.
>>
>> During checking I get the following dump:
>>
>> # src/crushtool -i
>> src/tes/cli/crushtool/check-overlapped-rules.crushmap.ref --check
>> Assertion failed: (this->_map.find(inter_val) == this->_map.end()),
>> function gap_insert, file
>> /usr/local/include/boost/icl/interval_base_map.hpp, line 555.
>> *** Caught signal (Abort trap) **
>>  in thread 803e15000
>>  ceph version Development (no_version)
>>  1: 0x86f865 <_ZN4ceph9BackTraceC2Ei+0x35> at
>> /usr/srcs/Ceph/work/ceph/src/crushtool
>>  2: 0x86e5f9 <_ZL19handle_fatal_signali+0xa9> at
>> /usr/srcs/Ceph/work/ceph/src/crushtool
>>  3: 0x801811c7d <pthread_sigmask+0x50d> at /lib/libthr.so.3
>>  4: 0x8018112b2 <pthread_getspecific+0xe22> at /lib/libthr.so.3
>> 2016-02-06 22:55:09.075189 803e15000 -1
>>
>> Deleted remainder of the output....
>>
>> And I appreciate any pointers in helping me debugging this.
>
> Willem, this is a newly added feature for detecting overlapped crush
> rules. i just created a minimal reproducer, hopefull with which one is
> able to have the same crash. it
>
>>
>> --WjW
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
>
> --
> Regards
> Kefu Chai



-- 
Regards
Kefu Chai

[-- Attachment #2: test.cc --]
[-- Type: application/octet-stream, Size: 1755 bytes --]

// to workaround https://svn.boost.org/trac/boost/ticket/9501
#ifdef _LIBCPP_VERSION
#include <boost/lexical_cast.hpp>
// #include <boost/version.hpp>
#if BOOST_VERSION < 105600
#define ICL_USE_BOOST_MOVE_IMPLEMENTATION
#endif
#endif
#include <boost/icl/interval_map.hpp>
#include <vector>
#include <string>
#include <utility>

using namespace std;

enum class Type {
  replicated,
  erasure,
};

struct Rule {
  string name;
  int ruleset;
  Type type;
  int min_size;
  int max_size;
};

void check_overlapped_rules(const vector<Rule>& rules)
{
  namespace icl = boost::icl;
  typedef std::set<string> RuleNames;
  typedef icl::interval_map<int, RuleNames> Rules;
  // <ruleset, type> => interval_map<size, {names}>
  typedef std::map<std::pair<int, Type>, Rules> RuleSets;
  using interval = icl::interval<int>;

  // mimic the logic of crush_find_rule(), but it only return the first matched
  // one, but I am collecting all of them by the overlapped sizes.
  RuleSets rulesets;
  for (auto rule : rules) {
    Rules& rules = rulesets[{rule.ruleset,
                             rule.type}];
    rules += std::make_pair(interval::closed(rule.min_size,
                                             rule.max_size),
                            RuleNames{rule.name});
  }
}

int main()
{
  vector<Rule> rules = {Rule{"0", 0, Type::replicated, 1, 3},
                        Rule{"1", 0, Type::replicated, 1, 1},
                        Rule{"2", 0, Type::replicated, 1, 2},
                        Rule{"3", 0, Type::replicated, 2, 3},
                        Rule{"4", 0, Type::replicated, 4, 5},
                        Rule{"5", 0, Type::erasure, 1, 10},
                        Rule{"6", 1, Type::erasure, 1, 10}};
  check_overlapped_rules(rules);
}

Re: FreeBSD crushtool crashing while checking a crushmap

[Willem Jan Withagen]

On 8-2-2016 06:53, kefu chai wrote:
> sorry, just sent an out-dated reply.
>
> On Sun, Feb 7, 2016 at 6:12 AM, Willem Jan Withagen <wjw@digiware.nl> wrote:
>> Hi,
>>
>> While running run-cli-tests one of the tests dumps while checking if the
>> previous command has build the correct crushmap.
>> I've verified the actual crushmap against one build on CentOS, and they
>> are byte for byte equal.
>>
>> Before I rebased (from beginning of januari) all these tests completed
>> just fine.
>>
>> During checking I get the following dump:
>>
>> # src/crushtool -i
>> src/tes/cli/crushtool/check-overlapped-rules.crushmap.ref --check
>> Assertion failed: (this->_map.find(inter_val) == this->_map.end()),
>> function gap_insert, file
>> /usr/local/include/boost/icl/interval_base_map.hpp, line 555.
>> *** Caught signal (Abort trap) **
>>   in thread 803e15000
>>   ceph version Development (no_version)
>>   1: 0x86f865 <_ZN4ceph9BackTraceC2Ei+0x35> at
>> /usr/srcs/Ceph/work/ceph/src/crushtool
>>   2: 0x86e5f9 <_ZL19handle_fatal_signali+0xa9> at
>> /usr/srcs/Ceph/work/ceph/src/crushtool
>>   3: 0x801811c7d <pthread_sigmask+0x50d> at /lib/libthr.so.3
>>   4: 0x8018112b2 <pthread_getspecific+0xe22> at /lib/libthr.so.3
>> 2016-02-06 22:55:09.075189 803e15000 -1
>>
>> Deleted remainder of the output....
>>
>> And I appreciate any pointers in helping me debugging this.
>
> hey Willem, thanks for looking into this. turns out this is a known issue of
> boost 1.55. just posted a pull request to workaround it at
> https://github.com/ceph/ceph/pull/7560.
>
> also did i create a minimal reproducer, it can help with reproducing this
> issue and verify the fix.
>
> you can compile it using:
>
> $ clang++ -std=c++11 -O0 -g test.cc -lm -o test
>

pffh,

That's a relieve, since I would otherwise be quite a chase....
and yes, I'm still at boost 1.55.

Perhaps in the test check-overlapped-rules.t an extra test:
	cmp "$TESTDIR/check-overlapped-rules.crushmap" 
"$TESTDIR/check-overlapped-rules.crushmap.ref"

Which will at least tell you that the output was consistent with what 
sould be there.
And it'll pin-point the source of the error perhaps a bit better.

Can I just "accept" a pull request in my fork without running into trouble
when I rebase?

--WjW

Re: FreeBSD crushtool crashing while checking a crushmap

[Willem Jan Withagen]

On 8-2-2016 12:01, Willem Jan Withagen wrote:
> On 8-2-2016 06:53, kefu chai wrote:
>> sorry, just sent an out-dated reply.
>>
>> On Sun, Feb 7, 2016 at 6:12 AM, Willem Jan Withagen <wjw@digiware.nl>
>> wrote:
>>> Hi,
>>>
>>> While running run-cli-tests one of the tests dumps while checking if the
>>> previous command has build the correct crushmap.
>>> I've verified the actual crushmap against one build on CentOS, and they
>>> are byte for byte equal.
>>>
>>> Before I rebased (from beginning of januari) all these tests completed
>>> just fine.
>>>
>>> During checking I get the following dump:
>>>
>>> # src/crushtool -i
>>> src/tes/cli/crushtool/check-overlapped-rules.crushmap.ref --check
>>> Assertion failed: (this->_map.find(inter_val) == this->_map.end()),
>>> function gap_insert, file
>>> /usr/local/include/boost/icl/interval_base_map.hpp, line 555.
>>> *** Caught signal (Abort trap) **
>>>   in thread 803e15000
>>>   ceph version Development (no_version)
>>>   1: 0x86f865 <_ZN4ceph9BackTraceC2Ei+0x35> at
>>> /usr/srcs/Ceph/work/ceph/src/crushtool
>>>   2: 0x86e5f9 <_ZL19handle_fatal_signali+0xa9> at
>>> /usr/srcs/Ceph/work/ceph/src/crushtool
>>>   3: 0x801811c7d <pthread_sigmask+0x50d> at /lib/libthr.so.3
>>>   4: 0x8018112b2 <pthread_getspecific+0xe22> at /lib/libthr.so.3
>>> 2016-02-06 22:55:09.075189 803e15000 -1
>>>
>>> Deleted remainder of the output....
>>>
>>> And I appreciate any pointers in helping me debugging this.
>>
>> hey Willem, thanks for looking into this. turns out this is a known
>> issue of
>> boost 1.55. just posted a pull request to workaround it at
>> https://github.com/ceph/ceph/pull/7560.
>>
>> also did i create a minimal reproducer, it can help with reproducing this
>> issue and verify the fix.
>>
>> you can compile it using:
>>
>> $ clang++ -std=c++11 -O0 -g test.cc -lm -o test
>>
> 
> pffh,
> 
> That's a relieve, since I would otherwise be quite a chase....
> and yes, I'm still at boost 1.55.
> 
> Perhaps in the test check-overlapped-rules.t an extra test:
>     cmp "$TESTDIR/check-overlapped-rules.crushmap"
> "$TESTDIR/check-overlapped-rules.crushmap.ref"
> 
> Which will at least tell you that the output was consistent with what
> sould be there.
> And it'll pin-point the source of the error perhaps a bit better.
> 
> Can I just "accept" a pull request in my fork without running into trouble
> when I rebase?

Not sure if I reported back, but the PULL fixed the issue.

--WjW

Re: FreeBSD crushtool crashing while checking a crushmap

[kefu chai]

On Mon, Feb 8, 2016 at 7:01 PM, Willem Jan Withagen <wjw@digiware.nl> wrote:
> On 8-2-2016 06:53, kefu chai wrote:
>>
>> sorry, just sent an out-dated reply.
>>
>> On Sun, Feb 7, 2016 at 6:12 AM, Willem Jan Withagen <wjw@digiware.nl>
>> wrote:
>>>
>>> Hi,
>>>
>>> While running run-cli-tests one of the tests dumps while checking if the
>>> previous command has build the correct crushmap.
>>> I've verified the actual crushmap against one build on CentOS, and they
>>> are byte for byte equal.
>>>
>>> Before I rebased (from beginning of januari) all these tests completed
>>> just fine.
>>>
>>> During checking I get the following dump:
>>>
>>> # src/crushtool -i
>>> src/tes/cli/crushtool/check-overlapped-rules.crushmap.ref --check
>>> Assertion failed: (this->_map.find(inter_val) == this->_map.end()),
>>> function gap_insert, file
>>> /usr/local/include/boost/icl/interval_base_map.hpp, line 555.
>>> *** Caught signal (Abort trap) **
>>>   in thread 803e15000
>>>   ceph version Development (no_version)
>>>   1: 0x86f865 <_ZN4ceph9BackTraceC2Ei+0x35> at
>>> /usr/srcs/Ceph/work/ceph/src/crushtool
>>>   2: 0x86e5f9 <_ZL19handle_fatal_signali+0xa9> at
>>> /usr/srcs/Ceph/work/ceph/src/crushtool
>>>   3: 0x801811c7d <pthread_sigmask+0x50d> at /lib/libthr.so.3
>>>   4: 0x8018112b2 <pthread_getspecific+0xe22> at /lib/libthr.so.3
>>> 2016-02-06 22:55:09.075189 803e15000 -1
>>>
>>> Deleted remainder of the output....
>>>
>>> And I appreciate any pointers in helping me debugging this.
>>
>>
>> hey Willem, thanks for looking into this. turns out this is a known issue
>> of
>> boost 1.55. just posted a pull request to workaround it at
>> https://github.com/ceph/ceph/pull/7560.
>>
>> also did i create a minimal reproducer, it can help with reproducing this
>> issue and verify the fix.
>>
>> you can compile it using:
>>
>> $ clang++ -std=c++11 -O0 -g test.cc -lm -o test
>>
>
> pffh,
>
> That's a relieve, since I would otherwise be quite a chase....
> and yes, I'm still at boost 1.55.
>
> Perhaps in the test check-overlapped-rules.t an extra test:
>         cmp "$TESTDIR/check-overlapped-rules.crushmap"
> "$TESTDIR/check-overlapped-rules.crushmap.ref"

we have compile-decompile-recompile.t for testing this.

>
> Which will at least tell you that the output was consistent with what sould
> be there.
> And it'll pin-point the source of the error perhaps a bit better.
>
> Can I just "accept" a pull request in my fork without running into trouble
> when I rebase?

i don't think you can. rebase against the master is the way to go, i guess.

>
> --WjW
>



-- 
Regards
Kefu Chai

Re: FreeBSD crushtool crashing while checking a crushmap

[Willem Jan Withagen]

On 10-2-2016 13:13, kefu chai wrote:
> On Mon, Feb 8, 2016 at 7:01 PM, Willem Jan Withagen <wjw@digiware.nl> wrote:
>> On 8-2-2016 06:53, kefu chai wrote:
>>>
>>> sorry, just sent an out-dated reply.
>>>
>>> On Sun, Feb 7, 2016 at 6:12 AM, Willem Jan Withagen <wjw@digiware.nl>
>>> wrote:
>>>>
>>>> Hi,
>>>>
>>>> While running run-cli-tests one of the tests dumps while checking if the
>>>> previous command has build the correct crushmap.
>>>> I've verified the actual crushmap against one build on CentOS, and they
>>>> are byte for byte equal.
>>>>
>>>> Before I rebased (from beginning of januari) all these tests completed
>>>> just fine.
>>>>
>>>> During checking I get the following dump:
>>>>
>>>> # src/crushtool -i
>>>> src/tes/cli/crushtool/check-overlapped-rules.crushmap.ref --check
>>>> Assertion failed: (this->_map.find(inter_val) == this->_map.end()),
>>>> function gap_insert, file
>>>> /usr/local/include/boost/icl/interval_base_map.hpp, line 555.
>>>> *** Caught signal (Abort trap) **
>>>>    in thread 803e15000
>>>>    ceph version Development (no_version)
>>>>    1: 0x86f865 <_ZN4ceph9BackTraceC2Ei+0x35> at
>>>> /usr/srcs/Ceph/work/ceph/src/crushtool
>>>>    2: 0x86e5f9 <_ZL19handle_fatal_signali+0xa9> at
>>>> /usr/srcs/Ceph/work/ceph/src/crushtool
>>>>    3: 0x801811c7d <pthread_sigmask+0x50d> at /lib/libthr.so.3
>>>>    4: 0x8018112b2 <pthread_getspecific+0xe22> at /lib/libthr.so.3
>>>> 2016-02-06 22:55:09.075189 803e15000 -1
>>>>
>>>> Deleted remainder of the output....
>>>>
>>>> And I appreciate any pointers in helping me debugging this.
>>>
>>>
>>> hey Willem, thanks for looking into this. turns out this is a known issue
>>> of
>>> boost 1.55. just posted a pull request to workaround it at
>>> https://github.com/ceph/ceph/pull/7560.
>>>
>>> also did i create a minimal reproducer, it can help with reproducing this
>>> issue and verify the fix.
>>>
>>> you can compile it using:
>>>
>>> $ clang++ -std=c++11 -O0 -g test.cc -lm -o test
>>>
>>
>> pffh,
>>
>> That's a relieve, since I would otherwise be quite a chase....
>> and yes, I'm still at boost 1.55.
>>
>> Perhaps in the test check-overlapped-rules.t an extra test:
>>          cmp "$TESTDIR/check-overlapped-rules.crushmap"
>> "$TESTDIR/check-overlapped-rules.crushmap.ref"
>
> we have compile-decompile-recompile.t for testing this.

But then still you do not know where the error is:
	upon generation
	or on decompile
But then it requires debugging anyways, and you'll find out soon enough.

>> Which will at least tell you that the output was consistent with what sould
>> be there.
>> And it'll pin-point the source of the error perhaps a bit better.
>>
>> Can I just "accept" a pull request in my fork without running into trouble
>> when I rebase?
>
> i don't think you can. rebase against the master is the way to go, i guess.

Yup, rebase will be the way to get it in.

--WjW