ceph mkfs failed

ceph mkfs failed

[sheng qiu]

Hi,

i am trying to port a local file system other than ext3,ext4,btrfs to
manage the OSD. There are some reasons for us to do it.
the mkcephfs stopped at this point:

2013-02-07 11:39:32.009889 7effe43d17a0 -1 filestore(/mnt/osd.0) mkfs
failed to create leveldb: IO error:
/mnt/osd.0/current/omap/MANIFEST-000001: Invalid argument
2013-02-07 11:39:32.009907 7effe43d17a0 -1 OSD::mkfs: FileStore::mkfs
failed with error -1
2013-02-07 11:39:32.009949 7effe43d17a0 -1  ** ERROR: error creating
empty object store in /mnt/osd.0: (1) Operation not permitted
failed: 'ssh root@mon-mds /usr/local/sbin/mkcephfs -d
/tmp/mkfs.ceph.2686 --init-daemon osd.0'

look into the code, it failed at os/FileStore.cc -->  leveldb::Status
status = leveldb::DB::Open(options, omap_dir, &db);

Does anyone have any suggestions?

Thanks,
Sheng

-- 
Sheng Qiu
Texas A & M University
Room 332B Wisenbaker
email: herbert1984106@gmail.com
College Station, TX 77843-3259

Re: ceph mkfs failed

[Dan Mick]

On 02/07/2013 09:59 AM, sheng qiu wrote:
> Hi,
>
> i am trying to port a local file system other than ext3,ext4,btrfs to
> manage the OSD. There are some reasons for us to do it.
> the mkcephfs stopped at this point:
>
> 2013-02-07 11:39:32.009889 7effe43d17a0 -1 filestore(/mnt/osd.0) mkfs
> failed to create leveldb: IO error:
> /mnt/osd.0/current/omap/MANIFEST-000001: Invalid argument
> 2013-02-07 11:39:32.009907 7effe43d17a0 -1 OSD::mkfs: FileStore::mkfs
> failed with error -1
> 2013-02-07 11:39:32.009949 7effe43d17a0 -1  ** ERROR: error creating
> empty object store in /mnt/osd.0: (1) Operation not permitted

Does your ssh user have permission to create dirs in /mnt?
(I'll grant you it looks like you're doing root, but, still)

> failed: 'ssh root@mon-mds /usr/local/sbin/mkcephfs -d
> /tmp/mkfs.ceph.2686 --init-daemon osd.0'

You could certainly try this command on the failing system as root,
and look with strace to see what system call is actually failing.

>
> look into the code, it failed at os/FileStore.cc -->  leveldb::Status
> status = leveldb::DB::Open(options, omap_dir, &db);
>
> Does anyone have any suggestions?
>
> Thanks,
> Sheng
>

Re: ceph mkfs failed

[sheng qiu]

Hi Dan,

thanks for your reply.

after some code tracking, i found it failed at this point :
in file leveldb/db/db_impl.cc  --> NewDB()

log::Writer log(file);
    std::string record;
    new_db.EncodeTo(&record);
    s = log.AddRecord(record);
    if (s.ok()) {
      fprintf(test, "NewDB: 2\n");
      s = file->Close();
    }else
      fprintf(test, "NewDB: 2.5\n");

the log.AddRecord return s which is not ok().

can you provide some hint why it fails?  i am reading the AddRecord()
function now.

Thanks,
Sheng


On Thu, Feb 7, 2013 at 2:22 PM, Dan Mick <dan.mick@inktank.com> wrote:
>
>
> On 02/07/2013 09:59 AM, sheng qiu wrote:
>>
>> Hi,
>>
>> i am trying to port a local file system other than ext3,ext4,btrfs to
>> manage the OSD. There are some reasons for us to do it.
>> the mkcephfs stopped at this point:
>>
>> 2013-02-07 11:39:32.009889 7effe43d17a0 -1 filestore(/mnt/osd.0) mkfs
>> failed to create leveldb: IO error:
>> /mnt/osd.0/current/omap/MANIFEST-000001: Invalid argument
>> 2013-02-07 11:39:32.009907 7effe43d17a0 -1 OSD::mkfs: FileStore::mkfs
>> failed with error -1
>> 2013-02-07 11:39:32.009949 7effe43d17a0 -1  ** ERROR: error creating
>> empty object store in /mnt/osd.0: (1) Operation not permitted
>
>
> Does your ssh user have permission to create dirs in /mnt?
> (I'll grant you it looks like you're doing root, but, still)
>
>
>> failed: 'ssh root@mon-mds /usr/local/sbin/mkcephfs -d
>> /tmp/mkfs.ceph.2686 --init-daemon osd.0'
>
>
> You could certainly try this command on the failing system as root,
> and look with strace to see what system call is actually failing.
>
>
>>
>> look into the code, it failed at os/FileStore.cc -->  leveldb::Status
>> status = leveldb::DB::Open(options, omap_dir, &db);
>>
>> Does anyone have any suggestions?
>>
>> Thanks,
>> Sheng
>>
>



-- 
Sheng Qiu
Texas A & M University
Room 332B Wisenbaker
email: herbert1984106@gmail.com
College Station, TX 77843-3259

Re: ceph mkfs failed

[Gregory Farnum]

On Thu, Feb 7, 2013 at 12:42 PM, sheng qiu <herbert1984106@gmail.com> wrote:
> Hi Dan,
>
> thanks for your reply.
>
> after some code tracking, i found it failed at this point :
> in file leveldb/db/db_impl.cc  --> NewDB()
>
> log::Writer log(file);
>     std::string record;
>     new_db.EncodeTo(&record);
>     s = log.AddRecord(record);
>     if (s.ok()) {
>       fprintf(test, "NewDB: 2\n");
>       s = file->Close();
>     }else
>       fprintf(test, "NewDB: 2.5\n");
>
> the log.AddRecord return s which is not ok().
>
> can you provide some hint why it fails?  i am reading the AddRecord()
> function now.

LevelDB is a generic library which we don't develop. My understanding
is that it's expected to work on any POSIX-compliant filesystem, but
you can check out the docs
(http://leveldb.googlecode.com/svn/trunk/doc/index.html) or source
code for more info.
-Greg

Re: ceph mkfs failed

[sheng qiu]

Is it possible comment the leveldb setup codes during mkfs call path?
i find the problem is because i am trying to use a bunch of memory as
the OSD, when the leveldb mmap() the file on OSD, it failed, as the
file actually located on memory.

Thanks,
Sheng

On Thu, Feb 7, 2013 at 2:53 PM, Gregory Farnum <greg@inktank.com> wrote:
> On Thu, Feb 7, 2013 at 12:42 PM, sheng qiu <herbert1984106@gmail.com> wrote:
>> Hi Dan,
>>
>> thanks for your reply.
>>
>> after some code tracking, i found it failed at this point :
>> in file leveldb/db/db_impl.cc  --> NewDB()
>>
>> log::Writer log(file);
>>     std::string record;
>>     new_db.EncodeTo(&record);
>>     s = log.AddRecord(record);
>>     if (s.ok()) {
>>       fprintf(test, "NewDB: 2\n");
>>       s = file->Close();
>>     }else
>>       fprintf(test, "NewDB: 2.5\n");
>>
>> the log.AddRecord return s which is not ok().
>>
>> can you provide some hint why it fails?  i am reading the AddRecord()
>> function now.
>
> LevelDB is a generic library which we don't develop. My understanding
> is that it's expected to work on any POSIX-compliant filesystem, but
> you can check out the docs
> (http://leveldb.googlecode.com/svn/trunk/doc/index.html) or source
> code for more info.
> -Greg



-- 
Sheng Qiu
Texas A & M University
Room 332B Wisenbaker
email: herbert1984106@gmail.com
College Station, TX 77843-3259

Re: ceph mkfs failed

[Dan Mick]

leveldb is required.  What filesystem are you using that doesn't support 
mmap?

On 02/07/2013 02:11 PM, sheng qiu wrote:
> Is it possible comment the leveldb setup codes during mkfs call path?
> i find the problem is because i am trying to use a bunch of memory as
> the OSD, when the leveldb mmap() the file on OSD, it failed, as the
> file actually located on memory.
>
> Thanks,
> Sheng
>
> On Thu, Feb 7, 2013 at 2:53 PM, Gregory Farnum <greg@inktank.com> wrote:
>> On Thu, Feb 7, 2013 at 12:42 PM, sheng qiu <herbert1984106@gmail.com> wrote:
>>> Hi Dan,
>>>
>>> thanks for your reply.
>>>
>>> after some code tracking, i found it failed at this point :
>>> in file leveldb/db/db_impl.cc  --> NewDB()
>>>
>>> log::Writer log(file);
>>>      std::string record;
>>>      new_db.EncodeTo(&record);
>>>      s = log.AddRecord(record);
>>>      if (s.ok()) {
>>>        fprintf(test, "NewDB: 2\n");
>>>        s = file->Close();
>>>      }else
>>>        fprintf(test, "NewDB: 2.5\n");
>>>
>>> the log.AddRecord return s which is not ok().
>>>
>>> can you provide some hint why it fails?  i am reading the AddRecord()
>>> function now.
>>
>> LevelDB is a generic library which we don't develop. My understanding
>> is that it's expected to work on any POSIX-compliant filesystem, but
>> you can check out the docs
>> (http://leveldb.googlecode.com/svn/trunk/doc/index.html) or source
>> code for more info.
>> -Greg
>
>
>

Re: ceph mkfs failed

[sheng qiu]

i see. it only support readonly mmap. i will figure it out.

Thanks a lot.

Sheng

On Thu, Feb 7, 2013 at 4:21 PM, Dan Mick <dan.mick@inktank.com> wrote:
> leveldb is required.  What filesystem are you using that doesn't support
> mmap?
>
>
> On 02/07/2013 02:11 PM, sheng qiu wrote:
>>
>> Is it possible comment the leveldb setup codes during mkfs call path?
>> i find the problem is because i am trying to use a bunch of memory as
>> the OSD, when the leveldb mmap() the file on OSD, it failed, as the
>> file actually located on memory.
>>
>> Thanks,
>> Sheng
>>
>> On Thu, Feb 7, 2013 at 2:53 PM, Gregory Farnum <greg@inktank.com> wrote:
>>>
>>> On Thu, Feb 7, 2013 at 12:42 PM, sheng qiu <herbert1984106@gmail.com>
>>> wrote:
>>>>
>>>> Hi Dan,
>>>>
>>>> thanks for your reply.
>>>>
>>>> after some code tracking, i found it failed at this point :
>>>> in file leveldb/db/db_impl.cc  --> NewDB()
>>>>
>>>> log::Writer log(file);
>>>>      std::string record;
>>>>      new_db.EncodeTo(&record);
>>>>      s = log.AddRecord(record);
>>>>      if (s.ok()) {
>>>>        fprintf(test, "NewDB: 2\n");
>>>>        s = file->Close();
>>>>      }else
>>>>        fprintf(test, "NewDB: 2.5\n");
>>>>
>>>> the log.AddRecord return s which is not ok().
>>>>
>>>> can you provide some hint why it fails?  i am reading the AddRecord()
>>>> function now.
>>>
>>>
>>> LevelDB is a generic library which we don't develop. My understanding
>>> is that it's expected to work on any POSIX-compliant filesystem, but
>>> you can check out the docs
>>> (http://leveldb.googlecode.com/svn/trunk/doc/index.html) or source
>>> code for more info.
>>> -Greg
>>
>>
>>
>>
>



-- 
Sheng Qiu
Texas A & M University
Room 332B Wisenbaker
email: herbert1984106@gmail.com
College Station, TX 77843-3259

Re: ceph mkfs failed

[sheng qiu]

Hi,

i want to update a little bit on my problem.
after some work, right now my local fs on OSD can works with ceph (at
least pass through the init-ceph script). i made a minor change on
ceph code, i changed the link_object() in LFNIndex.cc, basically i
changed the hard link call ::link() to symlink(), as my local fs does
not support hard link (the directory entry stores together with the
related inodes).

when i mount the client on ceph, it is fine. and the ceph -s shows at
least 50% degraded, although in HEALTH_WARN state, some pgs still
work.
i can manually create files and write them on the mount path (by vim
and edit them).
when i try to run some benchmarks, such as iozone and fio. it fails
with the following information:

2013-02-08 10:16:44.433532 7f692c55a700  0 -- 165.91.215.237:6801/7231
>> 165.91.215.237:0/1395168951 pipe(0x2b9c240 sd=57 :6801 pgs=0 cs=0
l=0).accept peer addr is really 165.91.215.237:0/1395168951 (socket is
165.91.215.237:60388/0)
2013-02-08 10:20:22.488334 7f692c55a700  0 -- 165.91.215.237:6801/7231
>> 165.91.215.237:0/1395168951 pipe(0x2b9c000 sd=57 :6801 pgs=0 cs=0
l=0).accept peer addr is really 165.91.215.237:0/1395168951 (socket is
165.91.215.237:60413/0)
2013-02-08 10:20:22.564453 7f6937873700 -1 *** Caught signal
(Segmentation fault) **
 in thread 7f6937873700

is there any suggestions?

Thanks,
Sheng

On Thu, Feb 7, 2013 at 4:40 PM, sheng qiu <herbert1984106@gmail.com> wrote:
> i see. it only support readonly mmap. i will figure it out.
>
> Thanks a lot.
>
> Sheng
>
> On Thu, Feb 7, 2013 at 4:21 PM, Dan Mick <dan.mick@inktank.com> wrote:
>> leveldb is required.  What filesystem are you using that doesn't support
>> mmap?
>>
>>
>> On 02/07/2013 02:11 PM, sheng qiu wrote:
>>>
>>> Is it possible comment the leveldb setup codes during mkfs call path?
>>> i find the problem is because i am trying to use a bunch of memory as
>>> the OSD, when the leveldb mmap() the file on OSD, it failed, as the
>>> file actually located on memory.
>>>
>>> Thanks,
>>> Sheng
>>>
>>> On Thu, Feb 7, 2013 at 2:53 PM, Gregory Farnum <greg@inktank.com> wrote:
>>>>
>>>> On Thu, Feb 7, 2013 at 12:42 PM, sheng qiu <herbert1984106@gmail.com>
>>>> wrote:
>>>>>
>>>>> Hi Dan,
>>>>>
>>>>> thanks for your reply.
>>>>>
>>>>> after some code tracking, i found it failed at this point :
>>>>> in file leveldb/db/db_impl.cc  --> NewDB()
>>>>>
>>>>> log::Writer log(file);
>>>>>      std::string record;
>>>>>      new_db.EncodeTo(&record);
>>>>>      s = log.AddRecord(record);
>>>>>      if (s.ok()) {
>>>>>        fprintf(test, "NewDB: 2\n");
>>>>>        s = file->Close();
>>>>>      }else
>>>>>        fprintf(test, "NewDB: 2.5\n");
>>>>>
>>>>> the log.AddRecord return s which is not ok().
>>>>>
>>>>> can you provide some hint why it fails?  i am reading the AddRecord()
>>>>> function now.
>>>>
>>>>
>>>> LevelDB is a generic library which we don't develop. My understanding
>>>> is that it's expected to work on any POSIX-compliant filesystem, but
>>>> you can check out the docs
>>>> (http://leveldb.googlecode.com/svn/trunk/doc/index.html) or source
>>>> code for more info.
>>>> -Greg
>>>
>>>
>>>
>>>
>>
>
>
>
> --
> Sheng Qiu
> Texas A & M University
> Room 332B Wisenbaker
> email: herbert1984106@gmail.com
> College Station, TX 77843-3259



-- 
Sheng Qiu
Texas A & M University
Room 332B Wisenbaker
email: herbert1984106@gmail.com
College Station, TX 77843-3259

Re: ceph mkfs failed

[Sage Weil]

Hi Sheng-

On Fri, 8 Feb 2013, sheng qiu wrote:
> least pass through the init-ceph script). i made a minor change on
> ceph code, i changed the link_object() in LFNIndex.cc, basically i
> changed the hard link call ::link() to symlink(), as my local fs does
> not support hard link (the directory entry stores together with the
> related inodes).

Unrelated question: which local fs are you using?

sage

Re: ceph mkfs failed

[sheng qiu]

Hi Sage,

it's a memory based fs similar to pramfs.

Thanks,
Sheng

On Fri, Feb 8, 2013 at 11:02 AM, Sage Weil <sage@inktank.com> wrote:
> Hi Sheng-
>
> On Fri, 8 Feb 2013, sheng qiu wrote:
>> least pass through the init-ceph script). i made a minor change on
>> ceph code, i changed the link_object() in LFNIndex.cc, basically i
>> changed the hard link call ::link() to symlink(), as my local fs does
>> not support hard link (the directory entry stores together with the
>> related inodes).
>
> Unrelated question: which local fs are you using?
>
> sage



-- 
Sheng Qiu
Texas A & M University
Room 332B Wisenbaker
email: herbert1984106@gmail.com
College Station, TX 77843-3259

Re: ceph mkfs failed

[sheng qiu]

Hi,

i think it's not related with my local FS. i build a ext4 on a ramdisk
and used it as OSD.
when i run the iozone or fio on the mounted client point, it  shows
the same info as before:

2013-02-08 11:45:06.803915 7f28ec7c4700  0 -- 165.91.215.237:6801/7101
>> 165.91.215.237:0/1990103183 pipe(0x2ded240 sd=803 :6801 pgs=0 cs=0
l=0).accept peer addr is really 165.91.215.237:0/1990103183 (socket is
165.91.215.237:60553/0)
2013-02-08 11:45:06.879009 7f28f7add700 -1 *** Caught signal
(Segmentation fault) **
 in thread 7f28f7add700

the ceph -s shows, also the same as using my own local FS:

  health HEALTH_WARN 384 pgs degraded; 384 pgs stuck unclean; recovery
21/42 degraded (50.000%)
   monmap e1: 1 mons at {0=165.91.215.237:6789/0}, election epoch 2, quorum 0 0
   osdmap e3: 1 osds: 1 up, 1 in
    pgmap v7: 384 pgs: 384 active+degraded; 21003 bytes data, 276 MB
used, 3484 MB / 3961 MB avail; 21/42 degraded (50.000%)
   mdsmap e4: 1/1/1 up {0=0=up:active}

dmesg shows:

[  656.799209] libceph: client4099 fsid da0fe76d-8506-4bf8-8b49-172fd8bc6d1f
[  656.800657] libceph: mon0 165.91.215.237:6789 session established
[  683.789954] libceph: osd0 165.91.215.237:6801 socket closed (con state OPEN)
[  683.790007] libceph: osd0 165.91.215.237:6801 socket error on write
[  684.909095] libceph: osd0 165.91.215.237:6801 socket error on write
[  685.903425] libceph: osd0 165.91.215.237:6801 socket error on write
[  687.903937] libceph: osd0 165.91.215.237:6801 socket error on write
[  691.897037] libceph: osd0 165.91.215.237:6801 socket error on write
[  699.899197] libceph: osd0 165.91.215.237:6801 socket error on write
[  715.903415] libceph: osd0 165.91.215.237:6801 socket error on write
[  747.912122] libceph: osd0 165.91.215.237:6801 socket error on write
[  811.929323] libceph: osd0 165.91.215.237:6801 socket error on write

Thanks,
Sheng


On Fri, Feb 8, 2013 at 11:07 AM, sheng qiu <herbert1984106@gmail.com> wrote:
> Hi Sage,
>
> it's a memory based fs similar to pramfs.
>
> Thanks,
> Sheng
>
> On Fri, Feb 8, 2013 at 11:02 AM, Sage Weil <sage@inktank.com> wrote:
>> Hi Sheng-
>>
>> On Fri, 8 Feb 2013, sheng qiu wrote:
>>> least pass through the init-ceph script). i made a minor change on
>>> ceph code, i changed the link_object() in LFNIndex.cc, basically i
>>> changed the hard link call ::link() to symlink(), as my local fs does
>>> not support hard link (the directory entry stores together with the
>>> related inodes).
>>
>> Unrelated question: which local fs are you using?
>>
>> sage
>
>
>
> --
> Sheng Qiu
> Texas A & M University
> Room 332B Wisenbaker
> email: herbert1984106@gmail.com
> College Station, TX 77843-3259



-- 
Sheng Qiu
Texas A & M University
Room 332B Wisenbaker
email: herbert1984106@gmail.com
College Station, TX 77843-3259

Re: ceph mkfs failed

[sheng qiu]

ok, this is tested using ext3/ext4 on a normal SSD as OSD.

ceph -s shows:
health HEALTH_WARN 384 pgs degraded; 384 pgs stuck unclean; recovery
22/44 degraded (50.000%)
   monmap e1: 1 mons at {0=165.91.215.237:6789/0}, election epoch 2, quorum 0 0
   osdmap e3: 1 osds: 1 up, 1 in
    pgmap v10: 384 pgs: 384 active+degraded; 26716 bytes data, 1184 MB
used, 55857 MB / 60093 MB avail; 22/44 degraded (50.000%)
   mdsmap e4: 1/1/1 up {0=0=up:active}

dmesg shows:
[  212.758376] libceph: client4106 fsid f60af615-67cb-4245-91cb-22752821f3e6
[  212.759869] libceph: mon0 165.91.215.237:6789 session established
[  338.292461] libceph: osd0 165.91.215.237:6801 socket closed (con state OPEN)
[  338.292483] libceph: osd0 165.91.215.237:6801 socket error on write
[  339.161231] libceph: osd0 165.91.215.237:6801 socket error on write
[  340.159003] libceph: osd0 165.91.215.237:6801 socket error on write
[  342.158514] libceph: osd0 165.91.215.237:6801 socket error on write
[  346.149549] libceph: osd0 165.91.215.237:6801 socket error on write

osd.0.log shows:
2013-02-08 14:52:51.649726 7f82780f6700  0 -- 165.91.215.237:6801/7135
>> 165.91.215.237:0/3238315774 pipe(0x2d61240 sd=803 :6801 pgs=0 cs=0
l=0).accept peer addr is really 165.91.215.237:0/3238315774 (socket is
165.91.215.237:57270/0)
2013-02-08 14:53:26.103770 7f8283c10700 -1 *** Caught signal
(Segmentation fault) **
 in thread 7f8283c10700

 ceph version 0.56.1 (e4a541624df62ef353e754391cbbb707f54b16f7)
 1: ./ceph-osd() [0x78648a]
 2: (()+0x10060) [0x7f828cb0e060]
 3: (fwrite()+0x34) [0x7f828aea3ec4]
 4: (leveldb::log::Writer::EmitPhysicalRecord(leveldb::log::RecordType,
char const*, unsigned long)+0x11f) [0x76d93f]
 5: (leveldb::log::Writer::AddRecord(leveldb::Slice const&)+0x74) [0x76dae4]
 6: (leveldb::DBImpl::Write(leveldb::WriteOptions const&,
leveldb::WriteBatch*)+0x160) [0x763050]
 7: (LevelDBStore::submit_transaction(std::tr1::shared_ptr<KeyValueDB::TransactionImpl>)+0x2a)
[0x74ec1a]
 8: (DBObjectMap::remove_xattrs(hobject_t const&,
std::set<std::string, std::less<std::string>,
std::allocator<std::string> > const&, SequencerPosition const*)+0x16a)
[0x746fca]
 9: (FileStore::_setattrs(coll_t, hobject_t const&,
std::map<std::string, ceph::buffer::ptr, std::less<std::string>,
std::allocator<std::pair<std::string const, ceph::buffer::ptr> > >&,
SequencerPosition const&)+0xe7f) [0x719aff]
 10: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned
long, int)+0x3cba) [0x71e7da]
 11: (FileStore::do_transactions(std::list<ObjectStore::Transaction*,
std::allocator<ObjectStore::Transaction*> >&, unsigned long)+0x4c)
[0x72152c]
 12: (FileStore::_do_op(FileStore::OpSequencer*)+0x1b1) [0x6f1331]
 13: (ThreadPool::worker(ThreadPool::WorkThread*)+0x4bc) [0x827dec]
 14: (ThreadPool::WorkThread::entry()+0x10) [0x829cb0]
 15: (()+0x7efc) [0x7f828cb05efc]
 16: (clone()+0x6d) [0x7f828af1cf8d]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.


any suggestions?

Thanks,
Sheng

On Fri, Feb 8, 2013 at 11:53 AM, sheng qiu <herbert1984106@gmail.com> wrote:
> Hi,
>
> i think it's not related with my local FS. i build a ext4 on a ramdisk
> and used it as OSD.
> when i run the iozone or fio on the mounted client point, it  shows
> the same info as before:
>
> 2013-02-08 11:45:06.803915 7f28ec7c4700  0 -- 165.91.215.237:6801/7101
>>> 165.91.215.237:0/1990103183 pipe(0x2ded240 sd=803 :6801 pgs=0 cs=0
> l=0).accept peer addr is really 165.91.215.237:0/1990103183 (socket is
> 165.91.215.237:60553/0)
> 2013-02-08 11:45:06.879009 7f28f7add700 -1 *** Caught signal
> (Segmentation fault) **
>  in thread 7f28f7add700
>
> the ceph -s shows, also the same as using my own local FS:
>
>   health HEALTH_WARN 384 pgs degraded; 384 pgs stuck unclean; recovery
> 21/42 degraded (50.000%)
>    monmap e1: 1 mons at {0=165.91.215.237:6789/0}, election epoch 2, quorum 0 0
>    osdmap e3: 1 osds: 1 up, 1 in
>     pgmap v7: 384 pgs: 384 active+degraded; 21003 bytes data, 276 MB
> used, 3484 MB / 3961 MB avail; 21/42 degraded (50.000%)
>    mdsmap e4: 1/1/1 up {0=0=up:active}
>
> dmesg shows:
>
> [  656.799209] libceph: client4099 fsid da0fe76d-8506-4bf8-8b49-172fd8bc6d1f
> [  656.800657] libceph: mon0 165.91.215.237:6789 session established
> [  683.789954] libceph: osd0 165.91.215.237:6801 socket closed (con state OPEN)
> [  683.790007] libceph: osd0 165.91.215.237:6801 socket error on write
> [  684.909095] libceph: osd0 165.91.215.237:6801 socket error on write
> [  685.903425] libceph: osd0 165.91.215.237:6801 socket error on write
> [  687.903937] libceph: osd0 165.91.215.237:6801 socket error on write
> [  691.897037] libceph: osd0 165.91.215.237:6801 socket error on write
> [  699.899197] libceph: osd0 165.91.215.237:6801 socket error on write
> [  715.903415] libceph: osd0 165.91.215.237:6801 socket error on write
> [  747.912122] libceph: osd0 165.91.215.237:6801 socket error on write
> [  811.929323] libceph: osd0 165.91.215.237:6801 socket error on write
>
> Thanks,
> Sheng
>
>
> On Fri, Feb 8, 2013 at 11:07 AM, sheng qiu <herbert1984106@gmail.com> wrote:
>> Hi Sage,
>>
>> it's a memory based fs similar to pramfs.
>>
>> Thanks,
>> Sheng
>>
>> On Fri, Feb 8, 2013 at 11:02 AM, Sage Weil <sage@inktank.com> wrote:
>>> Hi Sheng-
>>>
>>> On Fri, 8 Feb 2013, sheng qiu wrote:
>>>> least pass through the init-ceph script). i made a minor change on
>>>> ceph code, i changed the link_object() in LFNIndex.cc, basically i
>>>> changed the hard link call ::link() to symlink(), as my local fs does
>>>> not support hard link (the directory entry stores together with the
>>>> related inodes).
>>>
>>> Unrelated question: which local fs are you using?
>>>
>>> sage
>>
>>
>>
>> --
>> Sheng Qiu
>> Texas A & M University
>> Room 332B Wisenbaker
>> email: herbert1984106@gmail.com
>> College Station, TX 77843-3259
>
>
>
> --
> Sheng Qiu
> Texas A & M University
> Room 332B Wisenbaker
> email: herbert1984106@gmail.com
> College Station, TX 77843-3259



-- 
Sheng Qiu
Texas A & M University
Room 332B Wisenbaker
email: herbert1984106@gmail.com
College Station, TX 77843-3259

Re: ceph mkfs failed

[sheng qiu]

ok, i have figured out it.....

Thanks,
Sheng

On Fri, Feb 8, 2013 at 2:57 PM, sheng qiu <herbert1984106@gmail.com> wrote:
> ok, this is tested using ext3/ext4 on a normal SSD as OSD.
>
> ceph -s shows:
> health HEALTH_WARN 384 pgs degraded; 384 pgs stuck unclean; recovery
> 22/44 degraded (50.000%)
>    monmap e1: 1 mons at {0=165.91.215.237:6789/0}, election epoch 2, quorum 0 0
>    osdmap e3: 1 osds: 1 up, 1 in
>     pgmap v10: 384 pgs: 384 active+degraded; 26716 bytes data, 1184 MB
> used, 55857 MB / 60093 MB avail; 22/44 degraded (50.000%)
>    mdsmap e4: 1/1/1 up {0=0=up:active}
>
> dmesg shows:
> [  212.758376] libceph: client4106 fsid f60af615-67cb-4245-91cb-22752821f3e6
> [  212.759869] libceph: mon0 165.91.215.237:6789 session established
> [  338.292461] libceph: osd0 165.91.215.237:6801 socket closed (con state OPEN)
> [  338.292483] libceph: osd0 165.91.215.237:6801 socket error on write
> [  339.161231] libceph: osd0 165.91.215.237:6801 socket error on write
> [  340.159003] libceph: osd0 165.91.215.237:6801 socket error on write
> [  342.158514] libceph: osd0 165.91.215.237:6801 socket error on write
> [  346.149549] libceph: osd0 165.91.215.237:6801 socket error on write
>
> osd.0.log shows:
> 2013-02-08 14:52:51.649726 7f82780f6700  0 -- 165.91.215.237:6801/7135
>>> 165.91.215.237:0/3238315774 pipe(0x2d61240 sd=803 :6801 pgs=0 cs=0
> l=0).accept peer addr is really 165.91.215.237:0/3238315774 (socket is
> 165.91.215.237:57270/0)
> 2013-02-08 14:53:26.103770 7f8283c10700 -1 *** Caught signal
> (Segmentation fault) **
>  in thread 7f8283c10700
>
>  ceph version 0.56.1 (e4a541624df62ef353e754391cbbb707f54b16f7)
>  1: ./ceph-osd() [0x78648a]
>  2: (()+0x10060) [0x7f828cb0e060]
>  3: (fwrite()+0x34) [0x7f828aea3ec4]
>  4: (leveldb::log::Writer::EmitPhysicalRecord(leveldb::log::RecordType,
> char const*, unsigned long)+0x11f) [0x76d93f]
>  5: (leveldb::log::Writer::AddRecord(leveldb::Slice const&)+0x74) [0x76dae4]
>  6: (leveldb::DBImpl::Write(leveldb::WriteOptions const&,
> leveldb::WriteBatch*)+0x160) [0x763050]
>  7: (LevelDBStore::submit_transaction(std::tr1::shared_ptr<KeyValueDB::TransactionImpl>)+0x2a)
> [0x74ec1a]
>  8: (DBObjectMap::remove_xattrs(hobject_t const&,
> std::set<std::string, std::less<std::string>,
> std::allocator<std::string> > const&, SequencerPosition const*)+0x16a)
> [0x746fca]
>  9: (FileStore::_setattrs(coll_t, hobject_t const&,
> std::map<std::string, ceph::buffer::ptr, std::less<std::string>,
> std::allocator<std::pair<std::string const, ceph::buffer::ptr> > >&,
> SequencerPosition const&)+0xe7f) [0x719aff]
>  10: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned
> long, int)+0x3cba) [0x71e7da]
>  11: (FileStore::do_transactions(std::list<ObjectStore::Transaction*,
> std::allocator<ObjectStore::Transaction*> >&, unsigned long)+0x4c)
> [0x72152c]
>  12: (FileStore::_do_op(FileStore::OpSequencer*)+0x1b1) [0x6f1331]
>  13: (ThreadPool::worker(ThreadPool::WorkThread*)+0x4bc) [0x827dec]
>  14: (ThreadPool::WorkThread::entry()+0x10) [0x829cb0]
>  15: (()+0x7efc) [0x7f828cb05efc]
>  16: (clone()+0x6d) [0x7f828af1cf8d]
>  NOTE: a copy of the executable, or `objdump -rdS <executable>` is
> needed to interpret this.
>
>
> any suggestions?
>
> Thanks,
> Sheng
>
> On Fri, Feb 8, 2013 at 11:53 AM, sheng qiu <herbert1984106@gmail.com> wrote:
>> Hi,
>>
>> i think it's not related with my local FS. i build a ext4 on a ramdisk
>> and used it as OSD.
>> when i run the iozone or fio on the mounted client point, it  shows
>> the same info as before:
>>
>> 2013-02-08 11:45:06.803915 7f28ec7c4700  0 -- 165.91.215.237:6801/7101
>>>> 165.91.215.237:0/1990103183 pipe(0x2ded240 sd=803 :6801 pgs=0 cs=0
>> l=0).accept peer addr is really 165.91.215.237:0/1990103183 (socket is
>> 165.91.215.237:60553/0)
>> 2013-02-08 11:45:06.879009 7f28f7add700 -1 *** Caught signal
>> (Segmentation fault) **
>>  in thread 7f28f7add700
>>
>> the ceph -s shows, also the same as using my own local FS:
>>
>>   health HEALTH_WARN 384 pgs degraded; 384 pgs stuck unclean; recovery
>> 21/42 degraded (50.000%)
>>    monmap e1: 1 mons at {0=165.91.215.237:6789/0}, election epoch 2, quorum 0 0
>>    osdmap e3: 1 osds: 1 up, 1 in
>>     pgmap v7: 384 pgs: 384 active+degraded; 21003 bytes data, 276 MB
>> used, 3484 MB / 3961 MB avail; 21/42 degraded (50.000%)
>>    mdsmap e4: 1/1/1 up {0=0=up:active}
>>
>> dmesg shows:
>>
>> [  656.799209] libceph: client4099 fsid da0fe76d-8506-4bf8-8b49-172fd8bc6d1f
>> [  656.800657] libceph: mon0 165.91.215.237:6789 session established
>> [  683.789954] libceph: osd0 165.91.215.237:6801 socket closed (con state OPEN)
>> [  683.790007] libceph: osd0 165.91.215.237:6801 socket error on write
>> [  684.909095] libceph: osd0 165.91.215.237:6801 socket error on write
>> [  685.903425] libceph: osd0 165.91.215.237:6801 socket error on write
>> [  687.903937] libceph: osd0 165.91.215.237:6801 socket error on write
>> [  691.897037] libceph: osd0 165.91.215.237:6801 socket error on write
>> [  699.899197] libceph: osd0 165.91.215.237:6801 socket error on write
>> [  715.903415] libceph: osd0 165.91.215.237:6801 socket error on write
>> [  747.912122] libceph: osd0 165.91.215.237:6801 socket error on write
>> [  811.929323] libceph: osd0 165.91.215.237:6801 socket error on write
>>
>> Thanks,
>> Sheng
>>
>>
>> On Fri, Feb 8, 2013 at 11:07 AM, sheng qiu <herbert1984106@gmail.com> wrote:
>>> Hi Sage,
>>>
>>> it's a memory based fs similar to pramfs.
>>>
>>> Thanks,
>>> Sheng
>>>
>>> On Fri, Feb 8, 2013 at 11:02 AM, Sage Weil <sage@inktank.com> wrote:
>>>> Hi Sheng-
>>>>
>>>> On Fri, 8 Feb 2013, sheng qiu wrote:
>>>>> least pass through the init-ceph script). i made a minor change on
>>>>> ceph code, i changed the link_object() in LFNIndex.cc, basically i
>>>>> changed the hard link call ::link() to symlink(), as my local fs does
>>>>> not support hard link (the directory entry stores together with the
>>>>> related inodes).
>>>>
>>>> Unrelated question: which local fs are you using?
>>>>
>>>> sage
>>>
>>>
>>>
>>> --
>>> Sheng Qiu
>>> Texas A & M University
>>> Room 332B Wisenbaker
>>> email: herbert1984106@gmail.com
>>> College Station, TX 77843-3259
>>
>>
>>
>> --
>> Sheng Qiu
>> Texas A & M University
>> Room 332B Wisenbaker
>> email: herbert1984106@gmail.com
>> College Station, TX 77843-3259
>
>
>
> --
> Sheng Qiu
> Texas A & M University
> Room 332B Wisenbaker
> email: herbert1984106@gmail.com
> College Station, TX 77843-3259



-- 
Sheng Qiu
Texas A & M University
Room 332B Wisenbaker
email: herbert1984106@gmail.com
College Station, TX 77843-3259

Re: ceph mkfs failed

[Gregory Farnum]

On Fri, Feb 8, 2013 at 1:18 PM, sheng qiu <herbert1984106@gmail.com> wrote:
> ok, i have figured out it.....

That looks like a LevelDB issue given the backtrace (and the OSD isn't
responding because it crashed). If you figured out why LevelDB
crashed, it'd be good to know so that other people can reference this
if they see something similar. :)
-Greg


> On Fri, Feb 8, 2013 at 2:57 PM, sheng qiu <herbert1984106@gmail.com> wrote:
>> ok, this is tested using ext3/ext4 on a normal SSD as OSD.
>>
>> ceph -s shows:
>> health HEALTH_WARN 384 pgs degraded; 384 pgs stuck unclean; recovery
>> 22/44 degraded (50.000%)
>>    monmap e1: 1 mons at {0=165.91.215.237:6789/0}, election epoch 2, quorum 0 0
>>    osdmap e3: 1 osds: 1 up, 1 in
>>     pgmap v10: 384 pgs: 384 active+degraded; 26716 bytes data, 1184 MB
>> used, 55857 MB / 60093 MB avail; 22/44 degraded (50.000%)
>>    mdsmap e4: 1/1/1 up {0=0=up:active}
>>
>> dmesg shows:
>> [  212.758376] libceph: client4106 fsid f60af615-67cb-4245-91cb-22752821f3e6
>> [  212.759869] libceph: mon0 165.91.215.237:6789 session established
>> [  338.292461] libceph: osd0 165.91.215.237:6801 socket closed (con state OPEN)
>> [  338.292483] libceph: osd0 165.91.215.237:6801 socket error on write
>> [  339.161231] libceph: osd0 165.91.215.237:6801 socket error on write
>> [  340.159003] libceph: osd0 165.91.215.237:6801 socket error on write
>> [  342.158514] libceph: osd0 165.91.215.237:6801 socket error on write
>> [  346.149549] libceph: osd0 165.91.215.237:6801 socket error on write
>>
>> osd.0.log shows:
>> 2013-02-08 14:52:51.649726 7f82780f6700  0 -- 165.91.215.237:6801/7135
>>>> 165.91.215.237:0/3238315774 pipe(0x2d61240 sd=803 :6801 pgs=0 cs=0
>> l=0).accept peer addr is really 165.91.215.237:0/3238315774 (socket is
>> 165.91.215.237:57270/0)
>> 2013-02-08 14:53:26.103770 7f8283c10700 -1 *** Caught signal
>> (Segmentation fault) **
>>  in thread 7f8283c10700
>>
>>  ceph version 0.56.1 (e4a541624df62ef353e754391cbbb707f54b16f7)
>>  1: ./ceph-osd() [0x78648a]
>>  2: (()+0x10060) [0x7f828cb0e060]
>>  3: (fwrite()+0x34) [0x7f828aea3ec4]
>>  4: (leveldb::log::Writer::EmitPhysicalRecord(leveldb::log::RecordType,
>> char const*, unsigned long)+0x11f) [0x76d93f]
>>  5: (leveldb::log::Writer::AddRecord(leveldb::Slice const&)+0x74) [0x76dae4]
>>  6: (leveldb::DBImpl::Write(leveldb::WriteOptions const&,
>> leveldb::WriteBatch*)+0x160) [0x763050]
>>  7: (LevelDBStore::submit_transaction(std::tr1::shared_ptr<KeyValueDB::TransactionImpl>)+0x2a)
>> [0x74ec1a]
>>  8: (DBObjectMap::remove_xattrs(hobject_t const&,
>> std::set<std::string, std::less<std::string>,
>> std::allocator<std::string> > const&, SequencerPosition const*)+0x16a)
>> [0x746fca]
>>  9: (FileStore::_setattrs(coll_t, hobject_t const&,
>> std::map<std::string, ceph::buffer::ptr, std::less<std::string>,
>> std::allocator<std::pair<std::string const, ceph::buffer::ptr> > >&,
>> SequencerPosition const&)+0xe7f) [0x719aff]
>>  10: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned
>> long, int)+0x3cba) [0x71e7da]
>>  11: (FileStore::do_transactions(std::list<ObjectStore::Transaction*,
>> std::allocator<ObjectStore::Transaction*> >&, unsigned long)+0x4c)
>> [0x72152c]
>>  12: (FileStore::_do_op(FileStore::OpSequencer*)+0x1b1) [0x6f1331]
>>  13: (ThreadPool::worker(ThreadPool::WorkThread*)+0x4bc) [0x827dec]
>>  14: (ThreadPool::WorkThread::entry()+0x10) [0x829cb0]
>>  15: (()+0x7efc) [0x7f828cb05efc]
>>  16: (clone()+0x6d) [0x7f828af1cf8d]
>>  NOTE: a copy of the executable, or `objdump -rdS <executable>` is
>> needed to interpret this.
>>
>>
>> any suggestions?
>>
>> Thanks,
>> Sheng
>>
>> On Fri, Feb 8, 2013 at 11:53 AM, sheng qiu <herbert1984106@gmail.com> wrote:
>>> Hi,
>>>
>>> i think it's not related with my local FS. i build a ext4 on a ramdisk
>>> and used it as OSD.
>>> when i run the iozone or fio on the mounted client point, it  shows
>>> the same info as before:
>>>
>>> 2013-02-08 11:45:06.803915 7f28ec7c4700  0 -- 165.91.215.237:6801/7101
>>>>> 165.91.215.237:0/1990103183 pipe(0x2ded240 sd=803 :6801 pgs=0 cs=0
>>> l=0).accept peer addr is really 165.91.215.237:0/1990103183 (socket is
>>> 165.91.215.237:60553/0)
>>> 2013-02-08 11:45:06.879009 7f28f7add700 -1 *** Caught signal
>>> (Segmentation fault) **
>>>  in thread 7f28f7add700
>>>
>>> the ceph -s shows, also the same as using my own local FS:
>>>
>>>   health HEALTH_WARN 384 pgs degraded; 384 pgs stuck unclean; recovery
>>> 21/42 degraded (50.000%)
>>>    monmap e1: 1 mons at {0=165.91.215.237:6789/0}, election epoch 2, quorum 0 0
>>>    osdmap e3: 1 osds: 1 up, 1 in
>>>     pgmap v7: 384 pgs: 384 active+degraded; 21003 bytes data, 276 MB
>>> used, 3484 MB / 3961 MB avail; 21/42 degraded (50.000%)
>>>    mdsmap e4: 1/1/1 up {0=0=up:active}
>>>
>>> dmesg shows:
>>>
>>> [  656.799209] libceph: client4099 fsid da0fe76d-8506-4bf8-8b49-172fd8bc6d1f
>>> [  656.800657] libceph: mon0 165.91.215.237:6789 session established
>>> [  683.789954] libceph: osd0 165.91.215.237:6801 socket closed (con state OPEN)
>>> [  683.790007] libceph: osd0 165.91.215.237:6801 socket error on write
>>> [  684.909095] libceph: osd0 165.91.215.237:6801 socket error on write
>>> [  685.903425] libceph: osd0 165.91.215.237:6801 socket error on write
>>> [  687.903937] libceph: osd0 165.91.215.237:6801 socket error on write
>>> [  691.897037] libceph: osd0 165.91.215.237:6801 socket error on write
>>> [  699.899197] libceph: osd0 165.91.215.237:6801 socket error on write
>>> [  715.903415] libceph: osd0 165.91.215.237:6801 socket error on write
>>> [  747.912122] libceph: osd0 165.91.215.237:6801 socket error on write
>>> [  811.929323] libceph: osd0 165.91.215.237:6801 socket error on write
>>>
>>> Thanks,
>>> Sheng
>>>
>>>
>>> On Fri, Feb 8, 2013 at 11:07 AM, sheng qiu <herbert1984106@gmail.com> wrote:
>>>> Hi Sage,
>>>>
>>>> it's a memory based fs similar to pramfs.
>>>>
>>>> Thanks,
>>>> Sheng
>>>>
>>>> On Fri, Feb 8, 2013 at 11:02 AM, Sage Weil <sage@inktank.com> wrote:
>>>>> Hi Sheng-
>>>>>
>>>>> On Fri, 8 Feb 2013, sheng qiu wrote:
>>>>>> least pass through the init-ceph script). i made a minor change on
>>>>>> ceph code, i changed the link_object() in LFNIndex.cc, basically i
>>>>>> changed the hard link call ::link() to symlink(), as my local fs does
>>>>>> not support hard link (the directory entry stores together with the
>>>>>> related inodes).
>>>>>
>>>>> Unrelated question: which local fs are you using?
>>>>>
>>>>> sage
>>>>
>>>>
>>>>
>>>> --
>>>> Sheng Qiu
>>>> Texas A & M University
>>>> Room 332B Wisenbaker
>>>> email: herbert1984106@gmail.com
>>>> College Station, TX 77843-3259
>>>
>>>
>>>
>>> --
>>> Sheng Qiu
>>> Texas A & M University
>>> Room 332B Wisenbaker
>>> email: herbert1984106@gmail.com
>>> College Station, TX 77843-3259
>>
>>
>>
>> --
>> Sheng Qiu
>> Texas A & M University
>> Room 332B Wisenbaker
>> email: herbert1984106@gmail.com
>> College Station, TX 77843-3259
>
>
>
> --
> Sheng Qiu
> Texas A & M University
> Room 332B Wisenbaker
> email: herbert1984106@gmail.com
> College Station, TX 77843-3259