mkcephfs 0.20 failure on debian: FAILED assert(r == 0)

mkcephfs 0.20 failure on debian: FAILED assert(r == 0)

[Thomas Mueller]

Hi

I wanted to do some testing. but i failed on mkcephfs.  see mkcephfs 
output below. 

this is on debian lenny, i follow the howto in the wiki http://
ceph.newdream.net/wiki/Debian and 
http://ceph.newdream.net/wiki/Installing_on_Debian

0.19 worked with the wiki-howto.

.. and what happend to the ceph-kclient-source package? 

- Thomas


node2:~# /usr/sbin/mkcephfs -c /etc/ceph/ceph.conf --mkbtrfs 
/usr/bin/monmaptool --create --clobber --add 192.168.236.211:6789 --add 
192.168.236.212:6789 --print /tmp/monmap.8485
/usr/bin/monmaptool: monmap file /tmp/
monmap.8485                                                                    
/usr/bin/monmaptool: generated fsid 9a3cf759-8e42-e69c-dc48-6926ac0e2599
epoch 1
fsid 9a3cf759-8e42-e69c-dc48-6926ac0e2599
last_changed 10.05.01 17:09:52.257140
created 10.05.01 17:09:52.257140
        mon0 192.168.236.211:6789/0
        mon1 192.168.236.212:6789/0
/usr/bin/monmaptool: writing epoch 1 to /tmp/monmap.8485 (2 monitors)
max osd in /etc/ceph/ceph.conf is 1, num osd is 2
/usr/bin/osdmaptool: osdmap file '/tmp/osdmap.8485'
/usr/bin/osdmaptool: writing epoch 1 to /tmp/osdmap.8485
Building admin keyring at /tmp/admin.keyring.8485
creating /tmp/admin.keyring.8485
Building monitor keyring with all service keys
creating /tmp/monkeyring.8485
importing contents of /tmp/admin.keyring.8485 into /tmp/monkeyring.8485
creating /tmp/keyring.mds.node1
importing contents of /tmp/keyring.mds.node1 into /tmp/monkeyring.8485
creating /tmp/keyring.mds.node2
importing contents of /tmp/keyring.mds.node2 into /tmp/monkeyring.8485
creating /tmp/keyring.osd.0
importing contents of /tmp/keyring.osd.0 into /tmp/monkeyring.8485
creating /tmp/keyring.osd.1
importing contents of /tmp/keyring.osd.1 into /tmp/monkeyring.8485
=== mon1 ===
10.05.01 17:09:52.773759 7fee853196f0 store(/data/mon1) mkfs
10.05.01 17:09:52.774204 7fee853196f0 store(/data/mon1) test -d /data/mon1 
&& /bin/rm -rf /data/mon1 ; mkdir -p /data/mon1
10.05.01 17:09:53.442052 7fee853196f0 mon1(starting).class v0 
create_initial -- creating initial map
10.05.01 17:09:53.641892 7fee853196f0 mon1(starting).auth v0 
create_initial -- creating initial map
10.05.01 17:09:53.641927 7fee853196f0 mon1(starting).auth v0 reading 
initial keyring
/usr/bin/mkmonfs: created monfs at /data/mon1 for mon1
`/tmp/admin.keyring.8485' -> `/data/mon1/admin_keyring.bin'
=== mds.node2 ===
`/tmp/keyring.mds.node2' -> `/data/keyring.mds.node2'
=== osd1 ===
umount: /data/osd1: not mounted
umount: /dev/vdb: not mounted

WARNING! - Btrfs Btrfs v0.19 IS EXPERIMENTAL
WARNING! - see http://btrfs.wiki.kernel.org before using

fs created label (null) on /dev/vdb
        nodesize 4096 leafsize 4096 sectorsize 4096 size 5.00GB
Btrfs Btrfs v0.19
Scanning for Btrfs filesystems
failed to read /dev/sr0
mount: /dev/vdb already mounted or /data/osd1 busy
 ** WARNING: Ceph is still under heavy development, and is only suitable 
for **
 **          testing and review.  Do not trust it with important 
data.       **
os/FileJournal.cc: In function 'int FileJournal::_open(bool, bool)':
os/FileJournal.cc:74: FAILED assert(r == 0)
 1: (FileJournal::create()+0x7f) [0x55725f]
 2: (FileStore::mkjournal()+0x74) [0x542f04]
 3: (FileStore::mkfs()+0x4c9) [0x540dd9]
 4: (OSD::mkfs(char const*, char const*, ceph_fsid, int)+0x4a) [0x4c2dca]
 5: (main()+0x751) [0x4585e1]
 6: (__libc_start_main()+0xe6) [0x7fc412c811a6]
 7: (std::ios_base::Init::~Init()+0x61) [0x457be9]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed 
to interpret this.
terminate called after throwing an instance of 'ceph::FailedAssertion*'
/usr/lib/ceph/ceph_common.sh: line 95:  8680 Aborted                 
(core dumped) bash -c "$1"
failed: '/usr/bin/cosd -c /etc/ceph/ceph.conf --monmap /tmp/monmap.8485 -
i 1 --mkfs --osd-data /data/osd1'

Re: mkcephfs 0.20 failure on debian: FAILED assert(r == 0)

[Sage Weil]

Hi Thomas,

On Sat, 1 May 2010, Thomas Mueller wrote:

> Hi
> 
> I wanted to do some testing. but i failed on mkcephfs.  see mkcephfs 
> output below. 
> 
> this is on debian lenny, i follow the howto in the wiki http://
> ceph.newdream.net/wiki/Debian and 
> http://ceph.newdream.net/wiki/Installing_on_Debian
> 
> 0.19 worked with the wiki-howto.

It sounds like the osd journal file exists, but is 0 bytes.  You can 
include

	osd journal size = 100    ; measured in MB

to your ceph.conf and it will be resized during mkfs (iff it is 0 bytes).  
Or use 'dd'.  Or (better yet) point it at a raw block device.

I'm fixing up the error handling to return an error instead of asserting.

> .. and what happend to the ceph-kclient-source package? 

Well, I'm not sure if we should keep building that package.  The kernel 
client versioning is somewhat divorced from the server side now that the 
client is in the mainline kernel (and releases are dictated by kernel 
releases).  But I take it people use that package?  I suppose as long as 
there is some 'stable' release that coincides with the final kernel 
release (e.g., v0.20.1 for v2.6.34 in a few weeks, in this case).

sage

Re: mkcephfs 0.20 failure on debian: FAILED assert(r == 0)

[Thomas Mueller]

hi sage

> It sounds like the osd journal file exists, but is 0 bytes.  You can
> include
> 
> 	osd journal size = 100    ; measured in MB
> 
> to your ceph.conf and it will be resized during mkfs (iff it is 0
> bytes). Or use 'dd'.  Or (better yet) point it at a raw block device.
> 

ok, this worked.



> I'm fixing up the error handling to return an error instead of
> asserting.
> 
>> .. and what happend to the ceph-kclient-source package?
> 
> Well, I'm not sure if we should keep building that package.  The kernel
> client versioning is somewhat divorced from the server side now that the
> client is in the mainline kernel (and releases are dictated by kernel
> releases).  But I take it people use that package?  I suppose as long as
> there is some 'stable' release that coincides with the final kernel
> release (e.g., v0.20.1 for v2.6.34 in a few weeks, in this case).

debian squeeze will be released with kernel 2.6.32 - so it won't have the 
upstream ceph module. 

the kvm project releases "kvm-kmod" source tarballs numberd like the 
kernel for use with older kernels. something like that would be nice. 

even more nice would be a dkms* package. maybe i get the time this week 
to try creating one. 

- Thomas

* http://linux.die.net/man/8/dkms

Re: mkcephfs 0.20 failure on debian: FAILED assert(r == 0)

[Jean-Adrien]

Thomas Mueller <thomas <at> chaschperli.ch> writes:

> 
> Hi
> 
> I wanted to do some testing. but i failed on mkcephfs.  see mkcephfs 
> output below. 
>  **          testing and review.  Do not trust it with important 
> data.       **
> os/FileJournal.cc: In function 'int FileJournal::_open(bool, bool)':
> os/FileJournal.cc:74: FAILED assert(r == 0)
>  1: (FileJournal::create()+0x7f) [0x55725f]
>  2: (FileStore::mkjournal()+0x74) [0x542f04]
>  3: (FileStore::mkfs()+0x4c9) [0x540dd9]
>  4: (OSD::mkfs(char const*, char const*, ceph_fsid, int)+0x4a) [0x4c2dca]
>  5: (main()+0x751) [0x4585e1]
>  6: (__libc_start_main()+0xe6) [0x7fc412c811a6]
>  7: (std::ios_base::Init::~Init()+0x61) [0x457be9]
>  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed 
> to interpret this.
> terminate called after throwing an instance of 'ceph::FailedAssertion*'
> /usr/lib/ceph/ceph_common.sh: line 95:  8680 Aborted                 
> (core dumped) bash -c "$1"
> failed: '/usr/bin/cosd -c /etc/ceph/ceph.conf --monmap /tmp/monmap.8485 -
> i 1 --mkfs --osd-data /data/osd1'


Hi,

I try to setup a little test cluster using kvm, and I have exactly the same
problem.
Moreover, I notice that when I want to delete the data created in the btrfs
partition it is not possible to remove the current directory, even if it is
empty:

root@cods-1:/mnt/btrfs/data# rm -rf osd0/
rm: cannot remove directory `osd0/current': Directory not empty

The only way is to reformat the partition (mkfs.btrfs /dev/sdb1)

But the next time I try to make the fs I have the same problem 
./mkcephfs -c ceph.conf -a -k /var/ceph-data/keyring.bin

I tried the --mkbtrfs option too, but I have the same result.
Did you made the same observations ? Any clue ?

Have a nice day

-- Jean-Adrien

Re: mkcephfs 0.20 failure on debian: FAILED assert(r == 0)

[Sage Weil]

On Wed, 12 May 2010, Jean-Adrien wrote:
> I try to setup a little test cluster using kvm, and I have exactly the same
> problem.

This is fixed in git, and there are 'testing' packages built that include 
the fix (a v0.20.1 release with these will be out shortly).  Just adjust 
your /etc/apt/source.list to use 'ceph-testing', like so:

 deb http://ceph.newdream.net/debian/ squeeze ceph-testing
 deb-src http://ceph.newdream.net/debian/ squeeze ceph-testing

Alternatively, adding

        osd journal size = 100

to your ceph.conf, or simply creating the journal yourself with 'dd' will 
avoid the issue.

> Moreover, I notice that when I want to delete the data created in the btrfs
> partition it is not possible to remove the current directory, even if it is
> empty:
> 
> root@cods-1:/mnt/btrfs/data# rm -rf osd0/
> rm: cannot remove directory `osd0/current': Directory not empty
>
> The only way is to reformat the partition (mkfs.btrfs /dev/sdb1)

This is a btrfs thing.. you can't remove subvolumes with rmdir.  'btrfsctl 
-D current' will do the trick.  Or, the latest cosd will clean them up for 
you.

sage