ceph next branch: error creating empty object store in /srv/osd.10: (22) Invalid argument

ceph next branch: error creating empty object store in /srv/osd.10: (22) Invalid argument

[Stefan Priebe - Profihost AG]

Hi,

as i had some problems with the stable branch. Then i wanted to try the 
next branch to look if my problems are solved.

When i wanted to create the ceph FS with the next branch i get this error:

=== osd.10 ===
pushing conf and monmap to ssdstor000:/tmp/mkfs.ceph.3279
2012-06-14 13:54:21.001555 7fea16922780 -1 provided osd id 10 != 
superblock's -1
2012-06-14 13:54:21.001874 7fea16922780 -1  ** ERROR: error creating 
empty object store in /srv/osd.10: (22) Invalid argument
failed: 'ssh root@ssdstor000 /sbin/mkcephfs -d /tmp/mkfs.ceph.3279 
--init-daemon osd.10'

/dev/sdb1 on /srv/osd.10 type xfs 
(rw,noatime,nodiratime,nobarrier,discard,logbufs=8,logbsize=256k)

Stefan

Re: ceph next branch: error creating empty object store in /srv/osd.10: (22) Invalid argument

[Stefan Priebe - Profihost AG]

I got this fixed by deleting all content in /srv/osd.X. But now i get 
with the next branch this one:

=== osd.32 ===
pushing conf and monmap to ssdstor002:/tmp/mkfs.ceph.4885
2012-06-14 14:02:09.333309 7f815cd26780 -1 journal FileJournal::_open: 
unable to open journal: open() failed: (2) No such file or directory
2012-06-14 14:02:09.690097 7f815cd26780 -1 filestore(/srv/osd.32) could 
not find 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or 
directory
2012-06-14 14:02:09.758456 7f815cd26780 -1 created object store 
/srv/osd.32 journal /journal/osd.32.journal for osd.32 fsid 
308b2ada-5c2a-4b54-b094-b7035413d247
creating private key for osd.32 keyring /etc/ceph/osd.32.keyring
creating /etc/ceph/osd.32.keyring
collecting osd.32 key

Stefan

Re: ceph next branch: error creating empty object store in /srv/osd.10: (22) Invalid argument

[Sage Weil]

On Thu, 14 Jun 2012, Stefan Priebe - Profihost AG wrote:
> I got this fixed by deleting all content in /srv/osd.X. But now i get with the
> next branch this one:
> 
> === osd.32 ===
> pushing conf and monmap to ssdstor002:/tmp/mkfs.ceph.4885
> 2012-06-14 14:02:09.333309 7f815cd26780 -1 journal FileJournal::_open: unable
> to open journal: open() failed: (2) No such file or directory
> 2012-06-14 14:02:09.690097 7f815cd26780 -1 filestore(/srv/osd.32) could not
> find 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory
> 2012-06-14 14:02:09.758456 7f815cd26780 -1 created object store /srv/osd.32
> journal /journal/osd.32.journal for osd.32 fsid
> 308b2ada-5c2a-4b54-b094-b7035413d247
> creating private key for osd.32 keyring /etc/ceph/osd.32.keyring
> creating /etc/ceph/osd.32.keyring
> collecting osd.32 key

Is it possible that the osd journal you configured is a file in the osd 
data directory that doesn't already exist, and you didn't specify the 'osd 
journal size' option in ceph.conf?

The change here is that ceph-osd --mkfs no longer wipes out previous data, 
so that's now your job.  But it only creates the journal file when the 
size is specified, otherwise it needs to already be present...

sage

Re: ceph next branch: error creating empty object store in /srv/osd.10: (22) Invalid argument

[Stefan Priebe - Profihost AG]

Am 14.06.2012 22:17, schrieb Sage Weil:
> On Thu, 14 Jun 2012, Stefan Priebe - Profihost AG wrote:
> Is it possible that the osd journal you configured is a file in the osd
> data directory that doesn't already exist
Sure is this wrong?
[osd]
         osd journal = /journal/osd.$id.journal
         osd journal size = 1000

and the /journal dir is empty.

 >, and you didn't specify the 'osd
> journal size' option in ceph.conf?
no it is - see above


> The change here is that ceph-osd --mkfs no longer wipes out previous data,
> so that's now your job.
ah OK ;-) didn't know that.

 > But it only creates the journal file when the
> size is specified, otherwise it needs to already be present...
It wasn't present but the size was set. Strangely the error occours but 
it still creates the file... ?!

Stefan

Re: ceph next branch: error creating empty object store in /srv/osd.10: (22) Invalid argument

[Sage Weil]

On Fri, 15 Jun 2012, Stefan Priebe - Profihost AG wrote:
> Am 14.06.2012 22:17, schrieb Sage Weil:
> > On Thu, 14 Jun 2012, Stefan Priebe - Profihost AG wrote:
> > Is it possible that the osd journal you configured is a file in the osd
> > data directory that doesn't already exist
> Sure is this wrong?
> [osd]
>         osd journal = /journal/osd.$id.journal
>         osd journal size = 1000
> 
> and the /journal dir is empty.
> 
> >, and you didn't specify the 'osd
> > journal size' option in ceph.conf?
> no it is - see above

Hmm, ok, so it's not that then.

> > The change here is that ceph-osd --mkfs no longer wipes out previous data,
> > so that's now your job.
> ah OK ;-) didn't know that.
> 
> > But it only creates the journal file when the
> > size is specified, otherwise it needs to already be present...
> It wasn't present but the size was set. Strangely the error occours but it
> still creates the file... ?!

It sure is.

Can you reproduce with 'debug filestore = 20' and attach the log to this 
bug?

	http://tracker.newdream.net/issues/2595

Thanks!
sage

Re: ceph next branch: error creating empty object store in /srv/osd.10: (22) Invalid argument

[Stefan Priebe]

Am 15.06.2012 18:29, schrieb Sage Weil:
>>> But it only creates the journal file when the
>>> size is specified, otherwise it needs to already be present...
>> It wasn't present but the size was set. Strangely the error occours but it
>> still creates the file... ?!
>
> It sure is.
>
> Can you reproduce with 'debug filestore = 20' and attach the log to this
> bug?
>
> 	http://tracker.newdream.net/issues/2595

done and log attached

Stefan