Suicide

[Zenon Panoussis <oracle@provocation.net>]

Hello again.

A new day, a new problem:

ceph version 0.26.commit: 9981ff90968398da43c63106694d661f5e3d07d5. process: cmds. pid: 8613
2011-04-15 15:50:11.945105 7fd081c07700 mds-1.0 ms_handle_connect on 192.168.178.100:6789/0
2011-04-15 15:50:16.277166 7fd081c07700 mds-1.0 handle_mds_map standby
2011-04-15 15:50:17.662254 7fd081c07700 mds-1.0 handle_mds_map standby
2011-04-15 15:50:33.050509 7fd081c07700 mds0.41 handle_mds_map i am now mds0.41
2011-04-15 15:50:33.050614 7fd081c07700 mds0.41 handle_mds_map state change up:standby --> up:replay
2011-04-15 15:50:33.050812 7fd081c07700 mds0.41 replay_start
2011-04-15 15:50:33.050908 7fd081c07700 mds0.41  recovery set is
2011-04-15 15:50:33.050962 7fd081c07700 mds0.41  need osdmap epoch 184, have 178
2011-04-15 15:50:33.051143 7fd081c07700 mds0.cache handle_mds_failure mds0 : recovery peers are
2011-04-15 15:50:33.057263 7fd081c07700 mds0.41 ms_handle_connect on 192.168.178.100:6801/8729
2011-04-15 15:50:33.071290 7fd081c07700 mds0.41 ms_handle_connect on 192.168.178.101:6801/4572
2011-04-15 15:50:33.972268 7fd081c07700 mds0.cache creating system inode with ino:100
2011-04-15 15:50:33.972783 7fd081c07700 mds0.cache creating system inode with ino:1
2011-04-15 15:50:44.568883 7fd0735fc700 mds0.journaler try_read_entry got 0 len entry at offset 390071095
2011-04-15 15:50:44.569039 7fd0735fc700 mds0.log _replay journaler got error -22, aborting
2011-04-15 15:50:44.576781 7fd0735fc700 mds0.41 boot_start encountered an error, failing
2011-04-15 15:50:44.576871 7fd0735fc700 mds0.41 suicide.  wanted up:replay, now down:dne
2011-04-15 15:50:44.580226 7fd082c0a720 stopped.

Last night I mounted the ceph file system OK and started copying data to it.
Logging was at the sample.ceph.conf defaults, so pretty hefty. The logs go
to /var/log and the journal is in /var, which in this case are on different
partitions. At some point copying hung. It was really very late, so instead
of troubleshooting I switched off the machines and went to bed.

Today I tried to mount the ceph file system and failed, several times. I then
found that /var/log was full to the last byte on (only) one of my two nodes.
I cleaned up and restarted ceph, but trying to mount the file system continued
giving me "error 5" and the mds log entries above. It looks very much like issue
451 reproduced, but I don't grasp the logic of it. It would make more sense if
the journal had been on the full partition, but it wasn't.

cmds [-i node01|mds.node01|0] -c /etc/ceph/ceph.conf --journal_check -f
returns a usage message and no error, no matter if/how I specify the name
and/or -m IP:6789. What's the correct syntax for it?

The full logs (what's now left of them) are at
http://www.provocation.net/tmp/mds_logs.tar.bz2 . Is there anything I can do
to solve this, apart from re-creating the file system?

ceph 0.26, kernel 2.38.2, ext4, ceph.conf as posted yesterday in the "Mounting"
thread but without auth.

Z