Could ceph-deploy handle unknown or custom distribution? (Was: Mourning the demise of mkcephfs)

[Mark Kirkwood <mark.kirkwood@catalyst.net.nz>]

[-- Attachment #1: Type: text/plain, Size: 1777 bytes --]

On 13/11/13 16:33, Mark Kirkwood wrote:
> I believe he is using a self built (or heavily customized) Linux
> installation - so distribution detection is not going to work in this
> case. I'm wondering if there could be some sensible fall back for that,
> e.g:
>
> - refuse to install or purge
> - assume sysv init
>

It was raining here on Saturday so I thought I'd take a look at if this 
was even feasible (see attached). These patches are a rough experiment 
to see what *might* be involved in getting to a point where you could 
get a system going on one of the distributions (or a custom build) that 
we currently do not support. So view as a thought-in-progress rather 
than even wip :-) It could also be quite the wrong approach - that is 
ok, just let me know what you think. Hopefully even if it is wrong, it 
may help stimulate discussion (or patches) for how to do it right!

The 1st patch defines an 'unknown' distribution for cases where one of 
our well known cases is *not* detected. With it applied I can create 
mons and it will start them directly with ceph-mon.

The 2nd and 3rd patch do similarly for osd, defining a type of startup 
called 'direct' to mark the osd instance with. I needed to patch 
ceph-disk as well so it could start the resulting osd directly with 
ceph-osd, and add it into the crushmap at a (hopefully) sensible place.

(BTW I did want to do this all in ceph-deploy, but right now it does not 
know how to get an osd's id number from a disk device - mind you this 
could be useful to add for things like 'list' and 'destroy'... but I 
digress).

Anyway I have attached a log of me getting a system of 2 Archlinux nodes 
up. These were KVM guests built identically and ceph (0.72) was compiled 
from src and installed.

Cheers

Mark




[-- Attachment #2: ceph-deploy-unknown-0.patch --]
[-- Type: text/x-patch, Size: 3563 bytes --]

commit 0629a803d7a1ecb651ed34c402c41f72bafaec7d
Author: Mark Kirkwood <mark.kirkwood@gmail.com>
Date:   Sun Nov 17 00:43:20 2013 +1300

    Initial prototype for handling unknown distribution types.

diff --git a/ceph_deploy/hosts/__init__.py b/ceph_deploy/hosts/__init__.py
index f4ba203..8f6a541 100644
--- a/ceph_deploy/hosts/__init__.py
+++ b/ceph_deploy/hosts/__init__.py
@@ -6,7 +6,7 @@ on the type of distribution/version we are dealing with.
 """
 import logging
 from ceph_deploy import exc, lsb
-from ceph_deploy.hosts import debian, centos, fedora, suse, remotes
+from ceph_deploy.hosts import debian, centos, fedora, suse, remotes, unknown
 from ceph_deploy.connection import get_connection
 
 logger = logging.getLogger()
@@ -62,6 +62,7 @@ def _get_distro(distro, fallback=None):
         'redhat': centos,
         'fedora': fedora,
         'suse': suse,
+        'unknown': unknown,
         }
     try:
         return distributions[distro]
@@ -79,4 +80,6 @@ def _normalized_distro_name(distro):
         return 'scientific'
     elif distro.startswith(('suse', 'opensuse')):
         return 'suse'
+    elif distro.startswith('unknown'):
+        return 'unknown'
     return distro
diff --git a/ceph_deploy/hosts/remotes.py b/ceph_deploy/hosts/remotes.py
index 62766f7..c759982 100644
--- a/ceph_deploy/hosts/remotes.py
+++ b/ceph_deploy/hosts/remotes.py
@@ -18,10 +18,21 @@ def platform_information():
         major_version = release.split('.')[0]
         codename = debian_codenames.get(major_version, '')
 
+    distro = str(distro).rstrip(),
+    distro = str(release).rstrip(),
+    distro = str(codename).rstrip()
+
+    if distro == '':
+        distro = 'unknown'
+    if release == '':
+        release = 'unknown'
+    if codename == '':
+        codename = 'unknown'
+
     return (
-        str(distro).rstrip(),
-        str(release).rstrip(),
-        str(codename).rstrip()
+        str(distro),
+        str(release),
+        str(codename)
     )
 
 
diff --git a/ceph_deploy/hosts/unknown/__init__.py b/ceph_deploy/hosts/unknown/__init__.py
new file mode 100644
index 0000000..d67ad7e
--- /dev/null
+++ b/ceph_deploy/hosts/unknown/__init__.py
@@ -0,0 +1,8 @@
+import mon
+
+# Allow to set some information about this distro
+#
+
+distro = None
+release = None
+codename = None
diff --git a/ceph_deploy/hosts/unknown/mon/__init__.py b/ceph_deploy/hosts/unknown/mon/__init__.py
new file mode 100644
index 0000000..fca0e0d
--- /dev/null
+++ b/ceph_deploy/hosts/unknown/mon/__init__.py
@@ -0,0 +1 @@
+from create import create
diff --git a/ceph_deploy/hosts/unknown/mon/create.py b/ceph_deploy/hosts/unknown/mon/create.py
new file mode 100644
index 0000000..ed2a3ba
--- /dev/null
+++ b/ceph_deploy/hosts/unknown/mon/create.py
@@ -0,0 +1,30 @@
+from ceph_deploy.hosts import common
+from ceph_deploy.lib.remoto import process
+
+
+def create(distro, args, monitor_keyring):
+    hostname = distro.conn.remote_module.shortname()
+    common.mon_create(distro, args, monitor_keyring, hostname)
+
+    # no idea about startup architecture so start directly, run + create keys
+    process.run(
+        distro.conn,
+        [
+            'ceph-mon',
+            '--cluster', args.cluster,
+            '-c', '/etc/ceph/{cluster}.conf'.format(cluster=args.cluster),
+            '-i', hostname,
+        ],
+        timeout=7,
+    )
+
+    process.run(
+        distro.conn,
+        [
+            'ceph-create-keys',
+            '--cluster', args.cluster,
+            '-i', hostname,
+        ],
+        timeout=7,
+    )
+

[-- Attachment #3: ceph-deploy-unknown-1.patch --]
[-- Type: text/x-patch, Size: 331 bytes --]

diff --git a/ceph_deploy/lsb.py b/ceph_deploy/lsb.py
index b7f3e73..2f608de 100644
--- a/ceph_deploy/lsb.py
+++ b/ceph_deploy/lsb.py
@@ -118,4 +118,6 @@ def choose_init(distro, codename):
     """
     if distro == 'Ubuntu':
         return 'upstart'
+    elif  distro == 'unknown':
+        return 'direct'
     return 'sysvinit'

[-- Attachment #4: ceph-disk-unknown-0.patch --]
[-- Type: text/x-patch, Size: 2321 bytes --]

*** ceph-disk.orig	Sun Nov 17 11:34:33 2013
--- ceph-disk	Mon Nov 18 13:21:20 2013
***************
*** 96,101 ****
--- 96,102 ----
      'upstart',
      'sysvinit',
      'systemd',
+     'direct',
      'auto',
      ]
  
***************
*** 1438,1443 ****
--- 1439,1470 ----
                      'osd.{osd_id}'.format(osd_id=osd_id),
                      ],
                  )
+         elif os.path.exists(os.path.join(path, 'direct')):
+             # no idea about which init, start directly and set crush location
+             subprocess.check_call(
+                 args=[
+                     '/usr/bin/ceph-osd',
+                     #'--cluster', args.cluster,
+                     #'-c', '/etc/ceph/{cluster}.conf'.format(cluster=args.cluster),
+                     '-i', '{osd_id}'.format(osd_id=osd_id),
+                     ],
+                 )
+ 
+             # assume weight 1 (wrong but easy to change)!
+             subprocess.check_call(
+                 args=[
+                     '/usr/bin/ceph',
+                     #'--cluster', args.cluster,
+                     #'-c', '/etc/ceph/{cluster}.conf'.format(cluster=args.cluster),
+                     'osd',
+                     'crush',
+                      'add',
+                     'osd.{osd_id}'.format(osd_id=osd_id),
+                     '1',
+                     'root=default',
+                     'host={hostname}'.format(hostname=platform.node()),
+                     ],
+                 )
          else:
              raise Error('{cluster} osd.{osd_id} is not tagged with an init system'.format(
                      cluster=cluster,
***************
*** 1683,1690 ****
--- 1710,1722 ----
                      (distro, release, codename) = platform.dist()
                      if distro == 'Ubuntu':
                          init = 'upstart'
+                     if distro == 'unknown' or distro == '':
+                         LOG.debug('distro is empty, assuming init is direct')
+                         init = 'direct'
                      else:
                          init = 'sysvinit'
+             if init == 'direct':
+                 LOG.debug('init is direct')
  
              LOG.debug('Marking with init system %s', init)
              with file(os.path.join(path, init), 'w'):

[-- Attachment #5: NOTES-unknown --]
[-- Type: text/plain, Size: 1650 bytes --]

After applying patch for unknown distro
=======================================

State:

- monitors created and start ok
- osd created ok
  * not started (startup by ceph-disk)
  * no crush setting (set by init scripts)

Test on Archlinux hosts (zor[2,3]), running ceph-deploy from Ubuntu workstation 
(localhost)


1/ Do src build installing binaries and udev:

(zor2) $ sudo make install
(zor2) $ sudo cp udev/* /lib/udev/rules.d/
(zor2) $ cd /var/lib/ceph;sudo mkdir bootstrap-mds bootstrap-osd mds mon osd tmp
(zor2) $ sudo mkdir /etc/ceph

[same for zor3]


2/ Deploy with (patched) ceph-deploy 

(localhost) $ ceph-deploy new zor2
(localhost) $ ceph-deploy mon create zor2
(localhost) $ ceph-deploy gatherkeys zor2
(localhost) $ ceph-deploy disk zap zor2:/dev/vdb
(localhost) $ ceph-deploy disk zap zor3:/dev/vdb
(localhost) $ ceph-deploy osd prepare zor2:/dev/vdb
(localhost) $ ceph-deploy osd prepare zor3:/dev/vdb
(localhost) $ ceph-deploy osd activate zor2:/dev/vdb1
(localhost) $ ceph-deploy osd activate zor3:/dev/vdb1


3/ Check state

(localhost) $ ceph -c ceph.conf -k ceph.client.admin.keyring -s
    cluster de5d8fac-58b1-411a-8047-dd46d0d91246
     health HEALTH_OK
     monmap e1: 1 mons at {zor2=192.168.122.12:6789/0}, election epoch 2, quorum 0 zor2
     osdmap e36: 2 osds: 2 up, 2 in
      pgmap v55: 192 pgs, 3 pools, 0 bytes data, 0 objects
            70852 kB used, 6052 MB / 6121 MB avail
                 192 active+clean

(localhost) $ ceph -c ceph.conf -k ceph.client.admin.keyring osd tree
# id	weight	type name	up/down	reweight
-1	2	root default
-2	1		host zor3
1	1			osd.1	up	1	
-3	1		host zor2
0	1			osd.0	up	1