OSD hotplugging & Chef cookbook ("chef-1")

OSD hotplugging & Chef cookbook ("chef-1")

[Tommi Virtanen]

Hi. I've been working on easy deployability and manageability of Ceph.
This work is intended to be a complete replacement for mkcephfs, and
integrate new product features instead of just automating the
previous, clumsier, administration mechanisms. I'm using Chef to
create and expand the cluster, but most of the new functionality is in
making the OSDs more dynamic.

The current work is in a branch called "ceph-1", and will be improved
upon, but it is now at a stage where others should start looking at
it.

Here's a quick intro to what's there right now. Apologies for the
formatting, I need to be on a plane fairly soon.. Rest assured, any
command that looks clumsy is that way mostly because I haven't had
time to make it prettier. I'll go through this with our QA and tech
writer once the dust settles, to clean up the instructions.


Limitations (all to be removed later):
- supports only 1 monitor
- journal is a file inside osd data directory
- only supports 1 cluster (name hardcoded to "ceph"); later you will
be able to run multiple clusters on the same hardware
- no rgw, mds, or anything else but a RADOS/rbd cluster tested yet
- no integration with e.g. OpenStack yet


Open questions:
- I removed the sysv-style init script (from the debian packaging).
I'm not sure what to do with that. Older debs will still need it?
- details of what goes where in e.g. the chef environment will change;
input is welcome



How to try it out:

I need to leave to make to the airport in time, but the latest change
is still compiling :(
Wait till http://gitbuilder.ceph.com/ceph-deb-oneiric-x86_64-basic/ref/chef-1/sha1
says 4b75bccd52104d0ecd551e0656a30791b25fe032, hope for the best, and
proceed:


# create 3 vms running ubuntu 11.10 server; mine ended up being named
chef02, inst03, inst04
# they need to be able to talk to each other, so do not use KVM's
"user" networking, but NAT or bridged. (NAT is default for libvirt.)

# make sure your vm has a unique hostname first, or it'll get
confusing later; edit /etc/hostname, /etc/hosts, run "sudo hostname
newname", re-login

# source for this:
http://wiki.opscode.com/display/chef/Installing+Chef+Server+on+Debian+or+Ubuntu+using+Packages

# figure out the IP address of your chef server vm
gpg --keyserver keys.gnupg.net --recv-keys 83EF826A
gpg --export packages@opscode.com | sudo apt-key add -
sudo tee /etc/apt/sources.list.d/chef.list <<EOF
deb http://apt.opscode.com/ oneiric-0.10 main
deb-src http://apt.opscode.com/ oneiric-0.10 main
EOF
sudo apt-get update
sudo apt-get install chef
# answer using the IP address of your chef server vm
				┌───────────────────────┤ Configuring chef ├───────────────────────┐
				│  This is the full URI that clients will use to connect to the    │
				│  server.                                                         │
				│  .                                                               │
				│  This will be used in /etc/chef/client.rb as 'chef_server_url'.  │
				│                                                                  │
				│ URL of Chef Server (e.g., http://chef.example.com:4000):         │
				│                                                                  │
				│ http://192.168.122.168:4000/____________________________________ │
				│                                                                  │
				│                              <Ok>                                │
				│                                                                  │
				└──────────────────────────────────────────────────────────────────┘

sudo apt-get install chef-server
# you MUST enter some password here or the installation will fail; no
human will need to type this ever again
			┌────────────────────────────┤ Configuring chef-solr
├─────────────────────────────┐
			│  Set the password for the chef user in the AMQP server queue. Use
               │
			│  RabbitMQ's rabbitmqctl program to set this password. The default
user           │
			│  and vhost are assumed (chef and /chef, respectively).
               │
			│  .
               │
			│  RabbitMQ does not have the capability to read the password from
a file, and     │
			│  this will be passed via "" on the command-line. As such, do not
use shell       │
			│  meta-characters that could cause errors such as !.
               │
			│  .
               │
			│  This will be used in /etc/chef/solr.rb and /etc/chef/server.rb
as 'amqp_pass'.  │
			│
               │
			│ New password for the 'chef' AMQP user in the RabbitMQ vhost
"/chef":             │
			│
               │
			│ aipo6EY2i_______________________________________________________________________
│
			│
               │
			│                                      <Ok>
               │
			│
               │
			└──────────────────────────────────────────────────────────────────────────────────┘

# if you use the chef web ui, you'll be typing this in a lot.. "foo"
is nice for demo-level security
			 ┌───────────────────────┤ Configuring chef-server-webui
├───────────────────────┐
			 │  This sets a temporary first-use password to log into the Chef
Server WebUI   │
			 │  as the 'admin' user for the first time. Once logged in, the
password should  │
			 │  be changed immediately.
             │
			 │  .
             │
			 │  Once the chef-server-webui process is running, login using the
username      │
			 │  'admin' using the password set here.
             │
			 │  .
             │
			 │  If a password is not entered, the webui default password for
'admin' will    │
			 │  be used, which is displayed on the webui home page. The
password must be     │
			 │  at least 6 characters or the webui will not start properly.
             │
			 │  .
             │
			 │  This will be used in /etc/chef/webui.rb as
'web_ui_admin_default_password'.  │
			 │  .
             │
			 │
             │
			 │ New password for the 'admin' user in the Chef Server WebUI:
             │
			 │
             │
			 │ foo__________________________________________________________________________
│
			 │
             │
			 │                                    <Ok>
             │
			 │
             │
			 └───────────────────────────────────────────────────────────────────────────────┘

# configure knife on the chef server
ssh ubuntu@chef02
mkdir -p ~/.chef
sudo cp /etc/chef/validation.pem /etc/chef/webui.pem ~/.chef
sudo chown -R $USER ~/.chef
knife configure -i
WARNING: No knife configuration file found
Where should I put the config file? [~/.chef/knife.rb]
Please enter the chef server URL: [http://ubuntu:4000] http://localhost:4000
Please enter a clientname for the new client: [ubuntu] chef-admin
Please enter the existing admin clientname: [chef-webui]
Please enter the location of the existing admin client's private key:
[/etc/chef/webui.pem] .chef/webui.pem
Please enter the validation clientname: [chef-validator]
Please enter the location of the validation key:
[/etc/chef/validation.pem] .chef/validation.pem
Please enter the path to a chef repository (or leave blank):
Creating initial API user...
Created client[chef-admin]
Configuration file written to /home/ubuntu/.chef/knife.rb

## OPTIONAL: if you want to use chef directly from your desktop,
configure that now, following the instructions on the webpage given at
the beginning
# knife-block is a nice utility for connecting to multiple
chef-servers, you might want that
# http://os.greenandsecure.co.uk/2012/03/using-multiple-chef-servers-with-knife/
# configure knife on your desktop machine; this writes to ~/.gem
gem1.9.1 install --no-rdoc --no-ri --user-install chef
gem1.9.1 install --no-rdoc --no-ri --user-install knife-block
ssh ubuntu@chef02 knife client create my-username -n -a -f /tmp/my-username.pem
Created client[tv-test2]

mkdir ~/.chef
# grab the validation key
cd ~/.chef
scp ubuntu@chef02:/tmp/tv-test2.pem ~/.chef/my-username.pem
knife block new

This will create a new knife configuration file for you to use with knife-block
Please provide a friendly name for the new configuration file: test2
Please enter the url to your Chef Server: http://192.168.122.168:4000/
Please enter an existing username or clientname for the API:
[your-login] my-username
Please enter the validation clientname: [chef-validator]
Please enter the location of the validation key:
[/etc/chef/validation.pem] .chef/validation-test2.pem
Please enter the path to a chef repository (or leave blank):
*****

You must place your client key in:
  /home/your-login/.chef/my-username.pem
Before running commands with Knife!

*****

You must place your validation key in:
  /etc/chef/validation.pem
Before generating instance data with Knife!

*****
Configuration file written to /home/your-login/.chef/knife-test2.rb
/home/your-login/.chef/knife-test2.rb has been sucessfully created
The available chef servers are:
  ...
You are asking to change from test to test2. Are you sure? (Y/N) y
The knife configuration has been updated to use test2
## END OPTIONAL

# now configure the actual ceph cluster vms

# give them a data disk or two, in addition to the disk containing the
root filesystem
# (hint: for experiments ONLY, uncheck "Allocate entire disk now",
choose "Device type: Virtio disk", "Cache mode: writeback" for best
performance)

# again, ensure unique hostnames

# source for this:
http://wiki.opscode.com/display/chef/Installing+Chef+Client+on+Ubuntu+or+Debian

# same stuff as beginning of the chef-server installation, repeat for every vm
gpg --keyserver keys.gnupg.net --recv-keys 83EF826A
gpg --export packages@opscode.com | sudo apt-key add -
sudo tee /etc/apt/sources.list.d/chef.list <<EOF
deb http://apt.opscode.com/ oneiric-0.10 main
deb-src http://apt.opscode.com/ oneiric-0.10 main
EOF
sudo apt-get update
sudo apt-get install chef
# answer using the IP address of your chef server vm
				┌───────────────────────┤ Configuring chef ├───────────────────────┐
				│  This is the full URI that clients will use to connect to the    │
				│  server.                                                         │
				│  .                                                               │
				│  This will be used in /etc/chef/client.rb as 'chef_server_url'.  │
				│                                                                  │
				│ URL of Chef Server (e.g., http://chef.example.com:4000):         │
				│                                                                  │
				│ http://192.168.122.168:4000/____________________________________ │
				│                                                                  │
				│                              <Ok>                                │
				│                                                                  │
				└──────────────────────────────────────────────────────────────────┘

# copy the validation key over, run on your desktop
ssh chef02 sudo cat /etc/chef/validation.pem | ssh inst03 sudo tee
/etc/chef/validation.pem >/dev/null
# chef should now work on the new node! you can trigger a run manually:
ssh inst03 sudo chef-client

# if you set up your workstation as a knife client, you can do the
following there; if not, you have to do this logged in to the chef
server

# doublecheck
knife node list
  chef02
  inst03
  inst04
# clone the cookbook
git clone https://github.com/ceph/ceph-cookbooks.git
cd ceph-cookbooks
# edit ~/.chef/knife-test2.rb (or knife.rb if you don't use knife-block) and add
#   cookbook_path
["/home/something/path/to/your/clone/ceph-cookbooks"]

# currently the rgw cookbook forces us to have the apache2 cookbook
available; do that
# source: http://wiki.opscode.com/display/chef/Managing+Cookbooks+With+Knife
knife cookbook site download apache2
tar xzf apache2-1.1.8.tar.gz
knife cookbook upload apache2
# TODO cleanup the btrfs and parted cookbook remains
knife cookbook upload btrfs ceph
# let's create some roles (the cookbooks currently hardcode the
"ceph-mon" role name, you must use that name for now)
cat >ceph-mon.rb <<EOF
name "ceph-mon"
description "Ceph monitor server"
run_list(
	 'recipe[ceph::single_mon]'
)
EOF
cat >ceph-osd.rb <<EOF
name "ceph-osd"
description "Ceph object store"
run_list(
	 'recipe[ceph::bootstrap_osd]',
)
EOF
knife role from file ceph-mon.rb ceph-osd.rb

# create a chef environment to contain the configuration, set the ceph
branch to deploy (must be "chef-1" for now)
knife environment create demo2
#  "default_attributes": {
#    "ceph_branch": "chef-1"
#  },
knife node edit inst03
#  "chef_environment": "demo2",
#  "run_list": [
#    "recipe[ceph::apt]",
#    "role[ceph-mon]",
#    "role[ceph-osd]"
#  ]
knife node edit inst04
#  "chef_environment": "demo2",
#  "run_list": [
#    "recipe[ceph::apt]",
#    "role[ceph-osd]"
#  ]

# run chef-client a few rounds (first to get mon going, then to get
the osd bootstrap files in place)
ssh inst03 sudo chef-client
ssh inst03 sudo chef-client
ssh inst04 sudo chef-client

# figure out what your ceph cluster uuid is; sorry, this is an eyeball
operation right now
ssh inst03 cat /var/lib/ceph/mon/ceph-single/log

# prep a disk for osd use; this can be done anywhere, in advance, and
a shelf of spare disks can be maintained at the data center
# run on inst03, inst04
# create a gpt partition with the magic uuid; repeat for /dev/vdb vdc etc
sudo apt-get install gdisk
sudo gdisk /dev/vdb
GPT fdisk (gdisk) version 0.6.14

Partition table scan:
  MBR: not present
  BSD: not present
  APM: not present
  GPT: not present

Creating new GPT entries.

Command (? for help): o
This option deletes all partitions and creates a new protective MBR.
Proceed? (Y/N): y

Command (? for help): n
Partition number (1-128, default 1):
First sector (34-16777182, default = 34) or {+-}size{KMGTP}:
Information: Moved requested sector from 34 to 2048 in
order to align on 2048-sector boundaries.
Use 'l' on the experts' menu to adjust alignment
Last sector (2048-16777182, default = 16777182) or {+-}size{KMGTP}:
Current type is 'Linux/Windows data'
Hex code or GUID (L to show codes, Enter = 0700):
4fbd7e29-9d25-41b8-afd0-062c0ceff05d
Changed type of partition to 'Unknown'

Command (? for help): w

Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
PARTITIONS!!

Do you want to proceed, possibly destroying your data? (Y/N): y
OK; writing new GUID partition table (GPT).
he operation has completed successfully.

# mkfs and allocate disk to cluster; any filesystem is ok; use YOUR
uuid from mon
sudo mkfs -t ext4 /dev/vdb1
sudo mount -o user_xattr /dev/vdb1 /mnt
sudo ceph-disk-prepare --cluster-uuid=07535b46-8429-4efd-9b2d-6b1170752448 /mnt
sudo umount /mnt
# simulate hotplug event
sudo udevadm trigger --subsystem-match=block --action=add

# see it go!
sudo initctl list | grep ceph
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: OSD hotplugging & Chef cookbook ("chef-1")

[Tommi Virtanen]

On Mon, May 7, 2012 at 3:58 PM, Tommi Virtanen <tv@inktank.com> wrote:
> The current work is in a branch called "ceph-1", and will be improved
> upon, but it is now at a stage where others should start looking at
> it.

Make that "chef-1".

Re: OSD hotplugging & Chef cookbook ("chef-1")

[Guilhem LETTRON]

There is already many work on a ceph cookbook :
https://github.com/wonko/ceph-cookbook (very complete, I must test and
complete it)
and mine https://github.com/guilhem/ceph-cookbooks (not as good as I
want, not updated but works for me with a complete ceph-RGW)

We must "sing from the same hymn sheet". This thread can be a good begin.

Guilhem Lettron
Ingénieur - Architecte système
System Architect
Youscribe - www.youscribe.com


2012/5/8 Tommi Virtanen <tv@inktank.com>:
> On Mon, May 7, 2012 at 3:58 PM, Tommi Virtanen <tv@inktank.com> wrote:
>> The current work is in a branch called "ceph-1", and will be improved
>> upon, but it is now at a stage where others should start looking at
>> it.
>
> Make that "chef-1".
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: OSD hotplugging & Chef cookbook ("chef-1")

[Danny Kukawka]

[-- Attachment #1: Type: text/plain, Size: 388 bytes --]

Hi,

Am 08.05.2012 00:58, schrieb Tommi Virtanen:
[...]
> The current work is in a branch called "ceph-1", and will be improved
> upon, but it is now at a stage where others should start looking at
> it.
[...]

And where can I find this branch? I've checked the git repo at:

	https://github.com/ceph/ceph-cookbooks

But couldn't find any branch called ceph-1.

Danny



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: OSD hotplugging & Chef cookbook ("chef-1")

[Tommi Virtanen]

On Thu, Jun 14, 2012 at 6:32 AM, Danny Kukawka <danny.kukawka@bisect.de> wrote:
> And where can I find this branch? I've checked the git repo at:
>
>        https://github.com/ceph/ceph-cookbooks
>
> But couldn't find any branch called ceph-1.

Use master of ceph-cookbooks.git, and where those instructions said
"ceph-1", put in "master" (= use master of ceph.git).

The instructions from that email are being distilled into proper
documentation at

http://ceph.com/docs/master/install/chef/
http://ceph.com/docs/master/config-cluster/chef/
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html