appenv: Initial commit

This commit is contained in:
Javi Fontan committed 2012-09-10 17:48:01 +02:00
commit ae746b7b8a
6 files changed
+960

No files matched your search

+156
View File
@@ -0,0 +1,156 @@
# Overview
Installing services can sometimes be tedious. Getting the VM ins good shape, installing the packages needed, deploying the application and configuring it. On case of disaster or scaling the service you will need to perform again these tasks. To automate all this burden we now have Configuration Management software like [chef](http://www.opscode.com/chef/), [puppet](http://puppetlabs.com/puppet/what-is-puppet/) or [cfengine](http://cfengine.com/).
With this software we can script the installation and configuration of applications and services. You can also describe the services and their configuration that will be installed on a node so we can automate its installation. We will also be sure that each installation will be equal to other using the same configuration.
The tool oneenv will make use of [chef-solo](http://wiki.opscode.com/display/chef/Chef+Solo). It lets you register chef configurations and instantiate them using predefined templates and images.
# Guide
The first step you need to do is to prepare an OS image to be able to use chef recipes from contextualization (link to guide) or download an already prepared image from the marketplace. Then we need to create a template referring to this image.
The next step will be describing our new node. You can follow the [chef guide](http://wiki.opscode.com/display/chef/Nodes) to understand these files. For these examples we are going to use this file:
{
"name": "wordpress",
"run_list": [
"recipe[mysql::server]",
"recipe[wordpress]"
],
"wordpress": {
"db": {
"database": "${WP_DB_NAME|wordpress}",
"user": "${WP_DB_USER|wordpress}",
"password": "${WP_DB_PASSWORD|password}"
}
},
"mysql": {
"server_root_password": "${MYSQL_ROOT_PASSWORD|password}"
}
}
The node files are JSON encoded and contain the recipes that are going to be run on the VM and parameters that will be used to install and configure the software. You will notice that there are some strange values in this form:
${WP_DB_NAME|wordpress}
oneenv comes with a template engine that will let you create node files with variables. These variables are named and can contain default values. the form is:
${NAME[|<default value>]}
The name will be used to refer to it on VM instantiation (we will see this later on) and the default value will substitute the variable if it is not explicitly specified.
We save this file as node.json and now comes the registration part. oneenv will let us register it in the OpenNebula database. To register the node file we can use the command `oneenv create`:
$ oneenv create node.json
21
The number returned is the identifier to our newly registered node file. We can also specify some parameters:
-n, --name name Name of the role
-t, --templates t1,t2 Templates compatible with the role
-c, --cookbooks cookbooks URL to extra cookbooks
On VM creation the recipes specified in the node file will be automatically downloaded from the standard chef cookbooks repository but we can specify a URL of a tar.gz containing extra cookbooks that you have created. For example:
$ oneenv create --name my_blog --templates ubuntu,centos --cookbooks http://some.url.com/cookbooks.tar.gz node.json
22
We can list the nodes we already have defined:
$ oneenv list
ID NAME
21 wordpress
22 my_blog
And show the properties:
$ ./oneenv show my_blog
ROLE 22 INFORMATION
ID : 22
NAME : my_blog
USER : oneadmin
GROUP : oneadmin
COMPATIBLE TEMPLATES: ubuntu, centos
COOKBOOKS : http://some.url.com/cookbooks.tar.gz
PERMISSIONS
OWNER : um-
GROUP : ---
OTHER : ---
DEFAULT VARIABLES
WP_DB_NAME : wordpress
WP_DB_USER : wordpress
WP_DB_PASSWORD : password
MYSQL_ROOT_PASSWORD : password
ROLE DEFINITION
{
"name": "wordpress",
"run_list": [
"recipe[mysql::server]",
"recipe[wordpress]"
],
"wordpress": {
"db": {
"database": "${WP_DB_NAME|wordpress}",
"user": "${WP_DB_USER|wordpress}",
"password": "${WP_DB_PASSWORD|password}"
}
},
"mysql": {
"server_root_password": "${MYSQL_ROOT_PASSWORD|password}"
}
}
There we can see the node file and its properties. It is also handy the default variables. Those are the ones we can change on instantiation.
We will be also able to update the node file using `oneenv update <node>`. A text editor will be run letting us modify the node file.
To instantiate a VM that will use this node we will use the command `oneenv instantiate <template> <node>`. This command will also let us change the variables in the node using the parameter `-d`. For example:
$ oneenv instantiate ubuntu my_blog -d WP_DB_NAME=myblog,MYSQL_ROOT_PASSWORD=some_other_password
Now we can check that a new VM was created:
$ onevm list
ID USER GROUP NAME STAT UCPU UMEM HOST TIME
16 oneadmin oneadmin ubuntu pend 0 0K 0d 00h00
And that the contextualization contains the parameters we provided:
$ onevm show 16
[...]
VIRTUAL MACHINE TEMPLATE
CONTEXT=[
COOKBOOKS="http://some.url.com/cookbooks.tar.gz",
MYSQL_ROOT_PASSWORD="some_other_password",
ROLE="eyJuYW1lIjoid29yZHByZXNzIiwicnVuX2xpc3QiOlsicmVjaXBlW215c3FsOjpzZXJ2ZXJdIiwicmVjaXBlW3dvcmRwcmVzc10iXSwid29yZHByZXNzIjp7ImRiIjp7ImRhdGFiYXNlIjoiJHtXUF9EQl9OQU1FfHdvcmRwcmVzc30iLCJ1c2VyIjoiJHtXUF9EQl9VU0VSfHdvcmRwcmVzc30iLCJwYXNzd29yZCI6IiR7V1BfREJfUEFTU1dPUkR8cGFzc3dvcmR9In19LCJteXNxbCI6eyJzZXJ2ZXJfcm9vdF9wYXNzd29yZCI6IiR7TVlTUUxfUk9PVF9QQVNTV09SRHxwYXNzd29yZH0ifX0=",
TARGET="hdb",
WP_DB_NAME="myblog" ]
[...]
These parameters will be used by the contextualization script inside the VM image and will install the described node.
# Preparing the VM
The VM images where these configurations will be applied need to be prepared to use the contextualization information provided by oneenv. The requirements are:
* chef-solo installed
* ruby installed
* OpenNebula contextualization+chef scripts package installed
To install chef solo you can follow [chef guides](http://wiki.opscode.com/display/chef/Installing+Chef+Client+and+Chef+Solo). Those guides have instructions on how to install it on various distributions. Most probably you wont need to install ruby as it is a chef requirement so after installing chef-solo you will have it available.
The OpenNebula contextualization package can be found at the [download page](http://downloads.c12g.com/context). There are two versions, one for rpm based distributions (CentOS/RedHat) and another for debian ones (Debian/Ubuntu).
Installing the contstualization package will enable the machine to use OpenNebula network configuration. Will clean udev configuration that could create errors and install the `script one-chef` that will launch chef on start.
Vendored Executable
+206
View File
@@ -0,0 +1,206 @@
#!/usr/bin/env ruby
# -------------------------------------------------------------------------- #
# Copyright 2010-2012, C12G Labs S.L. #
# #
# Licensed under the C12G Commercial Open-source License (the #
# "License"); you may not use this file except in compliance #
# with the License. You may obtain a copy of the License as part #
# of the software distribution. #
# #
# Unless agreed to in writing, software distributed under the #
# License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES #
# OR CONDITIONS OF ANY KIND, either express or implied. See the #
# License for the specific language governing permissions and #
# limitations under the License. #
#--------------------------------------------------------------------------- #
ONE_LOCATION=ENV["ONE_LOCATION"]
if !ONE_LOCATION
RUBY_LIB_LOCATION="/usr/lib/one/ruby"
else
RUBY_LIB_LOCATION=ONE_LOCATION+"/lib/ruby"
end
$: << RUBY_LIB_LOCATION
$: << RUBY_LIB_LOCATION+"/cli"
$: << '.'
require 'onechef.rb'
require 'base64'
require 'command_parser'
require 'appenv_helper'
require 'one_helper/onetemplate_helper'
require 'pp'
require 'ap'
NAME={
:name => "name",
:short => "-n name",
:large => "--name name",
:format => String,
:description => "Name of the node"
}
TEMPLATES={
:name => "templates",
:short => "-t t1,t2",
:large => "--templates t1,t2",
:format => Array,
:description => "Templates compatible with the node"
}
VARIABLES={
:name => "variables",
:short => "-d v1=d1,v2=d2",
:large => "--data v1=d1,v2=d2",
:format => Array,
:description => "Specific variables"
}
NODE={
:name => "node",
:short => "-f node",
:large => "--node node",
:format => String,
:description => "Node file"
}
COOKBOOKS={
:name => "cookbooks",
:short => "-c cookbooks",
:large => "--cookbooks cookbooks",
:format => String,
:description => "URL to extra cookbooks"
}
cmd = CommandParser::CmdParser.new(ARGV) do
usage "`appenv` <command> [<args>] [<options>]"
version OpenNebulaHelper::ONE_VERSION
helper = AppEnvHelper.new
template_helper = OneTemplateHelper.new
set :format, :node, AppEnvHelper.to_id_desc do |arg|
helper.to_id(arg)
end
set :format, :node_list, AppEnvHelper.list_to_id_desc do |arg|
helper.list_to_id(arg)
end
set :format, :templateid, OneTemplateHelper.to_id_desc do |arg|
template_helper.to_id(arg)
end
addnode_desc = <<-EOT.unindent
Adds a new node
EOT
command :create, addnode_desc, [:node_file],
:options => [NAME, TEMPLATES, COOKBOOKS] do
node=OpenNebula::ChefConf.new
node.node=File.read args.first
node.name=options[:name] if options[:name]
node.templates=options[:templates] if options[:templates]
node.cookbooks=options[:cookbooks] if options[:cookbooks]
client=Client.new
res=node.save(client)
if OpenNebula.is_error? res[0]
STDERR.puts(res[1].message)
-1
else
puts "ID: #{res.first.id}"
0
end
end
delete_desc = <<-EOT.unindent
Deletes the given node
States: ANY
EOT
command :delete, delete_desc, [:range, :node_list] do
helper.perform_actions(args[0],options,"deleted") do |node|
node.delete
end
end
show_desc = <<-EOT.unindent
Show defined node
EOT
command :show, show_desc, [:node], :options=>OpenNebulaHelper::XML do
helper.show_resource(args[0],options)
end
list_desc = <<-EOT.unindent
Lists defined node
EOT
command :list, list_desc,:options=>OpenNebulaHelper::XML do
helper.list_pool( options )
end
update_desc = <<-EOT.unindent
Update a defined node
EOT
command :update, update_desc, [:node],
:options=>[NAME, TEMPLATES, NODE, COOKBOOKS] do
client=OpenNebula::Client.new
doc=OpenNebula::ChefDoc.new(
OpenNebula::ChefDoc.build_xml(args[0]),
client)
doc.info
node=doc.node
if options[:node]
text=File.read(options[:node])
node.node=text
end
node.name=options[:name] if options[:name]
node.templates=options[:templates] if options[:templates]
node.cookbooks=options[:cookbooks] if options[:cookbooks]
doc.update(node.to_json)
0
end
instantiate_desc = <<-EOT.unindent
Instantiate a template with a node
EOT
command :instantiate, instantiate_desc, [:templateid], [:node], :options=>[VARIABLES] do
client=OpenNebula::Client.new
xml=OpenNebula::ChefDoc.build_xml(args[1].to_i)
chef_doc=OpenNebula::ChefDoc.new(xml, client)
res=chef_doc.instantiate(args[0].to_i, options[:variables]||{})
if OpenNebula.is_error? res
STDERR.puts res.message
next 255
end
puts "VM ID: #{res.id}"
0
end
end
+112
View File
@@ -0,0 +1,112 @@
# -------------------------------------------------------------------------- #
# Copyright 2010-2012, C12G Labs S.L. #
# #
# Licensed under the C12G Commercial Open-source License (the #
# "License"); you may not use this file except in compliance #
# with the License. You may obtain a copy of the License as part #
# of the software distribution. #
# #
# Unless agreed to in writing, software distributed under the #
# License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES #
# OR CONDITIONS OF ANY KIND, either express or implied. See the #
# License for the specific language governing permissions and #
# limitations under the License. #
#--------------------------------------------------------------------------- #
require 'one_helper'
class AppEnvHelper < OpenNebulaHelper::OneHelper
def self.rname
"DOCUMENT"
end
def self.conf_file
"appenv.yaml"
end
private
def factory(id = nil)
if id
OpenNebula::ChefDoc.new_with_id(id, @client)
else
xml = OpenNebula::ChefDoc.build_xml
OpenNebula::ChefDoc.new(xml, @client)
end
end
def factory_pool(filter)
OpenNebula::ChefDocPool.new(@client)
end
def format_resource(node)
str_h1="%-80s"
str="%-20s: %-20s"
CLIHelper.print_header(
str_h1 % "NODE #{node['ID']} INFORMATION")
puts str % ["ID", node.id.to_s]
puts str % ["NAME", node.name]
puts str % ["USER", node['UNAME']]
puts str % ["GROUP", node['GNAME']]
if node.node['templates']
puts str % ["COMPATIBLE TEMPLATES",
node.node['templates'].join(', ')]
end
if node.node['cookbooks']
puts str % ["COOKBOOKS", node.node['cookbooks']]
end
puts
CLIHelper.print_header(str_h1 % "PERMISSIONS",false)
["OWNER", "GROUP", "OTHER"].each { |e|
mask = "---"
mask[0] = "u" if node["PERMISSIONS/#{e}_U"] == "1"
mask[1] = "m" if node["PERMISSIONS/#{e}_M"] == "1"
mask[2] = "a" if node["PERMISSIONS/#{e}_A"] == "1"
puts str % [e, mask]
}
puts
CLIHelper.print_header(str_h1 % "DEFAULT VARIABLES",false)
node.node['defaults'].each do |var|
puts str % var
end if node.node['defaults']
puts
CLIHelper.print_header(str_h1 % "NODE DEFINITION",false)
puts JSON.pretty_generate(node.node['node'])
puts
end
def format_pool(options)
config_file = self.class.table_conf
table = CLIHelper::ShowTable.new(config_file, self) do
column :ID, "Node Identifier",
:size=>5 do |d|
d['ID']
end
column :NAME, "Name of the node",
:size=>16 do |d|
d['NAME']
end
default :ID, :NAME
end
table
end
end
+64
View File
@@ -0,0 +1,64 @@
# -------------------------------------------------------------------------- #
# Copyright 2010-2012, C12G Labs S.L. #
# #
# Licensed under the C12G Commercial Open-source License (the #
# "License"); you may not use this file except in compliance #
# with the License. You may obtain a copy of the License as part #
# of the software distribution. #
# #
# Unless agreed to in writing, software distributed under the #
# License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES #
# OR CONDITIONS OF ANY KIND, either express or implied. See the #
# License for the specific language governing permissions and #
# limitations under the License. #
#--------------------------------------------------------------------------- #
module OpenNebula
class OneCast
attr_reader :vars, :error
REG_VAR=/\$\{([^}|]+)(\|([^}]+))?\}/
def initialize(template, vars={})
@vars=ENV.to_hash.merge(vars)
@error=nil
@template=template.clone
end
def rewrite_template
@error=nil
@template.gsub(REG_VAR) do |var|
match=REG_VAR.match(var)
var_name=match[1]
default_value=match[3]
d=@vars[var_name]
d||=default_value
if !d
@error||=''
@error+="Variable '#{var_name}' not set. "
end
d
end
end
def get_defaults
defaults=Hash.new
@template.scan(REG_VAR) do |match|
var_name=match[0]
default_value=match[2]
defaults[var_name]=default_value
end
defaults
end
end
end
+157
View File
@@ -0,0 +1,157 @@
# -------------------------------------------------------------------------- #
# Copyright 2010-2012, C12G Labs S.L. #
# #
# Licensed under the C12G Commercial Open-source License (the #
# "License"); you may not use this file except in compliance #
# with the License. You may obtain a copy of the License as part #
# of the software distribution. #
# #
# Unless agreed to in writing, software distributed under the #
# License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES #
# OR CONDITIONS OF ANY KIND, either express or implied. See the #
# License for the specific language governing permissions and #
# limitations under the License. #
#--------------------------------------------------------------------------- #
require 'rubygems'
require 'json'
require 'OpenNebula'
#require 'OpenNebula/DocumentPool'
#require 'OpenNebula/Document'
require 'DocumentJSON'
require 'DocumentPoolJSON'
require 'onecast'
module OpenNebula
class ChefConf
TEMPLATE_TAG='BODY'
def initialize(conf=nil)
case conf
when String
@chef=JSON.parse(conf)
when Hash
@chef=conf
else
@chef=Hash.new
end
end
def name=(_name)
@chef['name']=_name
end
def node=(node)
cast=OneCast.new(node)
@chef['defaults']=cast.get_defaults
@chef['node']=JSON.parse(node)
@chef['name']=@chef['node']['name'] if @chef['node']['name']
end
def to_json
@chef.to_json
end
def [](key)
@chef[key]
end
def variables=(vars)
@chef['variables']=vars
end
def templates=(templates)
@chef['templates']=templates
end
def cookbooks=(cookbooks)
@chef['cookbooks']=cookbooks
end
def save(client)
doc=ChefDoc.new(ChefDoc.build_xml, client)
name=@chef['name']
[doc, doc.allocate(@chef.to_json, name)]
end
end
class ChefDoc < DocumentJSON
DOCUMENT_TYPE = 666
def node
if !defined?(@node)
@node=ChefConf.new(self["/DOCUMENT/TEMPLATE/#{TEMPLATE_TAG}"])
end
@node
end
def instantiate(template, variables={})
xml=OpenNebula::Template.build_xml(template.to_i)
template=OpenNebula::Template.new(xml, @client)
res=template.info
return res if OpenNebula.is_error? res
res=self.info
return res if OpenNebula.is_error? res
# Create context if it does not exist
if !template.has_elements?('/VMTEMPLATE/TEMPLATE/CONTEXT')
template.add_element('/VMTEMPLATE/TEMPLATE', 'CONTEXT'=>nil)
end
# Encode the node
node_string=node['node'].to_json
node_base64=Base64.encode64(node_string).delete("\n")
template.add_element('/VMTEMPLATE/TEMPLATE/CONTEXT', 'NODE'=>
node_base64)
if variables && !variables.empty?
hash=Hash.new
variables.each do |var|
key, val=var.split('=')
hash[key]=val
end
template.add_element('/VMTEMPLATE/TEMPLATE/CONTEXT', hash)
end
if node['cookbooks']
template.add_element('/VMTEMPLATE/TEMPLATE/CONTEXT',
'COOKBOOKS' => node['cookbooks'])
end
t=template.element_xml('/VMTEMPLATE/TEMPLATE')
xml=OpenNebula::VirtualMachine.build_xml()
vm=OpenNebula::VirtualMachine.new(xml, @client)
res=vm.allocate(t)
if OpenNebula.is_error? res
return res
else
vm
end
end
end
class ChefDocPool < DocumentPoolJSON
DOCUMENT_TYPE = 666
def factory(element_xml)
ChefDoc.new(element_xml, @client)
end
end
end
+265
View File
@@ -0,0 +1,265 @@
#!/usr/bin/env ruby
# -------------------------------------------------------------------------- #
# Copyright 2010-2012, C12G Labs S.L. #
# #
# Licensed under the C12G Commercial Open-source License (the #
# "License"); you may not use this file except in compliance #
# with the License. You may obtain a copy of the License as part #
# of the software distribution. #
# #
# Unless agreed to in writing, software distributed under the #
# License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES #
# OR CONDITIONS OF ANY KIND, either express or implied. See the #
# License for the specific language governing permissions and #
# limitations under the License. #
#--------------------------------------------------------------------------- #
require "json"
require 'pp'
require 'fileutils'
require 'tempfile'
require 'base64'
include FileUtils
class OneCast
attr_reader :vars, :error
REG_VAR=/\$\{([^}|]+)(\|([^}]+))?\}/
def initialize(template, vars)
@vars=ENV.to_hash.merge(vars)
pp vars
@error=nil
@template=template.clone
end
def rewrite_template
@error=nil
@template.gsub(REG_VAR) do |var|
match=REG_VAR.match(var)
var_name=match[1]
default_value=match[3]
d=@vars[var_name]
d||=default_value
if !d
@error||=''
@error+="Variable '#{var_name}' not set. "
end
d
end
end
private
def parse_data_variable(text)
m=(/^([^=]+)=["']?(.*)["']?/m).match(text)
if m
m[1,2]
else
nil
end
end
end
class Cookbook
DEFAULT_OPTIONS={
:path => '/var/tmp/chef'
}
def initialize(path, source_path, recipes)
@path=path
@source_path=source_path
@recipes=recipes
end
def cookbooks_path
@path+'/cookbooks'
end
def site_cookbooks_path
@path+'/site_cookbooks'
end
def prepare
mkdirs
prepare_site_cookbooks
uncompress_cookbooks
get_recipes
end
def mkdirs
mkdir_p(cookbooks_path)
mkdir_p(site_cookbooks_path)
end
def prepare_site_cookbooks
script=<<EOT
#!/bin/bash -e
#apt-get install -y git
cd #{site_cookbooks_path}
git init
git config user.name 'OpenNebula Chef Contextualization'
git config user.email 'chef@opennebula.org'
touch delete_me
git add delete_me
git commit -am'Initial Commit'
EOT
tempfile=Tempfile.new('one-chef-')
tempfile.write(script)
chmod(0555, tempfile.path)
tempfile.close
system(tempfile.path)
end
def recipe_exists?(name)
File.directory?(File.join(cookbooks_path, name))||
File.directory?(File.join(site_cookbooks_path, name))
end
def get_recipes
missing_recipes=@recipes.reject do |recipe|
recipe_exists?(recipe)
end
missing_recipes.each do |recipe|
system("knife cookbook site install #{recipe} -o " <<
site_cookbooks_path)
end
end
def uncompress_cookbooks
if ENV['COOKBOOKS']
command="wget #{ENV['COOKBOOKS']} -O #{cookbooks_path}.tar.gz"
system(command)
system("cd #{cookbooks_path} ; " <<
"tar xzf #{File.join(@source_path, 'cookbooks.tar.gz')}")
end
if File.exist?(File.join(@source_path, 'cookbooks.tar.gz'))
system("cd #{cookbooks_path} ; " <<
"tar xzf #{File.join(@source_path, 'cookbooks.tar.gz')}")
end
end
end
class Role
attr_reader :conf, :conf_json
DEFAULT_OPTIONS={
:path => '/var/tmp/chef',
:variables => {}
}
def initialize(conf, options={})
@options=DEFAULT_OPTIONS.merge(options)
@conf=cast(conf)
@conf_json=JSON.parse(@conf)
end
def json
@conf_json
end
def cast(conf)
c=OneCast.new(conf, @options[:variables])
c.rewrite_template
end
def get_recipes
run_list=conf_json['run_list']
recipes=run_list.map do |r|
m=r.match(/^recipe\[([^:]+)(::.*)?\]$/)
if m
m[1]
else
nil
end
end.compact
end
end
class Chef
DEFAULT_OPTIONS={
:path => '/var/tmp/chef',
:source_path => '/mnt'
}
def initialize(node, options={})
@role_file=node
@options=DEFAULT_OPTIONS.merge(options)
@role=Role.new(@role_file)
@cookbooks=Cookbook.new(
@options[:path],
@options[:source_path],
@role.get_recipes)
end
def prepare
prepare_directories
create_conf_file
@cookbooks.prepare
file=open(File.join(@options[:path], 'node.json'), 'w')
file.write(@role.conf)
file.close
end
def run
system("chef-solo -c #{File.join(@options[:path], 'chef.conf')} "<<
"-j #{File.join(@options[:path], 'node.json')}")
end
def prepare_directories
mkdir_p(@options[:path])
mkdir_p(File.join(@options[:path], 'cache'))
end
def create_conf_file
conf=<<EOT
file_cache_path "#{@options[:path]}/cache"
cookbook_path ["#{@options[:path]}/cookbooks", "#{@options[:path]}/site_cookbooks"]
EOT
file=open(File.join(@options[:path], 'chef.conf'), "w")
file.write(conf)
file.close
end
end
options={}
if ENV['DEBUG_PATH']
def system(arg)
puts arg
end
options={
:path => ENV['DEBUG_PATH']
}
end
role_encoded=ENV['ROLE']
role_decoded=Base64::decode64(role_encoded)
chef=Chef.new(role_decoded, options)
chef.prepare
chef.run