Actions: | Security

AllGoodBits.org

Navigation: Home | Services | Tools | Articles | Other

Chef Cookbook Development

This article aims to be simple distillation of my reading, experimentation and learning about how best to develop Chef cookbooks with more than a nod to the principles of Test Driven Development.

Prerequisites

Software tools

First, some software installation so that the example activity/code can be reproducible. Feel free to use your own processes and substitute alternative OS/distro, virtualization software choices. The process of cookbook development is more or less the same across all Linux/Windows/OSX platforms.

  • Chef Development Kit. You'll probably want to ensure that '/opt/chefdk/bin:/opt/chefdk/embedded/bin' are at the beginning of your $PATH. Rubocop and Foodcritic are both included.
  • VirtualBox
  • Vagrant
  • Vagrant plugins: vagrant-berkshelf, vagrant-omnibus, test-kitchen
  • rubygems: kitchen-vagrant

Tool configuration
Chef
Vagrant
  • Get a/some boxes, if you like to save time once you are ready to run Test Kitchen (although TK will get them for you, you might not want to wait at that point). Our Test Kitchen setup will by default use an Ubuntu box although Berkshelf's default scaffolding specifies a CentOS box as well.

Getting started

Goal

The intention here is to demonstrate a process for creating a new cookbook with example code/commands. For my example, I'm going to start with a small, simple to configure, network service: mini-httpd, a lesser known http server.

The new cookbook's requirements are:

  • install the package: mini-httpd

  • write a configfile: /etc/mini-httpd.conf

  • start the service: mini-httpd

  • some basic level of testing: serverspec

  • The content of the configfile should be:

    port=8080
    user=nobody
    nochroot
    data_dir=/usr/share/mini-httpd/html
    logfile=/var/log/mini-httpd.log
    pidfile=/var/run/mini-httpd.pid
    

which will result in the daemon listening on port 8080 on all interfaces for an HTTP request to which it will respond by serving a simple html page which will render to "It works!".


Implementation

$ berks cookbook mini-httpd --foodcritic

This creates extensive scaffolding for the new cookbook. Use berks help [cookbook] for details.

├── .git
|   <snip>
├── .gitignore
├── .kitchen
│   ├── kitchen-vagrant
│   └── logs
├── .kitchen.yml
├── Berksfile
├── Berksfile.lock
├── CHANGELOG.md
├── Gemfile
├── LICENSE
├── README.md
├── Thorfile
├── Vagrantfile
├── attributes
├── chefignore
├── files
│   └── default
├── libraries
├── metadata.rb
├── providers
├── recipes
│   └── default.rb
├── resources
├── templates
│   └── default
└── test
    └── integration
        └── default

Commit these files into git:

$ cd mini-httpd
mini-httpd $ git add .
mini-httpd $ git commit -m'initial scaffolding for cookbook: mini-httpd' -a

In order to get a minimal good cookbook, you'll want to edit/verify:

  • .kitchen.yml
  • README.md
  • metadata.rb
  • Berksfile
  • test/environments/kitchen.json
  • recipes/default.db

Edit/Verify .kitchen.yml

Change the provisioner to chef_zero because in constrast to Chef-Solo, Zero means that you don't have to special-case recipes/tests that use search. You'll want to verify that the list of platforms is correct. This is also the file where we specify the chef 'run list' and later might override the cookbook's default attributes for the purposes of testing, but more on that later. For now, we'll just use:

---
driver:
  name: vagrant

provisioner:
  name: chef_zero
  environments_path: test/environments
  client_rb:
    environment: kitchen

platforms:
  - name: ubuntu-12.04

suites:
  - name: default
    run_list:
      - recipe[mini-httpd::default]
    attributes:

If you want to be able to customize the VM that kitchen/vagrant makes for testing, you can make modifications to the 'driver' section, for example, specifying the hostname and forwarding ports from host to guest looks like:

---
driver:
  name: vagrant
  vm_hostname: 'mhv1.dev1.example.com'
  network:
  # Allow access to a web server which is useful when troubleshooting
  - - forwarded_port
    - guest: 80
      host: 8088
      auto_correct: true

Edit README.md

Berkshelf creates an example README.md; change it to document your cookbook: what it does, attributes you can pass and generally how to use it.

Edit/Verify metadata.rb

Verify that the Berkshelf-created boilerplate in here is good. This is also where you record any cookbooks that this cookbook depends upon and their version restrictions.

Edit/Verify Berksfile

If you need to tell berkshelf from where it should fetch dependencies.

cookbook 'openstack-common', github: 'stackforge/cookbook-openstack-common', branch: 'stable/grizzly'
cookbook 'ntp-wrapper', git: 'ssh://git@github.com:my-github-org/agb-ntp.git'

And if you do, you'll want to run berks install or berks update from your cookbook directory.


Edit/Verify test/environments/kitchen.json

This file allows kitchen access to "dummy" environment data, so that tests can be performed with chef-zero which doesn't have access to environment data that is in the real Chef Server.

{
  "name": "kitchen",
  "default_attributes": {
    "AZ": {
      "name": "kitchen",
      "mcps": [
        "localhost"
      ],
      "ops": [ "mc-tech-ops@cisco.com"
      ]
    }
  },
  "json_class": "Chef::Environment",
  "description": "AZ specific data for the kitchen AZ",
  "cookbook_versions": {
    },
  "chef_type": "environment"
}

Edit/Verify recipes/default.rb

This is where you write the code that actually makes something happen. At last!

package 'mini-httpd' do
  action :install
end
cookbook_file 'mini-httpd_conf' do
  source 'mini-httpd.conf'
  path '/etc/mini-httpd.conf'
  action :create
  mode 0644
  owner 'root'
  group 'root'
  notifies :restart, 'service[mini-httpd]'
end
cookbook_file 'mini-httpd' do
  source 'mini-httpd'
  path '/etc/default/mini-httpd'
  action :create
  mode 0644
  owner 'root'
  group 'root'
  notifies :restart, 'service[mini-httpd]'
end
service 'mini-httpd' do
  action [ :enable, :start ]
  supports :restart => true
end

The cookbook_file resource declared by cookbook_file 'mini-httpd_conf' takes its contents from the COOKBOOK_NAME/files/default sub-directory, the filename is specified by the source attribute. Not really: it searches a series of paths for a filename that matches the source attribute: files/<platform>-<version>/, files/<platform>/, files/default/, for example: files/ubuntu-12.04/, files/ubuntu, files/default. The analogous equivalent is also true for the sources of template resources and integration tests.

In our case, the contents of mini-httpd/files/default/mini-httpd.conf should be:

port=8080
user=nobody
nochroot
data_dir=/usr/share/mini-httpd/html
logfile=/var/log/mini-httpd.log
pidfile=/var/run/mini-httpd.pid

And the contents of mini-httpd/files/default/mini-httpd should be:

START=1
DAEMON_OPTS="-C /etc/mini-httpd.conf"

As we get more sophisticated, using attributes to allow variable configuration, we will remove this resource in favour of a template resource that, by default, will create the same configuration file with the same contents, but allow for alternative values for the mini-httpd directives.

At this point, you might commit your work to git.

mini-httpd $ git add files/
mini-httpd $ git commit -m'configure package, file, service for mini-httpd' -a

Test Kitchen, first run

kitchen test will result in vagrant instantiating a single ubuntu-12.04 VM, installing chef using the omnibus plugin, provisioning with chef-solo, configuring the VM according to our newly created default recipe from the mini-httpd cookbook and running any test suites we have specified in .kitchen.yml.

But at this stage, our process requires a manual verification step to satisfy ourselves that the package, file and service we attempted to manage are actually managed correctly.

$ kitchen create
$ kitchen converge
$ kitchen login

Now, logged in to your test VM, you can poke around manually to determine whether everything has been configured according to your will.

Later on, when we are running automated tests, we will just use:

$ kitchen test

If you are making lots of experimental changes to your cookbook, you can probably accelerate your test-kitchen workflow by separating the steps:

  • kitchen create
  • kitchen converge
  • kitchen setup
  • kitchen verify
  • kitchen login

If you do this, you can edit your cookbook and just run kitchen converge, avoiding the delay of the previous steps, in particular, the creation/provisioning of a new VM every time.

Extras
  • Most kitchen subcommands can be run with concurrency greater than zero, specify with -c N
  • Most kitchen subcommands can be given parameters to limit which instances will be affected, either by a list of instances or a regex: e.g. kitchen converge ubuntu
  • You can set the logging level for your kitchen commands with -l [debug | info | warn | error | fatal]

Static Analysis

Rubocop

rubocop will syntax-check and style-check your code based on the Ruby Style Guide. If you use -a, it will auto-correct where it can.

rubocop --show-cops

will show to which rules our check is subjected.

Foodcritic

Foodcritic is a style-checking tool for Chef cookbooks, hand it one or more paths to cookbook(s). We're hoping for a single blank line of output.


knife

If, from within the cookbook directory, you run:

mini-httpd $ knife cookbook test mini-httpd -o ..

knife will run a ruby syntax check against all files that end in .rb and .erb (respecting .chefignore).


Chefspec

This is (mostly?) for unit testing, so I'll leave this as a stub for later. Initially, we're going to concentrate our effort on functional testing with Serverspec.


Functional Testing

Serverspec

Create a directory for the Serverspec tests, then we can start to make tests in test/integration/default/serverspec/localhost/mini-httpd_spec.rb:

require 'serverspec'
set :backend, :exec

describe 'mini-httpd service' do
  it 'should be running and enabled' do
    expect(service 'mini-httpd').to be_running
    expect(service 'mini-httpd').to be_enabled
  end

  it 'should be listening on port 8080' do
    expect(port(8080)).to be_listening
  end
end

For details about what types are available see the serverspec documentation about its resource types.

NOTE Files that contain serverspec tests must be named to match /_spec.rb$/ or test kitchen will not use them.

If you are just modifying your tests, you can accelerate your test-kitchen workflow by separating the steps:

  • kitchen create
  • kitchen converge
  • kitchen setup
  • kitchen verify

If you do this, you can edit your tests and just run kitchen verify, avoiding the delay of the previous steps.


BATS

Bats, the Bourne-Again shell Test System.

Create a directory for the bats tests and add a file <cookbook_name>/test/integration/default/bats/<cookbook_name>.bats:

match() {
  local p=$1 v
  shift  for v
    do [[ $v = $p ]] && return
  done
return 1
}

@test "mini-httpd is listening to port 8080" {
  run nmap -sT -p8080 localhost
  match "80/tcp open  http" "${lines[@]}"
}
@test "Server accepts HTTP requests" {
  echo "GET / HTTP/1.1" | nc localhost 8080
}

This brings up another requirement to ensure that certain packages are available during testing, whether or not they're installed for real. Create a file <cookbook_name>/test/integration/default/bats/prepare_recipe.rb

%w{ nc nmap }.each { |pkg| package pkg }

Improvement: templating the config files

Instead of using a static file to provide the content for our configuration files, we can use erb templates. In order to do so, we modify recipes/default.rb using the template resource syntax:

# Install a templated configuration file for mini-httpd
template 'mini-httpd_conf' do
  source 'mini-httpd.conf.erb'
  path '/etc/mini-httpd.conf'
  action :create
  mode 0644
  owner 'root'
  group 'root'
  variables({
    :port => node['mini-httpd']['port'],
    :user => node['mini-httpd']['user'],
    :data_dir => node['mini-httpd']['data_dir'],
    :log_file => node['mini-httpd']['log_file']
  })
  notifies :restart, 'service[mini-httpd]'
end

and create

which defines the attributes (keys) that this cookbook recognizes and gives them default values.

default['mini-httpd']['port'] = '8080'
default['mini-httpd']['user'] = 'nobody'
default['mini-httpd']['data_dir'] = '/usr/share/mini-httpd/html'
default['mini-httpd']['log_file'] = '/var/log/mini-httpd.log'

as pointed to by the 'source' attribute of the template resource in our default recipe.

# /etc/mini-httpd.conf
#
<% if @port %>
  port=<%= @port %>
<% end %>
<% if @user %>
  user=<%= @user %>
<% end %>
<% if @data_dir %>
  data_dir=<%= @data_dir %>
<% end %>
<% if @log_file %>
  logfile=<%= @log_file %>
<% end %>

Modifications for wrapper cookbooks

Wrapper cookbooks are named agb-<cookbook_name> (I'm just using 'agb' for myself - pick your own prefix). Starting is the same as normal:

berks cookbook agb-ntp --foodcritic
cd agb-ntp
git init
git add .
git commit -m'Initial scaffolding for ntp wrapper cookbook' -a

Add the original to your dependency list, with an optional version requirement:

echo "depends 'ntp'" >> metadata.rb
echo "cookbook 'ntp', '~> 1.6.5'" >> Berksfile

For each recipe that you want to make available, create recipes(s) in your agb-<cookbook_name>/recipes/:

echo "include_recipe 'ntp'" >> recipes/default.rb

OR

echo "include_recipe 'graphite::carbon'" >> recipes/carbon.rb
echo "include_recipe 'graphite::web'" >> recipes/web.rb

Add that recipe to your default test suite, by modifying .kitchen.yml:

---
driver:
  name: vagrant

provisioner:
  name: chef_zero

platforms:
  - name: ubuntu-12.04

suites:
  - name: default
    run_list:
      - recipe[agb-ntp::default]
    attributes:

Overriding default attributes

One of the major reasons for using wrapper cookbooks is so that we can easily override attributes set by the cookbook maintainer as defaults to make them more useful for our purposes.

In this example, I set the array of upstream ntp servers to be MI[1-3].node[:domain], or otherwise, allow the cookbook default behaviour, setting default['ntp']['servers'] to various pool.ntp.org addresses.

Create attributes/default.rb:

unless node[:fqdn].start_with?('MI')
  (1..3).each do |n|
    default['ntp']['servers'].push('MI' + n.to_s + '.' + node[:domain])
  end
end

Overriding default attributes for testing only

In .kitchen.yml, we can set specific attributes that are appropriate/required for the test run. For example, preventing the default behaviour of using a apt cacher/proxy requires us to set ['apt']['cacher_ipaddress']:

```yaml
suites:
  - name: default
    run_list:
      - recipe[mc-global::default]
    attributes:
      apt:
        cacher_ipaddress: false
```

Summary

  1. Create:

    berks cookbook <cookbook_name> --foodcritic
    cd <cookbook_name>
    git add .
    git commit -m'initial scaffolding for cookbook: <cookbook_name>' -a
    
  2. Edit, minimally:

    • README.md
    • metadata.rb
    • recipes/default.rb
    • .kitchen.yml
  3. Test for syntax/style:

    rubocop -a
    foodcritic .
    
  4. Create functional tests:

    • serverspec tests go in <cookbook_name>/test/integration/default/serverspec/localhost/<cookbook_name>_spec.rb
  5. Commit, submit to review, celebrate!