Chef Cookbook Development
This article aims to be simple distillation of my reading, experimentation and learning about how best to develop Chef cookbooks with more than a nod to the principles of Test Driven Development.
Prerequisites
Software tools
First, some software installation so that the example activity/code can be reproducible. Feel free to use your own processes and substitute alternative OS/distro, virtualization software choices. The process of cookbook development is more or less the same across all Linux/Windows/OSX platforms.
- Chef Development Kit. You'll probably want to ensure that '/opt/chefdk/bin:/opt/chefdk/embedded/bin' are at the beginning of your $PATH. Rubocop and Foodcritic are both included.
- VirtualBox
- Vagrant
- Vagrant plugins: vagrant-berkshelf, vagrant-omnibus, test-kitchen
- rubygems: kitchen-vagrant
Tool configuration
Chef
- ~/.chef/knife.rb allows the configuration of interactions with the Chef server and some of these items are inherited by Berkshelf as well. Justin Timberman published a highly commented knife.rb.
Vagrant
- Get a/some boxes, if you like to save time once you are ready to run Test Kitchen (although TK will get them for you, you might not want to wait at that point). Our Test Kitchen setup will by default use an Ubuntu box although Berkshelf's default scaffolding specifies a CentOS box as well.
Getting started
Goal
The intention here is to demonstrate a process for creating a new cookbook with example code/commands. For my example, I'm going to start with a small, simple to configure, network service: mini-httpd, a lesser known http server.
The new cookbook's requirements are:
install the package: mini-httpd
write a configfile: /etc/mini-httpd.conf
start the service: mini-httpd
some basic level of testing: serverspec
The content of the configfile should be:
port=8080 user=nobody nochroot data_dir=/usr/share/mini-httpd/html logfile=/var/log/mini-httpd.log pidfile=/var/run/mini-httpd.pid
which will result in the daemon listening on port 8080 on all interfaces for an HTTP request to which it will respond by serving a simple html page which will render to "It works!".
Implementation
$ berks cookbook mini-httpd --foodcritic
This creates extensive scaffolding for the new cookbook. Use berks help [cookbook] for details.
├── .git | <snip> ├── .gitignore ├── .kitchen │ ├── kitchen-vagrant │ └── logs ├── .kitchen.yml ├── Berksfile ├── Berksfile.lock ├── CHANGELOG.md ├── Gemfile ├── LICENSE ├── README.md ├── Thorfile ├── Vagrantfile ├── attributes ├── chefignore ├── files │ └── default ├── libraries ├── metadata.rb ├── providers ├── recipes │ └── default.rb ├── resources ├── templates │ └── default └── test └── integration └── default
Commit these files into git:
$ cd mini-httpd mini-httpd $ git add . mini-httpd $ git commit -m'initial scaffolding for cookbook: mini-httpd' -a
In order to get a minimal good cookbook, you'll want to edit/verify:
- .kitchen.yml
- README.md
- metadata.rb
- Berksfile
- test/environments/kitchen.json
- recipes/default.db
Edit/Verify .kitchen.yml
Change the provisioner to chef_zero because in constrast to Chef-Solo, Zero means that you don't have to special-case recipes/tests that use search. You'll want to verify that the list of platforms is correct. This is also the file where we specify the chef 'run list' and later might override the cookbook's default attributes for the purposes of testing, but more on that later. For now, we'll just use:
--- driver: name: vagrant provisioner: name: chef_zero environments_path: test/environments client_rb: environment: kitchen platforms: - name: ubuntu-12.04 suites: - name: default run_list: - recipe[mini-httpd::default] attributes:
If you want to be able to customize the VM that kitchen/vagrant makes for testing, you can make modifications to the 'driver' section, for example, specifying the hostname and forwarding ports from host to guest looks like:
--- driver: name: vagrant vm_hostname: 'mhv1.dev1.example.com' network: # Allow access to a web server which is useful when troubleshooting - - forwarded_port - guest: 80 host: 8088 auto_correct: true
Edit README.md
Berkshelf creates an example README.md; change it to document your cookbook: what it does, attributes you can pass and generally how to use it.
Edit/Verify metadata.rb
Verify that the Berkshelf-created boilerplate in here is good. This is also where you record any cookbooks that this cookbook depends upon and their version restrictions.
Edit/Verify Berksfile
If you need to tell berkshelf from where it should fetch dependencies.
cookbook 'openstack-common', github: 'stackforge/cookbook-openstack-common', branch: 'stable/grizzly' cookbook 'ntp-wrapper', git: 'ssh://git@github.com:my-github-org/agb-ntp.git'
And if you do, you'll want to run berks install or berks update from your cookbook directory.
Edit/Verify test/environments/kitchen.json
This file allows kitchen access to "dummy" environment data, so that tests can be performed with chef-zero which doesn't have access to environment data that is in the real Chef Server.
{ "name": "kitchen", "default_attributes": { "AZ": { "name": "kitchen", "mcps": [ "localhost" ], "ops": [ "mc-tech-ops@cisco.com" ] } }, "json_class": "Chef::Environment", "description": "AZ specific data for the kitchen AZ", "cookbook_versions": { }, "chef_type": "environment" }
Edit/Verify recipes/default.rb
This is where you write the code that actually makes something happen. At last!
package 'mini-httpd' do action :install end cookbook_file 'mini-httpd_conf' do source 'mini-httpd.conf' path '/etc/mini-httpd.conf' action :create mode 0644 owner 'root' group 'root' notifies :restart, 'service[mini-httpd]' end cookbook_file 'mini-httpd' do source 'mini-httpd' path '/etc/default/mini-httpd' action :create mode 0644 owner 'root' group 'root' notifies :restart, 'service[mini-httpd]' end service 'mini-httpd' do action [ :enable, :start ] supports :restart => true end
The cookbook_file resource declared by cookbook_file 'mini-httpd_conf' takes its contents from the COOKBOOK_NAME/files/default sub-directory, the filename is specified by the source attribute. Not really: it searches a series of paths for a filename that matches the source attribute: files/<platform>-<version>/, files/<platform>/, files/default/, for example: files/ubuntu-12.04/, files/ubuntu, files/default. The analogous equivalent is also true for the sources of template resources and integration tests.
In our case, the contents of mini-httpd/files/default/mini-httpd.conf should be:
port=8080 user=nobody nochroot data_dir=/usr/share/mini-httpd/html logfile=/var/log/mini-httpd.log pidfile=/var/run/mini-httpd.pid
And the contents of mini-httpd/files/default/mini-httpd should be:
START=1 DAEMON_OPTS="-C /etc/mini-httpd.conf"
As we get more sophisticated, using attributes to allow variable configuration, we will remove this resource in favour of a template resource that, by default, will create the same configuration file with the same contents, but allow for alternative values for the mini-httpd directives.
At this point, you might commit your work to git.
mini-httpd $ git add files/ mini-httpd $ git commit -m'configure package, file, service for mini-httpd' -a
Test Kitchen, first run
kitchen test will result in vagrant instantiating a single ubuntu-12.04 VM, installing chef using the omnibus plugin, provisioning with chef-solo, configuring the VM according to our newly created default recipe from the mini-httpd cookbook and running any test suites we have specified in .kitchen.yml.
But at this stage, our process requires a manual verification step to satisfy ourselves that the package, file and service we attempted to manage are actually managed correctly.
$ kitchen create $ kitchen converge $ kitchen login
Now, logged in to your test VM, you can poke around manually to determine whether everything has been configured according to your will.
Later on, when we are running automated tests, we will just use:
$ kitchen test
If you are making lots of experimental changes to your cookbook, you can probably accelerate your test-kitchen workflow by separating the steps:
- kitchen create
- kitchen converge
- kitchen setup
- kitchen verify
- kitchen login
If you do this, you can edit your cookbook and just run kitchen converge, avoiding the delay of the previous steps, in particular, the creation/provisioning of a new VM every time.
Extras
- Most kitchen subcommands can be run with concurrency greater than zero, specify with -c N
- Most kitchen subcommands can be given parameters to limit which instances will be affected, either by a list of instances or a regex: e.g. kitchen converge ubuntu
- You can set the logging level for your kitchen commands with -l [debug | info | warn | error | fatal]
Static Analysis
Rubocop
rubocop will syntax-check and style-check your code based on the Ruby Style Guide. If you use -a, it will auto-correct where it can.
rubocop --show-cops
will show to which rules our check is subjected.
Foodcritic
Foodcritic is a style-checking tool for Chef cookbooks, hand it one or more paths to cookbook(s). We're hoping for a single blank line of output.
knife
If, from within the cookbook directory, you run:
mini-httpd $ knife cookbook test mini-httpd -o ..
knife will run a ruby syntax check against all files that end in .rb and .erb (respecting .chefignore).
Chefspec
This is (mostly?) for unit testing, so I'll leave this as a stub for later. Initially, we're going to concentrate our effort on functional testing with Serverspec.
Functional Testing
Serverspec
Create a directory for the Serverspec tests, then we can start to make tests in test/integration/default/serverspec/localhost/mini-httpd_spec.rb:
require 'serverspec' set :backend, :exec describe 'mini-httpd service' do it 'should be running and enabled' do expect(service 'mini-httpd').to be_running expect(service 'mini-httpd').to be_enabled end it 'should be listening on port 8080' do expect(port(8080)).to be_listening end end
For details about what types are available see the serverspec documentation about its resource types.
NOTE Files that contain serverspec tests must be named to match /_spec.rb$/ or test kitchen will not use them.
If you are just modifying your tests, you can accelerate your test-kitchen workflow by separating the steps:
- kitchen create
- kitchen converge
- kitchen setup
- kitchen verify
If you do this, you can edit your tests and just run kitchen verify, avoiding the delay of the previous steps.
BATS
Bats, the Bourne-Again shell Test System.
Create a directory for the bats tests and add a file <cookbook_name>/test/integration/default/bats/<cookbook_name>.bats:
match() { local p=$1 v shift for v do [[ $v = $p ]] && return done return 1 } @test "mini-httpd is listening to port 8080" { run nmap -sT -p8080 localhost match "80/tcp open http" "${lines[@]}" } @test "Server accepts HTTP requests" { echo "GET / HTTP/1.1" | nc localhost 8080 }
This brings up another requirement to ensure that certain packages are available during testing, whether or not they're installed for real. Create a file <cookbook_name>/test/integration/default/bats/prepare_recipe.rb
%w{ nc nmap }.each { |pkg| package pkg }
Improvement: templating the config files
Instead of using a static file to provide the content for our configuration files, we can use erb templates. In order to do so, we modify recipes/default.rb using the template resource syntax:
# Install a templated configuration file for mini-httpd template 'mini-httpd_conf' do source 'mini-httpd.conf.erb' path '/etc/mini-httpd.conf' action :create mode 0644 owner 'root' group 'root' variables({ :port => node['mini-httpd']['port'], :user => node['mini-httpd']['user'], :data_dir => node['mini-httpd']['data_dir'], :log_file => node['mini-httpd']['log_file'] }) notifies :restart, 'service[mini-httpd]' end
and create
- attributes/default.rb
which defines the attributes (keys) that this cookbook recognizes and gives them default values.
default['mini-httpd']['port'] = '8080' default['mini-httpd']['user'] = 'nobody' default['mini-httpd']['data_dir'] = '/usr/share/mini-httpd/html' default['mini-httpd']['log_file'] = '/var/log/mini-httpd.log'
- templates/default/mini-httpd.conf.erb
as pointed to by the 'source' attribute of the template resource in our default recipe.
# /etc/mini-httpd.conf # <% if @port %> port=<%= @port %> <% end %> <% if @user %> user=<%= @user %> <% end %> <% if @data_dir %> data_dir=<%= @data_dir %> <% end %> <% if @log_file %> logfile=<%= @log_file %> <% end %>
Modifications for wrapper cookbooks
Wrapper cookbooks are named agb-<cookbook_name> (I'm just using 'agb' for myself - pick your own prefix). Starting is the same as normal:
berks cookbook agb-ntp --foodcritic cd agb-ntp git init git add . git commit -m'Initial scaffolding for ntp wrapper cookbook' -a
Add the original to your dependency list, with an optional version requirement:
echo "depends 'ntp'" >> metadata.rb echo "cookbook 'ntp', '~> 1.6.5'" >> Berksfile
For each recipe that you want to make available, create recipes(s) in your agb-<cookbook_name>/recipes/:
echo "include_recipe 'ntp'" >> recipes/default.rb
OR
echo "include_recipe 'graphite::carbon'" >> recipes/carbon.rb echo "include_recipe 'graphite::web'" >> recipes/web.rb
Add that recipe to your default test suite, by modifying .kitchen.yml:
--- driver: name: vagrant provisioner: name: chef_zero platforms: - name: ubuntu-12.04 suites: - name: default run_list: - recipe[agb-ntp::default] attributes:
Overriding default attributes
One of the major reasons for using wrapper cookbooks is so that we can easily override attributes set by the cookbook maintainer as defaults to make them more useful for our purposes.
In this example, I set the array of upstream ntp servers to be MI[1-3].node[:domain], or otherwise, allow the cookbook default behaviour, setting default['ntp']['servers'] to various pool.ntp.org addresses.
Create attributes/default.rb:
unless node[:fqdn].start_with?('MI') (1..3).each do |n| default['ntp']['servers'].push('MI' + n.to_s + '.' + node[:domain]) end end
Overriding default attributes for testing only
In .kitchen.yml, we can set specific attributes that are appropriate/required for the test run. For example, preventing the default behaviour of using a apt cacher/proxy requires us to set ['apt']['cacher_ipaddress']:
```yaml suites: - name: default run_list: - recipe[mc-global::default] attributes: apt: cacher_ipaddress: false ```
Summary
Create:
berks cookbook <cookbook_name> --foodcritic cd <cookbook_name> git add . git commit -m'initial scaffolding for cookbook: <cookbook_name>' -a
Edit, minimally:
- README.md
- metadata.rb
- recipes/default.rb
- .kitchen.yml
Test for syntax/style:
rubocop -a foodcritic .
Create functional tests:
- serverspec tests go in <cookbook_name>/test/integration/default/serverspec/localhost/<cookbook_name>_spec.rb
Commit, submit to review, celebrate!