Chef 0.10 Preview: Environments

Chef 0.10 recently hit the release candidate phase of the release process, so I’d like to give you all a preview of the new features in Chef while we’re finalizing the code. The biggest new feature in Chef 0.10 is environment support, so we’ll start there. If you’d like to play along at home, you’ll need an Opscode Platform Account or have a Chef 0.10 server installed, and have Chef 0.10 (RC) installed on the client.

Environments Today

With previous versions of Chef, you face a tough choice when automating the various infrastructures (e.g., production, staging, qa, dev, etc.) you have: you could run a unique instance of Chef Server (or use a unique Opscode Platform organization) for each one, which keeps each environment nicely isolated, but you’ll have to maintain a configuration for each Chef server. Alternatively, you could try to keep your environments separate by assigning nodes in each environment an “environment role”, but in this configuration, an update to cookbooks for your development environment could impact production.

Chef Environments

Chef 0.10 solves this problem by allowing you to set policies to dictate which versions of a given cookbook may be used in an environment. To get a feel for how this works, let’s take a look at an example environment file using the Ruby DSL:

# `name' and `description' are what you'd expect
name "prod"
description "OMG production"

                    # Use version 11.0.0 *only*
cookbook_versions   "couchdb"     => "= 11.0.0",
                    # use versions greater than 0.99.0
                    # and less than 0.100.0
                    "application" => "~> 0.99.0"

The name and description fields are pretty standard fare. It’s in the cookbook_versions field that we set our allowed versions policy. For each cookbook you have, you may set a version constraint to describe the allowed versions, where the version constraint is an (in)equality operator combined with a version. The available operators are:

• = – Equality
• > – Greater than
• >= – Greater than or equal to
• < – Less than
• <= – Less than or equal to
• ~> – Pessimistic greater than

These are mostly self explanatory, though the pessimistic greater than may be new to some of you. This constraint operator is borrowed from rubygems’ dependency syntax; its exact behavior depends on the version you specify in the version constraint. This is best illustrated with some examples:

• ~> 1.0 matches any version greater than or equal to 1.0, but less than 2.0.
• ~> 1.1 matches any version greater than or equal to 1.1, but less than 2.0
• ~> 1.1.0 matches any version greater than or equal to 1.1.0, but less than 1.2.0
• ~> 1.1.5 matches any version greater than or equal to 1.1.5, but less than 1.2.0

Trying it Out

Now that you’ve got the gist of how environments specify cookbook version constraints, let’s see it in action. In this example, I’m using the Opscode platform with a node I’ve already set up for testing. I start by upgrading my test node to Chef 0.10:

dan@chef-client$ sudo gem install chef --pre

Do the same for your laptop you run knife from. Back on my laptop, I create the production environment:

dan@laptop$ knife environment create production

For now we’ll leave the version constraints blank:

{
  "name": "production",
  "description": "lets pretend it's prod",
  "cookbook_versions": {
  },
  "json_class": "Chef::Environment",
  "chef_type": "environment",
  "default_attributes": {
  },
  "override_attributes": {
  }
}

I’ve also edited my node to assign it to the production environment and have the tmux recipe in its run list:

dan@laptop$ knife node edit ghost.local

{
  "name": "ghost.local",
  "chef_environment": "production",
  "normal": {
    "tags": [
    ]
  },
  "run_list": [
    "recipe[tmux]"
  ]
}

When I run chef, I see that tmux is installed:

dan@chef-client$ sudo chef-client
INFO: *** Chef 0.10.0.rc.0 ***
INFO: Run List is [recipe[tmux]]
INFO: Run List expands to [tmux]
INFO: Starting Chef Run for ghost.local
INFO: Loading cookbooks [tmux]
INFO: Processing package[tmux] action install (tmux::default line 19)
INFO: package[tmux] installed version 1.1-1
INFO: Chef Run complete in 11.682407 seconds

So far this is pretty basic chef. But now let’s introduce a bug into the tmux recipe:

dan@laptop$ vim repo/cookbooks/tmux/recipes/default.rb

# This is a bug:
raise "oops"

package "tmux" do
  action :install
end

And we change the cookbook’s version from 1.0.0 to 1.1.0:

dan@laptop$ vim repo/cookbooks/tmux/metadata.rb

maintainer       "Opscode, Inc."
maintainer_email "cookbooks@opscode.com"
license          "Apache 2.0"
description      "Installs tmux"
long_description IO.read(File.join(File.dirname(__FILE__), 'README.md'))

###
# Changed this from "1.0.0" to "1.1.0"
###
version          "1.1.0"

Upload the cookbook, and run chef-client. chef-client fails because of the bug in the cookbook:

dan@laptop$ knife cookbook upload tmux
Uploading tmux...
upload complete

dan@chef-client$ sudo chef-client
INFO: *** Chef 0.10.0.rc.0 ***
INFO: Run List is [recipe[tmux]]
INFO: Run List expands to [tmux]
INFO: Starting Chef Run for ghost.local
INFO: Loading cookbooks [tmux]
INFO: Storing updated cookbooks/tmux/recipes/default.rb in the cache.
INFO: Storing updated cookbooks/tmux/metadata.rb in the cache.
ERROR: Running exception handlers
FATAL: Saving node information to /var/chef/cache/failed-run-data.json
ERROR: Exception handlers complete
FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
FATAL: RuntimeError: oops

Oops, indeed. Now let’s use environments to lock production down to the cookbooks that we know are good:

dan@laptop$ knife environment edit production

{
  "name": "production",
  "description": "lets pretend it's prod",
  "cookbook_versions": {
    "tmux": "= 1.0.0"
  },
  "json_class": "Chef::Environment",
  "chef_type": "environment",
  "default_attributes": {
  },
  "override_attributes": {
  }
}

When we run chef-client again, we’ll use the 1.0.0 version of the tmux cookbook (without the bug):

dan@chef-client$ sudo chef-client
INFO: *** Chef 0.10.0.rc.0 ***
INFO: Run List is [recipe[tmux]]
INFO: Run List expands to [tmux]
INFO: Starting Chef Run for ghost.local
INFO: Loading cookbooks [tmux]
INFO: Storing updated cookbooks/tmux/recipes/default.rb in the cache.
INFO: Storing updated cookbooks/tmux/metadata.rb in the cache.
INFO: Processing package[tmux] action install (tmux::default line 19)
INFO: Chef Run complete in 3.805665 seconds
INFO: Running report handlers
INFO: Report handlers complete

Freezing Cookbooks

If you followed the example closely, you probably noticed that we could have uploaded the broken version of our tmux cookbook at version 1.0.0 which would have overwritten the “good” version. This certainly throws a wrench in our carefully crafted cookbook policy. Luckily, in Chef 0.10, you can freeze a version of a cookbook by using the --freeze option to knife cookbook upload. When a cookbook is frozen, you can only upload a cookbook with the same name and version by using the --force option:

dan@laptop$ knife cookbook upload redis --freeze
Uploading redis...
upload complete

dan@laptop$ knife cookbook show redis 0.1.6 |grep frozen 
frozen?:        true

dan@laptop$ knife cookbook upload redis
Uploading redis...
ERROR: Version 0.1.6 of cookbook redis is frozen. Use --force to override.

By combining frozen cookbook versions with environments, you can make sure that production is safe from accidental updates when testing out changes in your development infrastructure.

Environments Workflows

Environments are clearly a powerful feature for keeping changes isolated, but chances are you’ll want to use them differently than our contrived example. Here are some factors to consider when designing your workflow:

• How important is it to keep your environment files in source control?
• Do you want to edit environments in the management console (Web UI)?
• Do you want to be able to use the -E ENVIRONMENT flag to cookbook upload to automatically set cookbook version constraints on an environment?

If you place supreme importance on always keeping every last bit of data in version control, then, you’ll want to drive your use of environments by editing files only. On the other hand, if you’re already managing your cookbooks for separate environments using git branches, you already have your versioning policy information stored in your cookbooks’ metadata, so you might opt for a lighter weight approach.

Below are two recommended strategies for using Chef’s environments as you develop cookbooks and move your changes into production. For simplicity, we’re assuming you have only a development and production environment; you can extend them to include staging, QA or whatever you might have. We anticipate that you’ll find lots of other ways to use environments as you get familiar with them—that’s why we made them so flexible.

Branch Tracking Strategy

In the branch tracking strategy, you have a branch in your source control repo for each environment, and your cookbook versioning policy tracks whatever is in the tip of each branch. This strategy is simple and lightweight. For development environments that track the latest cookbooks, you continue to work as you have before environments, except that you need to bump the version before you upload the cookbook for testing. For environments that need special protection, i.e., production, you always upload cookbooks using the -E ENVRIONMENT and --freeze flags.

In development:

1. Bump the version number as appropriate.
2. Hack.

3. Upload and test:

   # Upload for dev is the same as always: knife cookbook upload my-app

4. Repeat 2 and 3 as necessary.

When you’re ready to move your changes into production:

# Upload for prod with automatic version constraints:
knife cookbook upload  my-app -E production --freeze

Adding the -E ENVIRONMENT option will cause knife to automatically set the version constraint on that environment to match the version you’re uploading.

Maximum Version Control Strategy

If you prefer to version control absolutely everything, then you need to use a file-editing based approach to environments. In your development environment, you work the same as the branch tracking strategy:

In development:

1. Bump the version number as appropriate.
2. Hack.

3. Upload and test:

    # Upload for dev is the same as always:
    knife cookbook upload my-app
   

4. Repeat 2 and 3 as necessary.

When you’re ready to move your cookbooks into production, you’ll do the following:

1. Upload and freeze your cookbooks:

    knife cookbook upload my-app --freeze
   

2. Modify your environment to prefer the new version you uploaded:

    (vim|emacs|mate|ed) YOUR_REPO/environments/production.rb
   

3. Upload the updated environment:

    knife environment from file production.rb
   

4. Deploy!

Of course, version control stickler that you are, you’ll commit and push your updates to the right branches during the process.

Further Reading

This is a far from comprehensive overview of Chef environments—I didn’t get to cover environment attributes, the webui, or environment-specific run lists in Roles. You can find more information on the chef wiki, and ask further questions on our mailing list and on our IRC channel (irc.freenode.net#chef).

Be sure to keep an eye on the Opscode blog, as we’ll be previewing more Chef 0.10 features in the upcoming days.