WEB Advent 2011 / Facilitating Development Environment Consistency

Every developer is different, but your team’s development environment does not have to be. A modern web app uses many different technologies. Common dependencies for a functioning web development environment include PHP and its necessary extensions, a web server, a database, testing frameworks, and other apps and services. The classic approach to assist a team of developers is to deploy a server and install all of the necessary packages one would need for development. This methodology provides a consistent environment for all of the developers on the team, therefore ensuring that all contributors receive the same experience throughout the development cycle. However, as consistent as a remote, homogeneous development might be, developing remotely introduces a number of problems, including speed sacrifices, and a nasty dependency on a reliable connection to the Internet. It is also difficult to tinker with experimental packages when you know it could affect the environment of everyone on your team.

Navigating a remote file system in order to find the file you want to edit can be cumbersome. Many text editors and IDEs have features that attempt to mitigate this pain, but I have never found one that works reliably and is comparable to the speed of local development. Saving a file over a network can introduce delay that adds up to a significant amount of lost productivity over time. File navigation aside, nothing beats the speed of working locally. In order to work on your local machine, you have to install all of the packages that the team is using, and each member has to do so as well. This approach introduces inconsistency. As developers, we seek consistency, speed, and, as much as we love the Internet, we would like to be able to work without it sometimes. So, what do we do? We build a machine within a machine that works on all platforms and ensures homogeneity across development environments.

Thanks to advances in virtualization technology and the open source community, a number of free tools exist that one can leverage to facilitate providing consistent virtual machines to your team.

VirtualBox, Vagrant, and Puppet

There are a few apps that are required in order to get started. VirtualBox is hardware virtualization software that allows you to run another operating system on your local machine. Vagrant is a set of scripts that allow you to easily manipulate those virtual machines from the command line. Puppet allows you to define what configuration changes should be made to the machine to have it operate how you would like.

Install VirtualBox

Download and install VirtualBox. You are now ready to install any operating system and run it locally.

Install Vagrant

Vagrant is a command line tool for building and distributing virtualized development environments with VirtualBox. Executing only a couple of commands, you can download and provision a virtual machine which matches that of your team.

Ruby and its package manager RubyGems are required to get going with Vagrant. There are detailed instructions by platform to help.

Once Ruby and RubyGems are installed, execute the following command on your local machine to ensure that your packages are up-to-date.

gem update --system

To install Vagrant, issue the following command on your local machine:

gem install vagrant

Configure Vagrant

Assuming that everything has gone well, configuring Vagrant should be easy. I have created a sample Vagrantfile and base Puppet configuration in order to provision a machine with Apache, MySQL, PHP, MySQL, Xdebug, PEAR, PHP CodeSniffer, and PHPUnit. You can clone my repository.

Once you have cloned the sample repository, it is time to install the Vagrant base image. The following command will download the box via the Internet and cache it. It is a 300 MB download, but you should only have to do it once.

vagrant box add base http://files.vagrantup.com/lucid32.box

The next step is to modify your Vagrantfile. Its configuration will forward ports from your local machine to the VM, mount a local folder of your choice within your VM, and initialize Puppet as your default provisioner. I have already done this for you.

There are a couple important configuration directives in the Vagrantfile. Let’s take a look.

# This enables Puppet as the default provisioner
config.vm.provision :puppet

# For verbose output, and you wish to do a dry run, substitute this line
config.vm.provision :puppet, :options => "--verbose --debug"

Once provisioning is complete, you will access the LAMP environment from your browser via http://localhost:8080/. The following line is the magic that accomplishes this:

config.vm.forward_port "http", 80, 8080

Being that this entire exercise exists in order to work with local files that are served by the VM, we will need to share a local directory to the machine. The following line mounts the folder named www in your home directory to Apache’s DocumentRoot/var/www) on the virtual machine.

config.vm.share_folder "www", "/var/www", "~/www"

Whether you’ve elected to leave this configuration as-is or modify it, you can boot the virtual machine and begin provisioning it with the following command on your local machine. Make sure you are in the directory containing the Vagrantfile.

vagrant up

(If you get a stack dump and a permission error, an explanation is available.)

Vagrant boots the base image with VirtualBox, then executes Puppet with the configuration located in manifests/base.pp. I have commented the lines in the base.pp file for you to better understand their function. Also, I have kept the configuration deliberately simple. More advanced configuration will require that you create modules that can be inherited, included, parameterized, &c.

For reference, here is more documentation on Puppet directives.

Many developers have also written and shared Puppet scripts with the world. Here is a more complicated LAMP setup.

Once the machine is up and running, if you wish to SSH to the machine, you can do so like this:

vagrant ssh

In order to SSH to the machine without using Vagrant (for example, when creating a tunnel to use a GUI MySQL client), use the longer syntax:

ssh vagrant@localhost -p 2222 (password: vagrant)

There are very few reasons that you should have to configure the virtual machine from within itself. Connecting to the machine in this way should be limited to running environment-dependent command line tools such as PHPUnit and MySQL.

Distributing and Provisioning with Puppet

Puppet is the true hero here. Once a virtual machine is distributed, Puppet makes it easy to keep the configuration of machines synchronized. Think of your virtual machine as disposable. You should be able to destroy and recreate it without losing functionality. Any configuration changes that need to be made to the machine should be propagated to the Puppet configuration file and distributed via a Git repository or a Puppet server (puppetmaster). Remember, the key is for everyone to have the same environment, and to do that, no custom modifications should be made to the virtual machine without sharing your changes in configuration via the Puppet file.

If you have modified base.pp, you can easily reprovision the machine as follows:

vagrant reload

You can also completely destroy the machine and begin anew:

vagrant destroy
vagrant up

If you just want to suspend your virtual machine for later:

vagrant suspend
vagrant resume

Once you have configured a box, you can also package it up and distribute a binary version using Vagrant’s package command. I prefer creating a Git repository with many folders containing different configurations. For example, you could have a stable LAMP environment in one folder, and another containing a script to install the new PHP 5.4 release candidate. Yet another one could contain a Ruby or Python environment. Switching between them is as simple as suspending one machine and booting up another.

Developing with a Consistent, Local Environment

Now you should be able to easily modify files within your www directory on your local machine, reload your browser at http://localhost:8080/, and see the results.

This configuration serves as a great development environment with which to begin. Each app will have its own set of dependencies. You can even distribute the configuration of the virtual machine within the app’s repository to serve as an explicit definition of the dependencies that it has. With this, you could configure your continuous integration server to deploy a machine, provision it, install the app, run its tests, and report back to you.

There is an ultimate level of configurability and flexibility with a setup like this. Above all, though, your team will now be able to work faster and more efficiently, and you won’t end up in a situation where something works differently in one environment from the next. I hope you find a way to use these tools for your own sanity.

Developer Gift

I can think of no better gift to give a developer than the Phantom II RC helicopter. When your codes get you down, and you just need to let off some steam, you can dive bomb your housemates or family members with this easily-manueverable craft. It’s reasonably durable and cheaper than other models — you can get two of them for $25.

Other posts