Pearls After Breakfast ~ Systems Orchestration with Ansible

Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away. ~ Antoine de Saint Exupéry

I've long been a fan and I've always been amazed at the skill of violinist Joshua Bell. I've seen him in concert a half-dozen times in the past 5 years, and in each performance he has succeeded in producing what I can only call "Joshua Bell Moments" - instances in time, perfect in themselves in which you can just watch, listen and marvel. Bell is a dynamic performer onstage, and it's remarkable to see musical precision generated so dynamically. If you're not familiar with Joshua Bell or his music, you should start with the Pulitzer Prize-winning introduction here: Pearls Before Breakfast

In Pearls Before Breakfast, Bell describes his skills as 'largely interpretive', and his ability to mesh with a symphony orchestra is a wonder to behold. Orchestration in any field is hard, and it's the holy grail of modern computer architectures. MapReduce might be elegant but Hadoop surely isn't - the wonder isn't that it works elegantly, but that it works at all. These days it's routine to require a dozen or dozens of systems to support modern applications, so when a system promises "Radically simple IT orchestration," I've got to take a look.

But is the problem really that bad? Can we just "do it by hand" or simply script these things? In his book, Taste Teste: Puppet - Chef - Salt - Ansible, Matt Jaynes writes

There were about 60 servers altogether and no two were exactly the same since they had all been set up by hand. Over time, the systems engineers would periodically update the servers, but they always seemed to miss some. Every server was a “beautiful snowflake” and unique from all the others.

The development environments were all slightly different too. So, an app release that worked on one developer’s environment, wouldn’t work on another developer’s environment. And an app release that worked on all the developers’ environments would have bugs in the staging environment. And an app release that worked in the staging environment would break in production. You get the idea.

If you've worked in cloud applications you certainly do get the idea. So what Ansible offers is a simple, YAML-based approach to managing the raft of servers that run most modern web applications. Does it work? Is it Radically Simple? Let's try it out and see what we find.

We'll be doing a lot of cloud-work on the new platform that I'll call "Cloudburst", so first we'll update the Linux base machine I'm running on. I'm on Ubuntu 13.04 "Raring Ringtail" for the usual reasons: not because it's necessarily a "better" distribution, but because there is practically limitless documentation on Ubuntu distributions online.

So first let's update Ringtail, and make sure we've got an up-to-date-enough Python version for Ansible:

$ sudo apt-get update
$ sudo apt-get upgrade
$ python --version
Python 2.7.4

Great so far. Ansible does it's work via SSH (not unlike the soon-to-be-written-about Github), so let's make sure our SSH keys are in place as well:

$ ssh-keygen -t rsa
$ cd ~/.ssh
$ cp id_rsa.pub authorized_keys

Ansible will look for its keys in the SSH directory file "authorized_keys", so once our copy is complete we're ready to install Ansible. Let's do that, then use ifconfig to identify the IP address of our target machine.

$ sudo apt-get install ansible
$ ifconfig
$ sudo vi /etc/ansible/hosts

    # Here's another example of host ranges, this time there are no
    # leading 0s:

    #db-[99:101]-node.example.com

    127.0.0.1

$ sudo apt-get install openssh-server

All set! Generally I'll use ifconfig got get the IP address for servers in my cloud stack, but we'll just use IP address of our base Ubuntu machine here. Now let's try a quick call-out from Ansible to our target machine to confirm that our installation is complete, and that all is well:

$ ansible all -m ping
127.0.0.1 | success >> {
    "changed": false, 
    "ping": "pong"
}

$ ansible all -a "/bin/echo hello Ansible"
127.0.0.1 | success | rc=0 >>
hello Ansible

GREAT! We're almost finished. To confirm Ansible (and have it do a little work for us), I'm going to create a simple YAML file for Ansible to confirm that the web server nginx is installed and the config file is in the right place. Our YAML file will just be a start, and even as an example it's great at showing how closely the YAML description matches our task list. So what we want to do is (in psuedocode):

• For all of our servers
  • Make sure https support is enabled
  • Make sure nginx is installed
  • Make sure the nginx.conf file is installed in the right place, and
  • Make sure nginx is running

So how much scripting and syntactic sugar do we need to get these tasks done? With Ansible, not much:

nginx.yml:

---
- hosts: all
  tasks:

  - name: Ensure https support for apt is installed
    apt: pkg=apt-transport-https state=present

  - name: Ensure nginx is installed
    apt: pkg=nginx-full state=present

  - name: Ensure the nginx configuration file is set
    copy: src=/app/config/nginx.conf dest=/etc/nginx/nginx.conf

  - name: Ensure nginx is running
    service: name=nginx state=started

Let's try it out, and see how our script works:

$ ansible-playbook nginx.yml

Success! This is just the beginning - from here we can use Ansible to

1. Make a standard server install definition (Note: Not a script!)
2. Apply that definition to all the servers in our stack
3. Use the definition for regular stack updates

Our stack in now just another variable in our solutions, and with the setup-magic taken away, we can focus on human interaction and delivery-magic! It might be too much to aspire to the performance of a Joshua Bell, but with Ansible we can set up and tune the whole backing orchestra correctly every time...