Category Archives: DevOps

Getting Started with Ansible, Part 0

Ansible is a popular open-source tool to automate common system administration tasks.

There’s a few other tools in the automation space; examples include Chef, Puppet, and SaltStack. These are called configuration management tools and they can be used to automate everything from mainframes to tiny IoT devices.

Ansible is arguably the most popular of these tools right now. There’s several reason for this. One is that it’s written in Python, a computer language exploding in popularity. Another is that the code you write to automate things (called “playbooks”) are written in a simple declarative file formatted in YAML. Even if you don’t know programming you can still get stuff done in Ansible. And finally, unlike most other configuration management tools, Ansible doesn’t require an always running agent; they just need a standard OpenSSH port open and accepting connections.

If you’ve ever worked in a large enterprise shop or had to deal with compliance issues, one less agent to manage on every node is a big deal.

For this simple tutorial I’m going to setup and run Ansible on my MacBook and the remote machines are six small Raspberry Pi single-board computers running Linux on my desk.

If you don’t have a Raspberry Pi or two to play with you can use another computer on your network, a cloud server you spin up just to play with, or even a Linux image running in Docker.

Ok, let’s get started on the MacBook.

First, let’s make a directory to hold our Ansible project:

[~/repo] $ mkdir pi_ansible
[~/repo] $ cd pi_ansible

We’re going to use a Python virtual environment to hold all the dependencies we need for Ansible. This creates the virtual environment in our Ansible project directory and sticks all the Python bits and packages inside the env directory:

[~/repo/pi_ansible] $ python3 -m venv env

Ok, now we need to activate the virtual environment. To do that just source the activate script that was created when the virtual environment was installed during the previous step:

[~/repo/pi_ansible] $ source env/bin/activate

Depending on your shell and setup, you’ll probably see your prompt to change to indicate you’ve activated the virtual environment. You can check it’s setup correctly by doing a which pip3 to make sure it’s using the one inside the virtual environment:

(env) [~/repo/pi_ansible] $ which pip3
/Users/mike/repo/pi_ansible/env/bin/pip3

Now we’re ready to install Ansible.

With Ansible version 2.10 there’s a slight change; previously you’d just install one Ansible package and all the things would get installed.

Now it’s split into two packages: ansible-base, which is the base that just includes the bits needed for Ansible to run, and ansible, which includes playbooks and modules for many common operations.

The reason for this change is many large sites have unique needs and don’t need all the playbooks and stuff. If you’re installing on thousands of servers all that extra stuff adds up.

For this example we’ll install both packages:

(env) [~/repo/pi_ansible] $ pip3 install ansible-base
Collecting ansible-base
Using cached ansible-base-2.10.2.tar.gz (6.0 MB)
Collecting jinja2
Using cached Jinja2-2.11.2-py2.py3-none-any.whl (125 kB)
Collecting PyYAML
...
(env) [~/repo/pi_ansible] $ pip3 install ansible
Collecting ansible
Using cached ansible-2.10.1.tar.gz (25.9 MB)
Requirement already satisfied: ansible-base<2.11,>=2.10.2 in ./env/lib/python3.9/site-packages (from ansible) (2.10.2)
...

Let’s check the version and make sure Ansible is installed:

(env) [~/repo/pi_ansible] $ ansible --version
ansible 2.10.2
config file = None
configured module search path = ['/Users/mike/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /Users/mike/repo/pi_ansible/env/lib/python3.9/site-packages/ansible
executable location = /Users/mike/repo/pi_ansible/env/bin/ansible
python version = 3.9.0 (v3.9.0:9cf6752276, Oct 5 2020, 11:29:23) [Clang 6.0 (clang-600.0.57)]

This is how the output looks on my Mac. If you’re on a different platform it might be different. Just make sure Ansible runs and returns a version number.

There’s a few more things to do before we take our first step.

Ansible works with an inventory. In large environments you’ll likely have this dynamic and pull information from some database that knows about your setup.

But for smaller environments you’ll just keep an inventory in a file in the Ansible directory. That’s what we’re going to do.

My Raspberry Pi cluster is connected to the network via WiFi and I don’t have them in DNS, so we’re just going to put them in a file by IP address. Let’s call this file hosts.txt and this is what it looks like:

[all]
192.168.1.138
192.168.1.148
192.168.1.117
192.168.1.101
192.168.1.126
192.168.1.114

The first line is a label we’ll just put all the systems under [all]. You could divide this up with labels such as [webservers] or [databases] but for now let’s keep it simple.

Next we need to tell Ansible where to look for the inventory. By default Ansible will look for the inventory in several default locations like /etc/ansible.cfg, but no one really uses it that way. It’s best to just put the inventory in the same project you use for the playbooks and other configuration files.

The primary configuration file is called ansible.cfg in the project directory. Let’s create it:

[defaults]
inventory = hosts.txt

Let’s run ansible in AdHoc mode with the ping module and see how it works.

(env) [~/repo/pi_ansible] $ ansible all -m ping
192.168.1.148 | UNREACHABLE! => {
"changed": false,
"msg": "Failed to connect to the host via ssh: mike@192.168.1.148: Permission denied (publickey,password).",
"unreachable": true
}
192.168.1.138 | UNREACHABLE! => {
"changed": false,
"msg": "Failed to connect to the host via ssh: mike@192.168.1.138: Permission denied (publickey,password).",
"unreachable": true
}
192.168.1.101 | UNREACHABLE! => {
"changed": false,
"msg": "Failed to connect to the host via ssh: mike@192.168.1.101: Permission denied (publickey,password).",
"unreachable": true
}
192.168.1.117 | UNREACHABLE! => {
"changed": false,
"msg": "Failed to connect to the host via ssh: mike@192.168.1.117: Permission denied (publickey,password).",
"unreachable": true
}
192.168.1.126 | UNREACHABLE! => {
"changed": false,
"msg": "Failed to connect to the host via ssh: mike@192.168.1.126: Permission denied (publickey,password).",
"unreachable": true
}
192.168.1.114 | UNREACHABLE! => {
"changed": false,
"msg": "Failed to connect to the host via ssh: mike@192.168.1.114: Permission denied (publickey,password).",
"unreachable": true
}

That’s not right. If you look at the output you can see ansible trying to login to the servers with my current MacBook user, but I don’t have that account on the Raspberry Pi systems.

We need to add a remote_user configuration to ansible.cfg and set it to the pi user:

[defaults]
inventory = hosts.txt
remote_user = pi

Let’s try it again:

(env) [~/repo/pi_ansible] $ ansible all -m ping
192.168.1.138 | UNREACHABLE! => {
"changed": false,
"msg": "Failed to connect to the host via ssh: pi@192.168.1.138: Permission denied (publickey,password).",
"unreachable": true
}
192.168.1.148 | UNREACHABLE! => {
"changed": false,
"msg": "Failed to connect to the host via ssh: pi@192.168.1.148: Permission denied (publickey,password).",
"unreachable": true
}
192.168.1.117 | UNREACHABLE! => {
"changed": false,
"msg": "Failed to connect to the host via ssh: pi@192.168.1.117: Permission denied (publickey,password).",
"unreachable": true
}
192.168.1.126 | UNREACHABLE! => {
"changed": false,
"msg": "Failed to connect to the host via ssh: pi@192.168.1.126: Permission denied (publickey,password).",
"unreachable": true
}
192.168.1.101 | UNREACHABLE! => {
"changed": false,
"msg": "Failed to connect to the host via ssh: pi@192.168.1.101: Permission denied (publickey,password).",
"unreachable": true
}
192.168.1.114 | UNREACHABLE! => {
"changed": false,
"msg": "Failed to connect to the host via ssh: pi@192.168.1.114: Permission denied (publickey,password).",
"unreachable": true
}

Ansible is trying to login with the remote user now, but it’s still not working.

The problem is that the pi user is setup with a password. That’s not going to work with automation.

What we need to do is copy our ssh public_key to each of these servers so ansible can login without having to use the password.

Note: this assumes you have a ssh key generated on your system. If you need to do that, go here to learn more: https://git-scm.com/book/en/v2/Git-on-the-Server-Generating-Your-SSH-Public-Key

I’m going to use a little shell magic to loop though the inventory and copy the ssh keys to the remote systems.

(env) [~/repo/pi_ansible] $ for x in $(awk NR-1 hosts.txt); do ssh-copy-id pi@$x; done
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/Users/mike/.ssh/id_ed25519.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
pi@192.168.1.138's password: 

Number of key(s) added: 1

Now try logging into the machine, with: "ssh 'pi@192.168.1.138'"
and check to make sure that only the key(s) you wanted were added.

/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/Users/mike/.ssh/id_ed25519.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
pi@192.168.1.148's password: 

Number of key(s) added: 1

Now try logging into the machine, with: "ssh 'pi@192.168.1.148'"
and check to make sure that only the key(s) you wanted were added.

/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/Users/mike/.ssh/id_ed25519.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
pi@192.168.1.117's password: 

Number of key(s) added: 1

Now try logging into the machine, with: "ssh 'pi@192.168.1.117'"
and check to make sure that only the key(s) you wanted were added.

/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/Users/mike/.ssh/id_ed25519.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
pi@192.168.1.101's password: 

Number of key(s) added: 1

Now try logging into the machine, with: "ssh 'pi@192.168.1.101'"
and check to make sure that only the key(s) you wanted were added.

/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/Users/mike/.ssh/id_ed25519.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
pi@192.168.1.126's password: 

Number of key(s) added: 1

Now try logging into the machine, with: "ssh 'pi@192.168.1.126'"
and check to make sure that only the key(s) you wanted were added.

/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/Users/mike/.ssh/id_ed25519.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
pi@192.168.1.114's password: 

Number of key(s) added: 1

Now try logging into the machine, with: "ssh 'pi@192.168.1.114'"
and check to make sure that only the key(s) you wanted were added.

The shell line reads the IP addresses from hosts.txt (the awk thing is to skip the first line, which is a label), and runs the ssh-copy-id command which copies the ssh public key to a remote machine. You still need to enter your password on each machine, but this will be the last time.

Ok, one more time:

(env) [~/repo/pi_ansible] $ ansible all -m ping
[DEPRECATION WARNING]: Distribution debian 10.4 on host 192.168.1.101 should use /usr/bin/python3, but is
using /usr/bin/python for backward compatibility with prior Ansible releases. A future Ansible release will
default to using the discovered platform python for this host. See
https://docs.ansible.com/ansible/2.10/reference_appendices/interpreter_discovery.html for more information.
This feature will be removed in version 2.12. Deprecation warnings can be disabled by setting
deprecation_warnings=False in ansible.cfg.
192.168.1.101 | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/bin/python"
},
"changed": false,
"ping": "pong"
}
[DEPRECATION WARNING]: Distribution debian 10.4 on host 192.168.1.126 should use /usr/bin/python3, but is
using /usr/bin/python for backward compatibility with prior Ansible releases. A future Ansible release will
default to using the discovered platform python for this host. See
https://docs.ansible.com/ansible/2.10/reference_appendices/interpreter_discovery.html for more information.
This feature will be removed in version 2.12. Deprecation warnings can be disabled by setting
deprecation_warnings=False in ansible.cfg.
192.168.1.126 | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/bin/python"
},
"changed": false,
"ping": "pong"
}
[DEPRECATION WARNING]: Distribution debian 10.4 on host 192.168.1.117 should use /usr/bin/python3, but is
using /usr/bin/python for backward compatibility with prior Ansible releases. A future Ansible release will
default to using the discovered platform python for this host. See
https://docs.ansible.com/ansible/2.10/reference_appendices/interpreter_discovery.html for more information.
This feature will be removed in version 2.12. Deprecation warnings can be disabled by setting
deprecation_warnings=False in ansible.cfg.
192.168.1.117 | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/bin/python"
},
"changed": false,
"ping": "pong"
}
[DEPRECATION WARNING]: Distribution debian 10.6 on host 192.168.1.138 should use /usr/bin/python3, but is
using /usr/bin/python for backward compatibility with prior Ansible releases. A future Ansible release will
default to using the discovered platform python for this host. See
https://docs.ansible.com/ansible/2.10/reference_appendices/interpreter_discovery.html for more information.
This feature will be removed in version 2.12. Deprecation warnings can be disabled by setting
deprecation_warnings=False in ansible.cfg.
192.168.1.138 | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/bin/python"
},
"changed": false,
"ping": "pong"
}
[DEPRECATION WARNING]: Distribution debian 10.4 on host 192.168.1.148 should use /usr/bin/python3, but is
using /usr/bin/python for backward compatibility with prior Ansible releases. A future Ansible release will
default to using the discovered platform python for this host. See
https://docs.ansible.com/ansible/2.10/reference_appendices/interpreter_discovery.html for more information.
This feature will be removed in version 2.12. Deprecation warnings can be disabled by setting
deprecation_warnings=False in ansible.cfg.
192.168.1.148 | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/bin/python"
},
"changed": false,
"ping": "pong"
}
[DEPRECATION WARNING]: Distribution debian 10.4 on host 192.168.1.114 should use /usr/bin/python3, but is
using /usr/bin/python for backward compatibility with prior Ansible releases. A future Ansible release will
default to using the discovered platform python for this host. See
https://docs.ansible.com/ansible/2.10/reference_appendices/interpreter_discovery.html for more information.
This feature will be removed in version 2.12. Deprecation warnings can be disabled by setting
deprecation_warnings=False in ansible.cfg.
192.168.1.114 | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/bin/python"
},
"changed": false,
"ping": "pong"
}

It’s working! But what’s with that huge [DEPRECATION WARNING]?

This is saying Ansible found both Python2 and Python3 on the remote servers and it’s using Python2. But that’s going to change in a future release and anything that depends on Python2 is going to break.

This doesn’t apply to us. Let’s enable the future option and use Python3 by adding one more line to ansible.cfg:

[defaults]
inventory = hosts.txt
remote_user = pi
interpreter_python = auto

Let’s run it one last time:

(env) [~/repo/pi_ansible] $ ansible all -m ping

192.168.1.138 | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/bin/python3"
},
"changed": false,
"ping": "pong"
}
192.168.1.126 | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/bin/python3"
},
"changed": false,
"ping": "pong"
}
192.168.1.148 | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/bin/python3"
},
"changed": false,
"ping": "pong"
}
192.168.1.117 | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/bin/python3"
},
"changed": false,
"ping": "pong"
}
192.168.1.101 | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/bin/python3"
},
"changed": false,
"ping": "pong"
}
192.168.1.114 | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/bin/python3"
},
"changed": false,
"ping": "pong"
}

All right! Ansible is now working and reporting SUCCESS with the ping module.

We got Ansible installed and configured at a basic level and you’ve seen a successful run along with a few failures.

In a future post we’ll talk about what modules are, what that fact thing being reported is all about, and how to do some actual automation.

Installing Docker on a Raspberry Pi

The Raspberry Pi is a fantastic low-cost way to experiment with Docker and Kubernetes.

There’s several ways to get the latest version of Docker installed on a Raspberry Pi; you can go with the Cypriot project or a full-fledged GUI-based Raspbian installation.

In my case I have a cluster of six Pi I need to configure for a Kubernetes install and I want the latest “Buster” release so I’m going with Raspbian Buster Lite.

After downloading the ISO image you need to burn it to the MicroSD card. On my modern Apple MacBook Pro this isn’t as easy as it used to be since there’s on SD card slot and all you get is USB-C ports. So, yeah, dongle life it is.

I used this Uni brand card adapter and it works perfectly.

There’s all kinds of ways to burn the Raspbian ISO file to the MicroSD card, but I prefer the command-line way.

First, insert the MicroSD card into the reader and do this to see which “disk” device it’s assigned to:

mike@jurassic ~ % diskutil list
/dev/disk0 (internal, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *500.3 GB   disk0
   1:                        EFI EFI                     314.6 MB   disk0s1
   2:                 Apple_APFS Container disk1         500.0 GB   disk0s2

/dev/disk1 (synthesized):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      APFS Container Scheme -                      +500.0 GB   disk1
                                 Physical Store disk0s2
   1:                APFS Volume Jurassic - Data         161.0 GB   disk1s1
   2:                APFS Volume Preboot                 87.4 MB    disk1s2
   3:                APFS Volume Recovery                528.5 MB   disk1s3
   4:                APFS Volume VM                      1.1 GB     disk1s4
   5:                APFS Volume Jurassic                10.8 GB    disk1s5

/dev/disk2 (external, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:     FDisk_partition_scheme                        *63.9 GB    disk2
   1:             Windows_FAT_32 boot                    268.4 MB   disk2s1
   2:                      Linux                         63.6 GB    disk2s2

mike@jurassic ~ % 

As you can see, in this case the 64GB MicroSD card I’m using is mounted as /dev/disk2. To write to it we need to unmount it. This is different than “ejecting”; we need to umount it but still leave it attached to the system so we can write to it.

mike@jurassic ~ % diskutil umountDisk /dev/disk2
Unmount of all volumes on disk2 was successful
mike@jurassic ~ % 

Now we’re ready to write the downloaded Raspbian image to the MicroSD card.

sudo dd bs=1m if=/Users/mike/Downloads/2019-09-26-raspbian-buster-lite.img of=/dev/rdisk2 conv=sync

This will take awhile to run and after it completes the image will automatically mount to the system. Leave it this way for the next step.

I run my Raspberry Pi cluster headless, so instead of hooking up a monitor and keyboard to each Pi to configure it there’s several features you can enable on the newly imaged MicroSD card to streamline it’s headless setup.

First we need to enable ssh so we can login remotely. You can do this by creating an empty file named ‘ssh’ on the /boot partition.

Next we need to configure networking. I network my cluster via Wifi so we can create a file called wpa_supplicant.conf also on the /boot filesystem and when the Pi boots it’ll copy this file and it’s contents to the correct location and fire up the network.

mike@jurassic ~ % cd /Volumes/boot 
mike@jurassic boot % touch ssh
mike@jurassic boot % touch wpa_supplicant.conf
mike@jurassic boot % vi wpa_supplicant.conf 

Here’s the contents I use in wpa_supplicant.conf to attach to my wifi. Change the attributes to match your wifi settings.

ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1
country=US

network={
    ssid="Cenozoic"
    key_mgmt=WPA-PSK
    psk="trilobite"
}

Ok, now we’re ready to eject the MicroSD card from the Mac and put it in the Pi and power it up.

mike@jurassic ~ % diskutil eject /dev/disk2
Disk /dev/disk2 ejected

After you power up the Pi you should be able to login to it via ssh. Of course, you’ll need to find out it’s IP address. There’s several ways you can do this with a wifi headless setting. You can just check your routers MAC table or DHCP settings if you have access. Or yu can use nmap to scan your network and find a list of devices with active ssh ports.

Once you figure that out and ssh in you’re ready to configure your Pi with Docker.

Login as the ‘pi’ user and change the default password to something more secure. I usually like update the OS, too.

passwd
sudo apt update 
sudo apt full-upgrade -y
sudo reboot

After the reboot with an updated system it’s time to get docker installed.

First, we need to install some packages we need to install docker:

sudo apt-get install apt-transport-https ca-certificates software-properties-common -y

Next we need to download and install the docker gpg key:

sudo wget -qO - https://download.docker.com/linux/raspbian/gpg | sudo apt-key add -

We need to setup the apt channel to fetch the docker pacakges:

sudo echo "deb https://download.docker.com/linux/raspbian/ buster stable" | sudo tee /etc/apt/sources.list.d/docker.list

Now, we’re ready to install Docker. The --no-install-recommends is to work around an issue with aufs-dkms you might run into. It’s not needed for the latest versions of Docker so we can skip it and avoid the issue.

sudo apt-get update
sudo apt-get install docker-ce --no-install-recommends -y

The last step for setup is to enable the pi user to run docker commands:

sudo usermod -aG docker pi

Logout of the pi account and back in and you should be good to go. Run docker info to check that Docker is running and is the latest version.

Some vi tips

Awhile back at a previous company I got hit with a last-minute request to update our Nginx redirect map with data provided in a spreadsheet. Normally I only have to do one or two redirect rules at a time. But today I got hit with 120 rules – each of which needed data from two cells in the spreadsheet. Doing it manually would require 240 cut-n-paste operations – not fun and error prone. Oh, and I only had 30-minutes to get this done and up in production.

We do these sorts of redirects when a customer changes their domain name. In this particular case the customer had a number of locations under one domain and they’re now splitting each location out into its own domain. But we don’t want visitors to the sites getting 404s due to the URL changing so we put in redirect rules to rewrite the request and forward the visitor to the new domain.

There’s a bunch of ways to do this, but this is how I did it.

First, a quick explanation of the Nginx redirect map format. There’s not much to it:

old.example.com new.example.com;

That’s the old URL, a space or spaces (or tab), the new URL and a semicolon terminating the line.

In this case, each row of the spreadsheet with the redirect data had 7 columns: the name of the property being redirected, and three URLs that needed to be redirected with the destination (i.e. Each location in this case only has three pages so there’s only three redirects each). Luckily the order of he data is just what I needed for the map.

The first thing I did was a CSV export of the data and opened it up in vi.

Some vi tips 1

The CSV export contained title information for the columns I don’t need so let’s just delete those right off.

Some vi tips 2

Ok, now the data is ready to be processed into something I can use. In another stroke of luck we can see each redirect pair is separated by two commas since the spreadsheet contained an empty column between the three pairs for each location. This will make things much easier.

Some vi tips 3

First, let’s get each redirect on its own line in the file. We can search and replace for the domain name to be redirected since they’re all the same. We’ll search for the domain and replace it as the first thing on its own line with:

:%s:www.myfavoriteapartment.com:\rwww.myfavoriteapartment.com:g

The % does all lines in the file; the s is for search; the \r is vi-speak for newline; and the g at the end is for global so it’ll process the whole line rather than the first match it finds on that line. By habit I use : for the separator in the search and replace command; you can also (and most people do) use /.

Some vi tips 4

Ok, we’re getting there. Next let’s get rid of the double commas and – since that’s always at the end of the redirect destination – put a semicolon at the end as Nginx requires:

:%s:,,:;:g

Some vi tips 5

Now we need to deal with the first column of data from the spreadsheet – the name of the location. We don’t need this information for the map file so let’s just delete it. Each location name is on its own line at this point and ends with a comma. So let’s search for lines ending with a comma and delete ‘em:

:g:,$:d

In this case, the g is for a global operation on all lines; the ,$ matches all lines with a , at the end of a line (the $); and the d is for delete.

Some vi tips 6

All we have left is to replace the single remaining comma on each line that separates the source and destination URLs. Nginx requires that the separation here be one or more spaces. I typically use a tab (although I probably shouldn’t – it makes the map file look messy) so let’s do this:

:%s:,:^I:g

The ^I is the tab character. Nowadays you can usually just press the tab key and vi will put in the ^I, but in the old days you had to do control-v and control-i to get a tab.

Some vi tips 7

And that’s it.

Some vi tips 8

Now all we got to do is save the file and do a copy and paste of its contents into the map file on our production configuration. Of course, you’ll want to scroll through the file and make sure it looks correct and do a nginx configtest before nginx reload to make sure it’s valid.

Configuring PTR records using the Rackspace Cloud DNS API

This is an archive originally posted on the Rackspace Developer Blog.

 

This is a guest post by Mike Sisk, a Racker working on DevOps for Rackspace’s new cloud products. You can read his blog at http://mikesisk.com or follow @msisk on Twitter.

I’m going to show you how to setup a PTR record in Rackspace Cloud DNS for a Cloud Load Balancer using the command-line utility cURL.

I’m using a Rackspace Cloud production account below; when you try it with your account change the account number and authentication token to yours.

A few notes about cURL: this is a command line utility found on Mac and most Linux machines. You can also use other tools called “REST Clients”. One popular one is an extension for Firefox and Chrome located here.

Here’s a quick reference to some of the options I’ll be sending with cURL in the examples below:

  • -D

    Dump header to file (- sends to stdout)

  • -H

    Extra header(s) to send

  • -I

    Get header only

  • -s

    silent (no progress bars or errors)

  • -X

    Request method (GET is default)

The first thing we need to do is get an Authentication Token:

$ curl -s -I -H "X-Auth-Key: auth-key-for-account" -H "X-Auth-User: username" https://auth.api.rackspacecloud.com/v1.0

This will return just the header from the request (note the -I option) and the one we need is the value after “X-Auth-Token:”

Let’s list the domain so we can grab its ID (I’ve obscured my Authentication Token below with $TOKEN):

$ curl -s -H "X-Auth-Token: $TOKEN" https://dns.api.rackspacecloud.com/v1.0/636983/domains {"domains":[{"name":"sisk.ws","id":3325158,"created":"2012-07-20T18:04:11.000 0000","accountId":636983,"updated":"2012-07-31T18:40:22.000 0000","emailAddress":"mike.sisk@rackspace.com"},{"name":"wikirr.com","id":3174423,"created":"2012-02-28T19:26:35.000 0000","accountId":636983,"updated":"2012-07-21T21:58:41.000 0000","emailAddress":"ipadmin@stabletransit.com"},{"name":"wikirr.net","id":3314413,"comment":"This is a test domain created via the Cloud DNS API","created":"2012-07-11T14:54:00.000 0000","accountId":636983,"updated":"2012-07-11T14:54:03.000 0000","emailAddress":"mike.sisk@rackspace.com"}],"totalEntries":3}

Ok, that’s a little hard to read. But I can see the domain ID in question I need.

$ curl -s -H "X-Auth-Token: $TOKEN" https://dns.api.rackspacecloud.com/v1.0/636983/domains/3325158 | python -m json.tool
{
 "accountId": 636983,
 "created": "2012-07-20T18:04:11.000 0000",
 "emailAddress": "mike.sisk@rackspace.com",
 "id": 3325158,
 "name": "sisk.ws",
 "nameservers": [],
 "recordsList": {
 "records": [],
 "totalEntries": 4
 },
 "ttl": 300,
 "updated": "2012-07-31T18:40:22.000 0000"
 }

The default output of the API is JSON, but it also supports XML if you send it another header. The little bit of Python on the end just sends the output though a parser to format it for humans. You can see I have already created an A record for the www address. The PTR we’re creating below is essentially the reverse of that to map the IP address to http://www.sisk.ws.

Through the control panel I had already created a Cloud Load Balancer on this account. We need its ID for the PTR record, so let’s list the load balancer though its API:

 $ curl -s -H "Accept: application/json" -H "X-Auth-Token: $TOKEN" https://dfw.loadbalancers.api.rackspacecloud.com/v1.0/636983/loadbalancers/47219 | python -m json.tool
{
 "loadBalancer": {
 "algorithm": "LEAST_CONNECTIONS",
 "cluster": {
 "name": "ztm-n09.dfw1.lbaas.rackspace.net"
 },
 "connectionLogging": {
 "enabled": false
 },
 "contentCaching": {
 "enabled": false
 },
 "created": {
 "time": "2012-07-31T18:36:34Z"
 },
 "id": 47219,
 "name": "Test1",
 "nodes": [],
 "port": 80,
 "protocol": "HTTP",
 "sourceAddresses": {
 "ipv4Public": "64.49.225.5",
 "ipv4Servicenet": "10.183.248.5",
 "ipv6Public": "2001:4800:7901::9/64"
 },
 "status": "ACTIVE",
 "updated": {
 "time": "2012-07-31T18:36:44Z"
 },
 "virtualIps": []
 }
 }

Ok, that returns a lot of stuff. The thing we need from this output is the ID, 114445 and the ipv4 Public IP, 64.49.225.5.

Now that we’ve collected all the required information, the next step is calling the Cloud DNS API with a POST operation to create the PTR record and associate it with the Load Balancer. First thing I did was create the following JSON data in a text editor:

{
"recordsList" : {
"records" : [ {
"name" : "www.sisk.ws",
"type" : "PTR",
"data" : "66.216.68.19",
"ttl" : 56000
}, {
"name" : "www.sisk.ws",
"type" : "PTR",
"data" : "2001:4800:7901:0000:290c:0b6b:0000:0001",
"ttl" : 56000
} ]
},
"link" : {
"content" : "",
"href" : "https://dfw.loadbalancers.api.rackspacecloud.com/v1.0/636983/loadbalancers/47219",
"rel" : "cloudLoadBalancers"
}
}

This lists all the data we need to create the PTR record. In this example I also added the IPV6 address, too. Let’s give it a try:

$ curl -D – -X POST -d '{
"recordsList" : {
"records" : [ {
"name" : "www.sisk.ws",
"type" : "PTR",
"data" : "66.216.68.19",
"ttl" : 56000
}, {
"name" : "www.sisk.ws",
"type" : "PTR",
"data" : "2001:4800:7901:0000:290c:0b6b:0000:0001",
"ttl" : 56000
} ]
},
"link" : {
"content" : "",
"href" : "https://dfw.loadbalancers.api.rackspacecloud.com/v1.0/636983/loadbalancers/47219",
"rel" : "cloudLoadBalancers"
}
}' -H "Content-Type: application/json" -H "X-Auth-Token: $TOKEN" https://dns.api.rackspacecloud.com/v1.0/636983/rdns

I just typed in the commands and pasted in the above JSON file between the single-quote marks after the -d. In the Cloud DNS API commands that create stuff are asynchronous — what we get back is a URL to check to see the status of the job.

This is what we get back from the above POST:

{"request":"{\n "recordsList" : {\n "records" : [ {\n "name" : "www.sisk.ws",\n "type" : "PTR",\n "data" : "66.216.68.19",\n "ttl" : 56000\n }, {\n "name" : "www.sisk.ws",\n "type" : "PTR",\n "data" : "2001:4800:7901:0000:290c:0b6b:0000:0001",\n "ttl" : 56000\n } ]\n },\n "link" : {\n "content" : "",\n "href" : "https://dfw.loadbalancers.api.rackspacecloud.com/v1.0/636983/loadbalancers/47219",\n "rel" : "cloudLoadBalancers"\n }\n}","status":"INITIALIZED","verb":"POST","jobId":"f99c1203-b42a-44c4-885f-c80bd7e3aba0","callbackUrl":"https://dns.api.rackspacecloud.com/v1.0/636983/status/f99c1203-b42a-44c4-885f-c80bd7e3aba0","requestUrl":"http://dns.api.rackspacecloud.com/v1.0/636983/rdns"}

Let’s check the job status:

$ curl -s -H "X-Auth-Token: $TOKEN" https://dns.api.rackspacecloud.com/v1.0/636983/status/f99c1203-b42a-44c4-885f-c80bd7e3aba0 | python -m json.tool
 {
 "callbackUrl": "https://dns.api.rackspacecloud.com/v1.0/636983/status/f99c1203-b42a-44c4-885f-c80bd7e3aba0",
 "jobId": "f99c1203-b42a-44c4-885f-c80bd7e3aba0",
 "status": "COMPLETED"
 }

These jobs complete quickly, so by the time you check it’s usually done. If the system is under a lot of load, or it’s a really big job (like updating hundreds of records) it might take a few minutes.

Ok, let’s see what the PTR record looks like. First, keep in mind the PTR record lives with the device, not the domain record. So doing a list of the domain won’t show us the PTR. We have to list it from the device like this:

$ curl -s -H "X-Auth-Token: $TOKEN" https://dns.api.rackspacecloud.com/v1.0/636983/rdns/cloudLoadBalancers?href=https://dfw.loadbalancers.api.rackspacecloud.com/v1.0/636983/loadbalancers/47219 | python -m json.tool
 {
 "records": []
 }

Ok, looks good. Let’s test it:

$ host www.sisk.ws
www.sisk.ws has address 66.216.68.19
$ host 66.216.68.19
19.68.216.66.in-addr.arpa domain name pointer www.sisk.ws.

Looks good — that’s what we expect to see. To compare, let’s see what it looks like with the root of the domain, which is pointing to a first generation cloud server that doesn’t support PTR records:

$ host sisk.ws
sisk.ws has address 108.166.119.217
$ host 108.166.119.217
217.119.166.108.in-addr.arpa domain name pointer 108-166-119-217.static.cloud-ips.com.

For more information on the Rackspace Cloud DNS API, refer to our API documentation.

s3cmd with Multiple AWS Accounts

Awhile back I was doing a lot of work involving Amazon’s Simple Storage Service (aka Amazon S3).

And while tools like Panic’s Transmit, the Firefox S3Fox extension, or even Amazon’s own S3 Management Console make it easy to use, sometimes you really just want a command-line tool.

There’s a lot of good tools out there, but the one I’ve been using is s3cmd. This tool is written in Python and is well documented. Installation on Linux or OS X is simple as is its configuration. And as a longtime Unix command-line user it’s syntax is simple. Some examples:

To list your buckets:

~ $ s3cmd ls
2010-04-28 23:50 s3://g5-images
2011-01-21 06:42 s3://g5-mongodb-backup
2011-03-21 21:23 s3://g5-mysql-backup
2010-06-03 17:45 s3://g5-west-images
2010-09-02 15:57 s3://g5engineering

List the size of a bucket with “human readable” units:

~ $ s3cmd du -H s3://g5-mongodb-backup
1132G s3://g5-mongodb-backup/

List the contents of a bucket:

~ $ s3cmd ls s3://g5-mongodb-backup
2011-08-08 14:43 3273232889 s3://g5-mongodb-backup/mongodb.2011-08-08-06.tar.gz
2011-08-08 21:12 3290592536 s3://g5-mongodb-backup/mongodb.2011-08-08-12.tar.gz
2011-08-09 03:16 3302734859 s3://g5-mongodb-backup/mongodb.2011-08-08-18.tar.gz
2011-08-09 09:09 3308369423 s3://g5-mongodb-backup/mongodb.2011-08-09-00.tar.gz
2011-08-09 14:51 3285753739 s3://g5-mongodb-backup/mongodb.2011-08-09-06.tar.gz

Show the MD5 hash of an asset:

~ $ s3cmd ls --list-md5 s3://g5-mongodb-backup/mongodb.2011-08-09-06.tar.gz
2011-08-09 14:51 3285753739 07747e3de16138799d9fe1846436a3ce \
s3://g5-mongodb-backup/mongodb.2011-08-09-06.tar.gz

Transferring a file to a bucket uses the get and put commands. And if you
forget an option or need a reminder of usage the very complete s3cmd –help
output will likely be all the help you need.

One problem I have with most tools for AWS is managing multiple accounts. Most
of these tools assume you have just one account, but I work with multiple
accounts and switching between them can be cumbersome.

Here’s how I work with multiple AWS accounts using s3cmd.

By default s3cmd puts its configuration file in ~/.s3cfg, but you can
override this and specify a configuration file with the -c option.

What I do is create a separate config file with the appropriate credentials for
each account I work with and give them unique names:

~ $ ls -1 .s3cfg*
.s3cfg-g5
.s3cfg-tcp

Another option is to keep the credentials for the account you use most often in
the standard ~/.s3cfg file and use the -c option when/if you need another
account. I don’t like this option because it’s too easy to mistakenly use the
wrong account. For example, without a ~/.s3cfg this is what happens when I use
s3cmd without specifying a configuration:

~ $ s3cmd ls
ERROR: /Users/mike/.s3cfg: No such file or directory
ERROR: Configuration file not available.
ERROR: Consider using --configure parameter to create one.

So, what to do? Using the -c all the time is a PITA. Answer: use Bash aliases!

Here’s a subset of the s3cmd aliases I have in my ~/.bashrc file:

# s3cmd aliases for different s3 accounts
alias s3g5='s3cmd -c ~/.s3cfg-g5'
alias s3tcp='s3cmd -c ~/.s3cfg-tcp'

Now, to list the buckets in my personal account I just do:

~ $ s3tcp ls
2011-07-01 06:10 s3://mikesisk-img
2011-07-05 23:16 s3://www.tcpipranch.com
2011-07-01 22:55 s3://www.watch4rocks.com

And I can still pass arguments:

~ $ s3tcp -H --list-md5 ls s3://mikesisk-img/me.jpg
2011-07-01 06:09 5k 13d7c86bccd8915dd93b085985305394 \
s3://mikesisk-img/me.jpg

Just keep in mind that calls to bash aliases from scripts and cronjobs might not work. Plus it’s bad form and will come back to bite you one of these days. Just use the long form with -c in these places and keep the aliases for your own interactive command-line usage.

Cron and Sewing Needles

Sometimes, even after decades of experience, you still screw up.

Consider this cron entry I put in last night:

    # Backup MongoDB every 6 hours, zip it up, and rsync it.
    * */6 * * * ~/bin/backup_mongo.sh

I wanted this to run the backup script for MongoDB every six hours. Instead, I got it running every
minute for an hour every six hours. You’d think I’d know better considering I put the next cron in correctly:

    # Remove MongoDB backups that are more than 24-hours old.
    00 02 * * * find /db/backup -mtime +1 -exec rm -f {} \;

What I meant to do is this:

    # Backup MongoDB every 6 hours, zip it up, and rsync it.
    00 */6 * * * ~/bin/backup_mongo.sh

Luckily we host our infrastructure at Engine Yard and their staff noticed the CPU spike on this server at midnight and fixed the cron.

Which brings up another point: name your scripts appropriately. In this case a quick scan of cron revealed this script was running a backup and doing that every six hours makes sense. If the script was just named mongo, it’s conceivable it could have been a metric collection script that runs every minute for an hour every six hours.

So what do sewing needles have to do with cron? I’m working from home this week and had just finished that MongoDB backup script and was putting it in cron when my daughter came running (Ok, make that hopping) into my office with a large sewing needle in-bedded about 1/4″ deep in the arch of her foot. I quickly saved the cron entry to take care of that problem and didn’t go back to check my work.

Moral of the story: whenever you set up a new cron job it’s a good idea to watch it run and see if it’s doing what you think it is. Especially if you think you know what you’re doing.