Mike Sisk

Today Is My Birthday and I’m a Perfect Square.

Last year I wrote about my being 30 in hexadecimal on my birthday.

This year I’m a perfect square!

In math a perfect square is the product of an integer with itself. For example, 49 is a perfect square since it can be written as 7 x 7.

My age today.

atom

Boom!

This past Friday night a remarkably intense thunderstorm rolled though the Austin area. Around midnight my wife and I were abruptly woken up by a very close lightning strike.

Instead of the soothing rumble of distant thunder, when you’re close enough it’s a blinding flash and an immediate BOOM! like that of a point-blank shotgun.

It took me a minute to realize that the power in our bedroom had gone out. Our son, who was online playing a game with friends, reported the internet connection was down, too.

The power outage was traced to the arc-fault interrupter. The internet issue was a bit more than that.

So far we have a dead cable modem, a non-repsonsive Wi-Fi base station, and one damaged Linux server.

Luckily I had a spare DOCSIS 3.0 cable modem that supports our 100 Mbit connection and a spare Wi-Fi base station.

I have a spare server in the garage, too, but the 4TB of data and the hardware entropy source on this server are a bit harder to replace. Not to mention all the time to get everything set back up.

All this stuff was plugged into uninterruptible power supplies, too — the EMP from a close-by lightning strike is a bit unpredictable. I guessed a pulse came in through the cable coax and jumped to the other equipment.

I took the failed cable modem apart and expected to either find a blown protection fuse or diode in the power supply or a shorted out transceiver in the coax side.

What I found instead was a blown Marvell Alaska 88E111 ethernet transceiver chip. In other words the pulse from the lightning strike didn’t come in from the coax or power, but either induced in the ethernet cable itself or from something else on the network.

That’s the 2nd time in 10 years I’ve lost equipment from a close lightning strike involving an ethernet cable EMP. I need to start investing in shielded ethernet cables.

I got the damaged Linux server to boot but can’t get it on the network — it’s built-in ethernet is probably blown out, too. I guess I’ll have to dig out the spare and transfer the disk array.

I’m still not sure of the actual ground-zero of the strike; our house is one of the higher ones in our subdivision so it could have been any number of power poles or street lights around us. I suspect it’s the street light in front of our house, though — since Friday night it’s been stuck on running all day.

atom

Today Is My Birthday and I’m 30.

In hexadecimal.

Hexadecimal — or base-16 — turns out to be surprisingly useful in computers.

Modern computers work on boolean algebra. No matter how complex your software gets, down inside the machine everything basically boils down to true or false operations.

A bit of data containing a true or false is, well, a bit. Four of these bits make up a nibble, and eight of ‘em are a byte. And a byte is conveniently represented by two hexadecimal digits. Got that?

Ok, look at it this way:

In decimal (or base-10) we have ten possible digits: 0, 1, 2, 3, 4, 5, 6, 7, 8, and 9. This is the system we use every day.

By the way, this is called the Hindu-Arabic numeral system and dates back to 8th century Islamic mathematics. So you can thank all those folks for building the foundations of modern society.

Computers at their core use base-2; that’s just 0 and 1. That’s fine for computers, but people have a hard time with numbers consisting of just two digits. Binary 101010 is hard for most folks to parse as the value 42 in base-10.

So other base systems that are compatible with base-2 are used in computing. There’s octal which is base-8, but hexadecimal base-16 is far more common nowadays.

What do the digits in hexadecimal look like? The 16 digits are this: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, A, B, C, D, E, and F. The first 10 digits are easy, but how do you represent the base-10 value of 10 in hexadecimal? A. 15 in base-10 is F in hexadecimal. And how about 16? That’s 10 in hexadecimal.

Ok, going forward I’ll refer to base-10 numbers in the normal way and hexadecimal numbers will start with 0x.

A byte of 8 bits is used all the time in computing. A bit can represent 2 values but string 8 of them together and you get 256 possible values. That’s 28. In hexadecimal you can represent any value from 0 to 255 with just 2 digits. So, 5 is 0×05 (remember, the 0x in front is just a convention in writing to distinguish base-16 numbers), and 255 is 0xFF. How about 155? That’s 0x9B.

Another place where hexadecimal can be found is in IP addresses. An IPv4 address is made up of 4 numbers separated by a period. Each number in the address is a value between 0 and 255 and, as we saw above, can be represented by a 2-digit hexadecimal number.

If you want to know more about how computers work and how all this number base stuff really relates, a fantastic and easy to read book on the subject is Code: The Hidden Language of Computer Hardware and Software by Charles Petzold.

It turns out my life fits into hexadecimal ranges nicely:

  • 0x10 — I’m 16. Just got my driver’s license and went on my first date. I’m living with my parents in Joplin, Missouri. I’m much thinner than I am now.

  • 0x20 — I’m 32. I’m working at Autodesk in the QA group doing testing on AutoCAD. I live Sonoma, California and I’m about a year away from co-founding my own Internet startup.

  • 0x30 — I’m 48. I’m working at Rackspace in Austin, Texas building new cloud computing services. I have 3 kids and I’m on my third and final wife, if she’ll keep putting up with me.

atom

Setting Up PTR Records Though the Rackspace Cloud DNS API With cURL

For this example I’m going to show you how to setup a PTR record in Rackspace Cloud DNS for a Cloud Load Balancer using the command-line utility cURL. 

I’m using a Rackspace Cloud production account below; when you try it with your account change the account number and auth key to yours.

A few notes about cURL: this is a command line utility found on Mac and most Linux machines. You can also use other tools called “Rest Clients”. One popular one is an extension for Firefox and Chrome located here: http://restclient.net

Here’s a quick reference to some of the options I’ll be sending with curl in the examples below: -D Dump header to file (– sends to stdout) -H Extra header(s) to send -I Get header only -s silent (no progress bars or errors) -X Request method (GET is default)

Ok, first thing we need to do is get an Auth Token:

$ curl -s -I -H "X-Auth-Key: auth-key-for-account" \
  -H "X-Auth-User: username" \
  https://auth.api.rackspacecloud.com/v1.0

This will return just the header from the request (note the -I option) and the one we need is the value after “X-Auth-Token:”

Let’s list the domain so we can grab it’s ID (I’ve obscured my Auth Token below with $TOKEN):

$ curl -s -H "X-Auth-Token: $TOKEN" \
https://dns.api.rackspacecloud.com/v1.0/636983/domains

{"domains":[{"name":"sisk.ws","id":3325158,"created":"2012-07-20T18:04:11.000+0000","accountId":636983,"updated":"2012-07-31T18:40:22.000+0000","emailAddress":"mike.sisk@rackspace.com"},{"name":"wikirr.com","id":3174423,"created":"2012-02-28T19:26:35.000+0000","accountId":636983,"updated":"2012-07-21T21:58:41.000+0000","emailAddress":"ipadmin@stabletransit.com"},{"name":"wikirr.net","id":3314413,"comment":"This is a test domain created via the Cloud DNS API","created":"2012-07-11T14:54:00.000+0000","accountId":636983,"updated":"2012-07-11T14:54:03.000+0000","emailAddress":"mike.sisk@rackspace.com"}],"totalEntries":3}

Ok, that’s a little hard to read. But I can see the domain ID in question I need. Let’s get info on that and do a trick to format the output:

    $ curl -s -H "X-Auth-Token: $TOKEN" https://dns.api.rackspacecloud.com/v1.0/636983/domains/3325158 | python -m json.tool
    {
        "accountId": 636983, 
        "created": "2012-07-20T18:04:11.000+0000", 
        "emailAddress": "mike.sisk@rackspace.com", 
        "id": 3325158, 
        "name": "sisk.ws", 
        "nameservers": [
            {
                "name": "ns.rackspace.com"
            }, 
            {
                "name": "ns2.rackspace.com"
            }
        ], 
        "recordsList": {
            "records": [
                {
                    "created": "2012-07-20T18:05:14.000+0000", 
                    "data": "108.166.119.217", 
                    "id": "A-8591849", 
                    "name": "sisk.ws", 
                    "ttl": 300, 
                    "type": "A", 
                    "updated": "2012-07-31T18:39:25.000+0000"
                },
            {
                    "created": "2012-07-31T18:40:21.000+0000", 
                    "data": "66.216.68.19", 
                    "id": "A-8633106", 
                    "name": "www.sisk.ws", 
                    "ttl": 300, 
                    "type": "A", 
                    "updated": "2012-07-31T18:40:21.000+0000"
                }, 
                {
                    "created": "2012-07-20T18:04:11.000+0000", 
                    "data": "dns1.stabletransit.com", 
                    "id": "NS-7882349", 
                    "name": "sisk.ws", 
                    "ttl": 300, 
                    "type": "NS", 
                    "updated": "2012-07-20T18:04:11.000+0000"
                }, 
                {
                    "created": "2012-07-20T18:04:12.000+0000", 
                    "data": "dns2.stabletransit.com", 
                    "id": "NS-7882350", 
                    "name": "sisk.ws", 
                    "ttl": 300, 
                    "type": "NS", 
                    "updated": "2012-07-20T18:04:12.000+0000"
                }
            ], 
            "totalEntries": 4
        }, 
        "ttl": 300, 
        "updated": "2012-07-31T18:40:22.000+0000"
    }

The default output of the API is JSON, but it also supports XML if you send it another header. The little bit of python on the end just sends the output though a parser to format it for humans. You can see I have already created an A record for the www address. The PTR we’re creating below is essentially the reverse of that to map the IP address to www.sisk.ws.

Through the control panel I had already created an Cloud Load Balancer on this account. We need it’s ID for the PTR record, so let’s list the load balancer though it’s API:

  $ curl -s -H "Accept: application/json" -H "X-Auth-Token: $TOKEN" https://dfw.loadbalancers.api.rackspacecloud.com/v1.0/636983/loadbalancers/47219 | python -m json.tool

  {
      "loadBalancer": {
          "algorithm": "LEAST_CONNECTIONS", 
          "cluster": {
              "name": "ztm-n09.dfw1.lbaas.rackspace.net"
          }, 
          "connectionLogging": {
              "enabled": false
          }, 
          "contentCaching": {
              "enabled": false
          }, 
          "created": {
              "time": "2012-07-31T18:36:34Z"
          }, 
          "id": 47219, 
          "name": "Test1", 
          "nodes": [
              {
                  "address": "108.166.119.217", 
                  "condition": "ENABLED", 
                  "id": 114445, 
                  "port": 80, 
                  "status": "ONLINE", 
                  "type": "PRIMARY"
              }
          ], 
          "port": 80, 
          "protocol": "HTTP", 
          "sourceAddresses": {
              "ipv4Public": "64.49.225.5", 
              "ipv4Servicenet": "10.183.248.5", 
              "ipv6Public": "2001:4800:7901::9/64"
          }, 
          "status": "ACTIVE", 
          "updated": {
              "time": "2012-07-31T18:36:44Z"
          }, 
          "virtualIps": [
              {
                  "address": "66.216.68.19", 
                  "id": 3925, 
                  "ipVersion": "IPV4", 
                  "type": "PUBLIC"
              }, 
              {
                  "address": "2001:4800:7901:0000:290c:0b6b:0000:0001", 
                  "id": 9041755, 
                  "ipVersion": "IPV6", 
                  "type": "PUBLIC"
              }
          ]
      }
  }

Ok, that returns a lot of stuff. The thing we need from this output is the ID, 114445 and the ipv4 Public IP, 64.49.225.5.

The next step now that we’ve collected all the required information is calling the Cloud DNS API with a POST operation to create the PTR record and associate it with the Load Balancer. First thing I did was create the following JSON data in a text editor:

  {
    "recordsList" : {
      "records" : [ {
        "name" : "www.sisk.ws",
        "type" : "PTR",
        "data" : "66.216.68.19",
        "ttl" : 56000
      }, {
        "name" : "www.sisk.ws",
        "type" : "PTR",
        "data" : "2001:4800:7901:0000:290c:0b6b:0000:0001",
        "ttl" : 56000
      } ]
    },
    "link" : {
      "content" : "",
      "href" : "https://dfw.loadbalancers.api.rackspacecloud.com/v1.0/636983/loadbalancers/47219",
      "rel" : "cloudLoadBalancers"
    }
  }

This lists all the data we need to create the PTR record. In this example I also added the IPV6 address, too. Let’s give it a try:

  $ curl -D – -X POST -d '{
     "recordsList" : {
       "records" : [ {
         "name" : "www.sisk.ws",
         "type" : "PTR",
         "data" : "66.216.68.19",
         "ttl" : 56000
       }, {
         "name" : "www.sisk.ws",
         "type" : "PTR",
         "data" : "2001:4800:7901:0000:290c:0b6b:0000:0001",
         "ttl" : 56000
       } ]
     },
     "link" : {
       "content" : "",
       "href" : "https://dfw.loadbalancers.api.rackspacecloud.com/v1.0/636983/loadbalancers/47219",
       "rel" : "cloudLoadBalancers"
     }
   }' -H "Content-Type: application/json" -H "X-Auth-Token: $TOKEN" https://dns.api.rackspacecloud.com/v1.0/636983/rdns

I just typed in the commands and pasted in the above json file between the single-quote marks after the -d. In the Cloud DNS API commands that create stuff are asynchronous — what we get back is a url to check to see the status of the job. 

This is what we get back from the above POST:

    {"request":"{\n  \"recordsList\" : {\n    \"records\" : [ {\n      \"name\" : \"www.sisk.ws\",\n      \"type\" : \"PTR\",\n      \"data\" : \"66.216.68.19\",\n      \"ttl\" : 56000\n    }, {\n      \"name\" : \"www.sisk.ws\",\n      \"type\" : \"PTR\",\n      \"data\" : \"2001:4800:7901:0000:290c:0b6b:0000:0001\",\n      \"ttl\" : 56000\n    } ]\n  },\n  \"link\" : {\n    \"content\" : \"\",\n    \"href\" : \"https://dfw.loadbalancers.api.rackspacecloud.com/v1.0/636983/loadbalancers/47219\",\n    \"rel\" : \"cloudLoadBalancers\"\n  }\n}","status":"INITIALIZED","verb":"POST","jobId":"f99c1203-b42a-44c4-885f-c80bd7e3aba0","callbackUrl":"https://dns.api.rackspacecloud.com/v1.0/636983/status/f99c1203-b42a-44c4-885f-c80bd7e3aba0","requestUrl":"http://dns.api.rackspacecloud.com/v1.0/636983/rdns"}

Let’s check the job status:

      $ curl -s -H "X-Auth-Token: $TOKEN" https://dns.api.rackspacecloud.com/v1.0/636983/status/f99c1203-b42a-44c4-885f-c80bd7e3aba0 | python -m json.tool
      {
          "callbackUrl": "https://dns.api.rackspacecloud.com/v1.0/636983/status/f99c1203-b42a-44c4-885f-c80bd7e3aba0", 
          "jobId": "f99c1203-b42a-44c4-885f-c80bd7e3aba0", 
          "status": "COMPLETED"
      }

These jobs complete quickly, so by the time you check it’s usually done. If the system is under a lot of load, or it’s a really big job (like updating hundreds of records) it might take a few minutes. 

Ok, let’s see what the PTR record looks like. First, keep in mind the PTR record lives with the device, not the domain record. So doing a list of the domain won’t show us the PTR. We have to list it from the device like this:

      $ curl -s -H "X-Auth-Token: $TOKEN" https://dns.api.rackspacecloud.com/v1.0/636983/rdns/cloudLoadBalancers?href=https://dfw.loadbalancers.api.rackspacecloud.com/v1.0/636983/loadbalancers/47219 | python -m json.tool
      {
          "records": [
              {
                  "created": "2012-07-31T20:16:32.000+0000", 
                  "data": "66.216.68.19", 
                  "id": "PTR-239719", 
                  "name": "www.sisk.ws", 
                  "ttl": 56000, 
                  "type": "PTR", 
                  "updated": "2012-07-31T20:16:32.000+0000"
              }, 
              {
                  "created": "2012-07-31T20:16:35.000+0000", 
                  "data": "2001:4800:7901:0:290c:b6b:0:1", 
                  "id": "PTR-239720", 
                  "name": "www.sisk.ws", 
                  "ttl": 56000, 
                  "type": "PTR", 
                  "updated": "2012-07-31T20:16:35.000+0000"
              }
          ]
      }

Ok, looks good. Let’s test it:

      $ host www.sisk.ws
      www.sisk.ws has address 66.216.68.19

      $  host 66.216.68.19
      19.68.216.66.in-addr.arpa domain name pointer www.sisk.ws.

Looks good — that’s what we expect to see. To compare, let’s see what it looks like with just the root of the domain which is pointing to a first generation cloud server that doesn’t support PTR records:

      $ host sisk.ws
      sisk.ws has address 108.166.119.217

      $ host 108.166.119.217
      217.119.166.108.in-addr.arpa domain name pointer 108-166-119-217.static.cloud-ips.com.

atom

S3cmd With Multiple Accounts

Recently I’ve been doing a lot of work involving Amazon Simple Storage Service (aka Amazon S3).

And while tools like Panic’s Transmit, the Firefox S3Fox extension, or even Amazon’s own S3 Management Console make it easy to use, sometimes you really need a command-line tool.

There’s a lot of good tools out there, but the one I’ve been using is s3cmd. This tool is done in Python, has been around awhile and is well documented. Installation on Linux or OS X is simple as is its configuration. And as a longtime Unix command-line user it’s syntax is simple and Unixy. Some examples:

To list your buckets:

~ $ s3cmd ls
2010-04-28 23:50 s3://g5-images
2011-01-21 06:42 s3://g5-mongodb-backup
2011-03-21 21:23 s3://g5-mysql-backup
2010-06-03 17:45 s3://g5-west-images
2010-09-02 15:57 s3://g5engineering

List the size of a bucket with “human readable” units:

~ $ s3cmd du -H s3://g5-mongodb-backup
1132G s3://g5-mongodb-backup/

List the contents of a bucket:

~ $ s3cmd ls s3://g5-mongodb-backup
2011-08-08 14:43 3273232889 s3://g5-mongodb-backup/mongodb.2011-08-08-06.tar.gz
2011-08-08 21:12 3290592536 s3://g5-mongodb-backup/mongodb.2011-08-08-12.tar.gz
2011-08-09 03:16 3302734859 s3://g5-mongodb-backup/mongodb.2011-08-08-18.tar.gz
2011-08-09 09:09 3308369423 s3://g5-mongodb-backup/mongodb.2011-08-09-00.tar.gz
2011-08-09 14:51 3285753739 s3://g5-mongodb-backup/mongodb.2011-08-09-06.tar.gz

Show the MD5 hash of an asset:

~ $ s3cmd ls --list-md5 s3://g5-mongodb-backup/mongodb.2011-08-09-06.tar.gz
2011-08-09 14:51 3285753739 07747e3de16138799d9fe1846436a3ce \
s3://g5-mongodb-backup/mongodb.2011-08-09-06.tar.gz

Transferring a file to a bucket uses the get and put commands. And if you forget an option or need a reminder of usage the very complete s3cmd —help output will likely be all the help you need.

One problem I have with most tools for AWS is managing multiple accounts. Most of these tools assume you have just one account, but I work with multiple accounts and switching between them can be cumbersome.

Here’s how I work with multiple AWS accounts using s3cmd.

By default s3cmd puts its configuration file in ~/.s3cfg, but you can override this and specify a configuration file with the -c option.

What I do is create a separate config file with the appropriate credentials for each account I work with and give them unique names:

~ $ ls -1 .s3cfg*
.s3cfg-g5
.s3cfg-tcp

Another option is to keep the credentials for the account you use most often in the standard ~/.s3cfg file and use the -c option when/if you need another account. I don’t like this option because it’s too easy to mistakenly use the wrong account. For example, without a ~/.s3cfg this is what happens when I use s3cmd without specifying a configuration:

~ $ s3cmd ls
ERROR: /Users/mike/.s3cfg: No such file or directory
ERROR: Configuration file not available.
ERROR: Consider using --configure parameter to create one.

So, what to do? Using the -c all the time is a PITA. Answer: use Bash aliases! Here’s a subset of the s3cmd aliases I have in my ~/.bashrc file:

# s3cmd aliases for different s3 accounts
alias s3g5='s3cmd -c ~/.s3cfg-g5'
alias s3tcp='s3cmd -c ~/.s3cfg-tcp'

Now, to list the buckets in my personal account I just do:

~ $ s3tcp ls
2011-07-01 06:10 s3://mikesisk-img
2011-07-05 23:16 s3://www.tcpipranch.com
2011-07-01 22:55 s3://www.watch4rocks.com

And I can still pass arguments:

~ $ s3tcp -H --list-md5 ls s3://mikesisk-img/me.jpg
2011-07-01 06:09 5k 13d7c86bccd8915dd93b085985305394 \
s3://mikesisk-img/me.jpg

Just keep in mind that calls to bash aliases from scripts and cronjobs might not work. Plus it’s bad form and will come back to bite you one of these days. Just use the long form with -c in these places and keep the aliases for your own interactive command-line usage.

atom

Cron and Needles

The actual needle.

Sometimes, even after decades of experience, you still screw up.

Consider this cron entry I put in last night:

# Backup MongoDB every 6 hours, zip it up, and rsync it.
* */6 * * * ~/bin/backup_mongo.sh

I wanted this to run the backup script for MongoDB every six hours. Instead, I got it running every minute for an hour every six hours. You’d think I’d know better considering I put the next cron in correctly:

# Remove MongoDB backups that are more than 24-hours old.
00 02 * * * find /db/backup -mtime +1 -exec rm -f {} \;

What I meant to do is this:

# Backup MongoDB every 6 hours, zip it up, and rsync it.
00 */6 * * * ~/bin/backup_mongo.sh

Luckily our hosting provider’s staff noticed the CPU spike on this server at midnight and fixed the cron.

Which brings up another point: name your scripts appropriately. In this case a quick scan of cron revealed this script was running a backup and doing that every six hours makes sense. If the script was just named mongo, it’s conceivable it could have been a metric collection script that runs every minute for an hour every six hours.

So what do needles have to do with cron? I was working from home the week this happened and had just finished that MongoDB backup script and was putting it in cron when my daughter came running (Ok, make that limping) into my office with a large sewing needle in-bedded in the arch of her foot. I quickly saved the cron entry to take care of that problem and didn’t go back to check my work.

Moral of the story: whenever you set up a new cron job it’s a good idea to watch it run and see if it’s doing what you think it is. Especially if you think you know what you’re doing.

atom

What Is DNS?

The Domain Name System (DNS) is a distributed database that maps domain names to network IP addresses.

That, essentially, is all DNS does. You could navigate around the Internet via IP addresses instead of domain names and just remember that google.com is 72.14.213.147. And in the early days of the Internet in the late 1960’s this is exactly what folks did.

By 1971 the nascent Internet had grown enough that remembering network addresses was a burdensome way to navigate. The solution was a database in a simple text file called hosts.txt that was installed on each machine on the Internet and provided a simple way to map a network address to an easy-to-remember name.

This system worked well but wasn’t very scalable — every time a host was added to the Internet or an network address changed every connected machine had to get an updated hosts.txt file.

A solution called DNS was conceived in 1981 and starting in 1985 the first domain names were registered. Some of the behind the scenes technical details have changed (and continue to evolve), but DNS essentially works today the same as it did in 1985.

We need to define a few terms used in DNS so we have a common vocabulary.

First up is domain name (sometimes just called a domain). You know this one already — g5platform.com, amazon.com, and google.com are all domains.

The information that defines a domain is called a zone.

The actual meat of DNS is contained inside the zone in resource records. These are things with names like “A”, “CNAME” and “MX”. The information in these resource records (or just records) is where the magic of DNS happens.

The “A” record is the most important. This is the address record and maps an IP address to a name.

More history: DNS was designed way before Tim Berners-Lee invented the World Wide Web (WWW or just “web”). Back then you usually didn’t need to access something like mit.edu; this is just the start (a.k.a. origin) of the domain name and before the web more just the origin wouldn’t give you anything. DNS was designed to connect machines, so back then you’d probably want to access another machine with something like swissnet.ai.mit.edu (which would connect you to a machine called swissnet inside the ai sub-zone of the mit.edu zone).

In the early days of the web a host running a domains website was given the name www, stuck in a corner somewhere and put in DNS as www.mit.edu. Eventually new web-only startups arrived who purpose online was the website. For these guys getting the website with just the origin of a domain was a handy way to work. Thus we needed a way to route the origin of a domain name somewhere so amazon.com would work just the same as www.amazon.com.

Luckily the folks that designed DNS were pretty smart and this capability was already implemented. It’s called the origin and you see it in most registrar control panels as the “@” sign. So, when you see an “A” record with an “@” as the host, that’s the origin and all it means is that the start of the domain name will go to the IP address assigned. Simple, huh?

Our next resource record to talk about is related to the above in a way. The “CNAME” record is short for “canonical name”, but is usually just called an alias. It differs from the “A” record in that it doesn’t map to an IP address — it maps to another domain name. It can be a full domain name (like www.amazon.com) or just a host (like www).

A CNAME is the most common way to specify the www record and you just point it to the origin (“@”).

Now, you could do the www as an A record, but if you ever needed to change the IP address your domain points to you’d now have two A records to change. With the www as a CNAME pointing to @ you have only one record to change if your IP address changes. Neat, huh?

Next is the MX records. This is the mail exchanger records and it tells email servers where to send email for your domain. Like the CNAME, it maps to a host or domain name, not an IP address.

There’s other resource records like TXT, SRV and AAAA, but they support more advanced DNS features and you don’t often need to change them.

In summary, DNS appears complicated but in reality it’s just a simple service that maps a name to an address.

atom

Some Vi Editor Tips

Awhile back at a previous company I got hit with a last-minute request to update our Nginx redirect map with data provided in a Google docs spreadsheet. Normally I only have to do one or two redirect rules at a time. But today I got hit with 120 rules – each of which needed data from two cells in the spreadsheet. Doing it manually would require 240 cut-n-paste operations – not fun and error prone. Oh, and I only had 30-minutes to get this done and up in production.

We do these sorts of redirects when a customer changes their domain name. In this particular case the customer had a number of locations under one domain and they’re now splitting each location out into its own domain. But we don’t want visitors to the sites getting 404s due to the URL changing so we put in redirect rules to rewrite the request and forward the visitor to the new domain.

There’s a bunch of ways to do this, but this is how I did it.

First, a quick explanation of the Nginx redirect map format. There’s not much to it:

old.example.com    new.example.com;

That’s the old URL, a space or spaces (or tab), the new URL and a semicolon terminating the line.

In this case, each row of the spreadsheet with the redirect data had 7 columns: the name of the property being redirected, and three URLs that needed to be redirected with the destination (i.e. Each location in this case only has three pages so there’s only three redirects each). Luckily the order of the data is just what I needed for the map.

The first thing I did was a CSV export of the data and opened it up in vi.

1

The CSV export contained title information for the columns I don’t need so let’s just delete those right off.

2

Ok, now the data is ready to be processed into something I can use. In another stroke of luck we can see each redirect pair is separated by two commas since the spreadsheet contained an empty column between the three pairs for each location. This will make things much easier.

3

First, let’s get each redirect on its own line in the file. We can search and replace for the domain name to be redirected since they’re all the same. We’ll search for the domain and replace it as the first thing on its own line with:

:%s:www.myfavoriteapartment.com:\rwww.myfavoriteapartment.com:g

The % does all lines in the file; the s is for search; the \r is vi-speak for newline; and the g at the end is for global so it’ll process the whole line rather than the first match it finds on that line. By habit I use : for the separator in the search and replace command; you can also (and most people do) use /.

4

Ok, we’re getting there. Next let’s get rid of the double commas and – since that’s always at the end of the redirect destination – put a semicolon at the end as Nginx requires:

:%s:,,:;:g

5

Now we need to deal with the first column of data from the spreadsheet – the name of the location. We don’t need this information for the map file so let’s just delete it. Each location name is on its own line at this point and ends with a comma. So let’s search for lines ending with a comma and delete ‘em:

:g:,$:d

In this case, the g is for a global operation on all lines; the ,$ matches all lines with a , at the end of a line (the $); and the d is for delete.

6

All we have left is to replace the single remaining comma on each line that separates the source and destination URLs. Nginx requires that the separation here be one or more spaces. I typically use a tab (although I probably shouldn’t – it makes the map file look messy) so let’s do this:

:%s:,:^I:g

The ^I is the tab character. Nowadays you can usually just press the tab key and vi will put in the ^I, but in the old days you had to do control-v and control-i to get a tab.

7

And that’s it.

8

Now all we got to do is save the file and do a copy and paste of its contents into the map file on our production configuration. Of course, you’ll want to scroll through the file and make sure it looks correct and do a nginx configtest before nginx reload to make sure it’s valid.

atom