Using the Joyent Cloud API

Here's some notes I took while doing some initial experiments with provisioning machines in the Joyent Cloud. I used their CloudAPI directly, although in the future I also want to try the libcloud Joyent driver. The promise of the Joyent Cloud 'SmartMachines' is that they are really Solaris zones running on a SmartOS host, and that gives you more performance (especially I/O performance) than regular virtual machines such as the ones offered by most cloud vendors. I have yet to fully verify this performance increase, but it's next on my TODO list.

Installing the Joyent CloudAPI tools


I did the following on an Ubuntu 10.04 server:

• installed node.js -- I downloaded it in tar.gz format from http://nodejs.org/dist/v0.6.19/node-v0.6.19.tar.gz then I ran the usual './configure; make; make install'
• installed the Joyent smartdc node package by runing 'npm install smartdc -g'
• created new ssh RSA keypair: id_rsa_joyentapi (private key) and id_rsa_joyentapi.pub (public key)
• ran the sdc-setup utility, pointing it to the US-EAST-1 region:

# sdc-setup https://us-east-1.api.joyentcloud.com
Username (login): (root) myjoyentusername
Password:
The following keys exist in SmartDataCenter:
   [1] grig
Would you like to use an existing key? (yes) no
SSH public key: (/root/.ssh/id_rsa.pub) /root/.ssh/id_rsa_joyentapi.pub

If you set these environment variables, your life will be easier:
export SDC_CLI_URL=https://us-east-1.api.joyentcloud.com
export SDC_CLI_ACCOUNT=myjoyentusername
export SDC_CLI_KEY_ID=id_rsa_joyentapi
export SDC_CLI_IDENTITY=/root/.ssh/id_rsa_joyentapi

• added recommended environment variables (above) to .bash_profile, sourced the file

Using the Joyent CloudAPI tools

At this point I was able to use the various 'sdc' commands included in the Joyent CloudAPI toolset. For example, to list the available Joyent datacenters, I used sdc-listdatacenters:

# sdc-listdatacenters
{
 "us-east-1": "https://us-east-1.api.joyentcloud.com",
 "us-west-1": "https://us-west-1.api.joyentcloud.com",
 "us-sw-1": "https://us-sw-1.api.joyentcloud.com"
}

To list the available operating system images available for provisioning, I used sdc-listdatasets (the following is just an excerpt of its output):

# sdc-listdatasets
[
 {
"id": "988c2f4e-4314-11e1-8dc3-2bc6d58f4be2",
"urn": "sdc:sdc:centos-5.7:1.2.1",
"name": "centos-5.7",
"os": "linux",
"type": "virtualmachine",
"description": "Centos 5.7 VM 1.2.1",
"default": false,
"requirements": {},
"version": "1.2.1",
"created": "2012-02-14T05:53:49+00:00"
 },
 {
"id": "e4cd7b9e-4330-11e1-81cf-3bb50a972bda",
"urn": "sdc:sdc:centos-6:1.0.1",
"name": "centos-6",
"os": "linux",
"type": "virtualmachine",
"description": "Centos 6 VM 1.0.1",
"default": false,
"requirements": {},
"version": "1.0.1",
"created": "2012-02-15T20:04:18+00:00"
 },
  {
"id": "a9380908-ea0e-11e0-aeee-4ba794c83c33",
"urn": "sdc:sdc:percona:1.0.7",
"name": "percona",
"os": "smartos",
"type": "smartmachine",
"description": "Percona SmartMachine",
"default": false,
"requirements": {},
"version": "1.0.7",
"created": "2012-02-13T19:24:17+00:00"
 },
etc

To list the available machine sizes available for provisioning, I used sdc-listpackages (again, this is just an excerpt of its output):

# sdc-listpackages
[
 {
"name": "Large 16GB",
"memory": 16384,
"disk": 491520,
"vcpus": 3,
"swap": 32768,Cloud Analytics API
"default": false
 },
 {
"name": "XL 32GB",
"memory": 32768,
"disk": 778240,
"vcpus": 4,
"swap": 65536,
"default": false
 },
 {
"name": "XXL 48GB",
"memory": 49152,
"disk": 1048576,
"vcpus": 8,
"swap": 98304,
"default": false
 },
 {
"name": "Small 1GB",
"memory": 1024,
"disk": 30720,
"vcpus": 1,
"swap": 2048,
"default": true
 },
etc

Provisioning and terminating machines

To provision a machine, you use sdc-createmachine and pass it the 'urn' field of the dataset (OS) you 

want, and the package name for the size you want. Example:

# sdc-createmachine --dataset sdc:sdc:percona:1.3.9 --package "Large 16GB"
{
 "id": "7ccc739e-c323-497a-88df-898dc358ea40",
 "name": "a0e7314",
 "type": "smartmachine",
 "state": "provisioning",
 "dataset": "sdc:sdc:percona:1.3.9",
 "ips": [
"A.B.C.D",
"X.Y.Z.W"
 ],
 "memory": 16384,
 "disk": 491520,
 "metadata": {
"credentials": {
  "root": "",
  "admin": "",
  "mysql": ""
}
 },
 "created": "2012-06-07T17:55:29+00:00",
 "updated": "2012-06-07T17:55:30+00:00"
}

The above command provisions a Joyent SmartMachine running the Percona distribution of MySQL in the 'large' size, with 16 GB RAM. Note that the output of the command contains the external IP of the provisioned machine (A.B.C.D) and also its internal IP (X.Y.Z.W). The output also contains the passwords for the root, admin and mysql accounts, in the metadata field.

Here's another example for provisioning a machine running Ubuntu 10.04 in the 'small' size (1 GB RAM). You can also specify a machine name when you provision it:

# sdc-createmachine --dataset sdc:sdc:ubuntu-10.04:1.0.1 --package "Small 1GB" --name ggtest
{
 "id": "dc856044-7895-4a52-bfee-35b404061920",
 "name": "ggtest",
 "type": "virtualmachine",
 "state": "provisioning",
 "dataset": "sdc:sdc:ubuntu-10.04:1.0.1",
 "ips": [
"A1.B1.C1.D1",
"X1.Y1.Z1.W1"
 ],
 "memory": 1024,
 "disk": 30720,
 "metadata": {
"root_authorized_keys": ""
 },
 "created": "2012-06-07T19:28:19+00:00",
 "updated": "2012-06-07T19:28:19+00:00"
}

For an Ubuntu machine, the 'metadata' field contains the list of authorized ssh keys (which I removed from my example above). Also, note that the Ubuntu machine is of type 'virtualmachine' (so a regular KVM virtual instance) as opposed to the Percona Smart Machine, which is of type 'smartmachine' and is actually a Solaris zone within a SmartOS physical host.

To list your provisioned machines, you use sdc-listmachines:

# sdc-listmachines
[
 {
"id": "36b50e4c-88d2-4588-a974-11195fac000b",
"name": "db01",
"type": "smartmachine",
"state": "running",
"dataset": "sdc:sdc:percona:1.3.9",
"ips": [
  "A.B.C.D",
  "X.Y.Z.W"
],
"memory": 16384,
"disk": 491520,
"metadata": {},
"created": "2012-06-04T18:03:18+00:00",
"updated": "2012-06-07T00:39:20+00:00"
 },

  {

    "id": "dc856044-7895-4a52-bfee-35b404061920",

    "name": "ggtest",

    "type": "virtualmachine",

    "state": "running",

    "dataset": "sdc:sdc:ubuntu-10.04:1.0.1",

    "ips": [

      "A1.B1.C1.D1",

      "X1.Y1.Z1.W1"

    ],

    "memory": 1024,

    "disk": 30720,

    "metadata": {

      "root_authorized_keys": ""

    },

    "created": "2012-06-07T19:30:29+00:00",

    "updated": "2012-06-07T19:30:38+00:00"

  },

]

Note that immediately after provisioning a machine, its state (as indicated by the 'state' field in the output of sdc-listmachines) will be 'provisioning'. The state will change to 'running' once the provisioning process is done. At that point you should be able to ssh into the machine using the private key you created when installing the CloudAPI tools.

To terminate a machine, you first need to stop it via sdc-stopmachine, then to delete it via sdc-deletemachine. Both of these tools take the id of the machine as a parameter. If you try to delete a machine without first stoppping it, or without waiting sufficient time for the machine to go into the 'stopped' state, you will get a message similar to Requested transition is not acceptable due to current resource state.

Bootstrapping a machine with user data

In my opinion, a cloud API for provisioning instances/machines is only useful if it offers a bootstrapping mechanism for running user-specified scripts upon the first run. This would enable an integration with configuration management tools such as Chef or Puppet. Fortunately, the Joyent CloudAPI does support this bootstrapping via its Metadata API.

For a quick example of a customized bootstrapping action, I changed the hostname of an Ubuntu machine and also added it to /etc/hostname. This is a toy example. In a real-life situation, you would instead download a script from one of your servers and run it in order to install whatever initial packages you need, then to configure the machine as a Chef or Puppet client, etc. In any case, you need to actually spell out the commands you need the machine to run during its initial provisioning boot process. You do that by defining the metadata 'user-script' variable:

# sdc-createmachine --dataset sdc:sdc:ubuntu-10.04:1.0.1 --package "Small 1GB" --name ggtest2 --metadata user-script='hostname ggtest2; echo ggtest2 > /etc/hostname'
{
 "id": "379c0cad-35bf-462a-b680-fc091c74061f",
 "name": "ggtest2",
 "type": "virtualmachine",
 "state": "provisioning",
 "dataset": "sdc:sdc:ubuntu-10.04:1.0.1",
 "ips": [
"A2.B2.C2.D2",
"X2.Y2.Z2.W2"
 ],
 "memory": 1024,
 "disk": 30720,
 "metadata": {
"user-script": "hostname ggtest2; echo ggtest2 > /etc/hostname",
"root_authorized_keys": ""
 },
 "created": "2012-06-08T23:17:44+00:00",
 "updated": "2012-06-08T23:17:44+00:00"
}

Note that the metadata field now contains the user-script variable that I specified.

Collecting performance metrics with Joyent Cloud Analytics

The Joyent Cloud Analytics API lets you define metrics that you want to query for on your machines in the Joyent cloud. Those metrics are also graphed on the Web UI dashboard as you define them, which is a nice touch. For now there aren't that many such metrics available, but I hope their number will increase.

Joyent uses a specific nomenclature for the Analytics API. Here are some definitions, verbatim from their documentation (CA means Cloud Analytics):

A metric is any quantity that can be instrumented using CA. For examples:

• Disk I/O operations
• Kernel thread executions
• TCP connections established
• MySQL queries
• HTTP server operations
• System load average

When you want to actually gather data for a metric, you create an instrumentation. The instrumentation specifies:

• which metric to collect
• an optional predicate based on the metric's fields (e.g., only collect data from certain hosts, or data for certain operations)
• an optional decomposition based on the metric's fields (e.g., break down the results by server hostname)
• how frequently to aggregate data (e.g., every second, every hour, etc.)
• how much data to keep (e.g., 10 minutes' worth, 6 months' worth, etc.)
• other configuration options

To get started with this API, you need to first see what analytics/metrics are available. You do that by calling sdc-describeanalytics (what follows is just a fragment of the output):

# sdc-describeanalytics
 "metrics": [
{
  "module": "cpu",
  "stat": "thread_samples",
  "label": "thread samples",
  "interval": "interval",
  "fields": [
    "zonename",
    "pid",
    "execname",
    "psargs",
    "ppid",
    "pexecname",
    "ppsargs",
    "subsecond"
  ],
  "unit": "samples"
},
{
  "module": "cpu",
  "stat": "thread_executions",
  "label": "thread executions",
  "interval": "interval",
  "fields": [
    "zonename",
    "pid",
    "execname",
    "psargs",
    "ppid",
    "pexecname",
    "ppsargs",
    "leavereason",
    "runtime",
    "subsecond"
  ],
etc

You can create instrumentations either via the Web UI (go to the Analytics tab) or via the command line API.

Here's an example of creating an instrumentation for file system logical operations via the sdc-createinstrumentation API:

# sdc-createinstrumentation -m fs -s logical_ops

{

  "module": "fs",

  "stat": "logical_ops",

  "predicate": {},

  "decomposition": [],

  "value-dimension": 1,

  "value-arity": "scalar",

  "enabled": true,

  "retention-time": 600,

  "idle-max": 3600,

  "transformations": {},

  "nsources": 0,

  "granularity": 1,

  "persist-data": false,

  "crtime": 1340228876662,

  "value-scope": "interval",

  "id": "17",

  "uris": [

    {

      "uri": "/myjoyentusername/analytics/instrumentations/17/value/raw",

      "name": "value_raw"

    }

  ]

}

To list the instrumentations you have created so far, you use sdc-listinstrumentations:

# sdc-listinstrumentations

[

  {

    "module": "fs",

    "stat": "logical_ops",

    "predicate": {},

    "decomposition": [],

    "value-dimension": 1,

    "value-arity": "scalar",

    "enabled": true,

    "retention-time": 600,

    "idle-max": 3600,

    "transformations": {},

    "nsources": 2,/

    "granularity": 1,

    "persist-data": false,

    "crtime": 1340228876662,

    "value-scope": "interval",

    "id": "17",

    "uris": [

      {

        "uri": "/myjoyentusername/analytics/instrumentations/17/value/raw",

        "name": "value_raw"

      }

    ]

  }

]

To retrieve the actual metrics captured by a given instrumentation, call sdc-getinstrumentation and pass it the instrumentation id:

# sdc-getinstrumentation -v 17 { "value": 1248, "transformations": {}, "start_time": 1340229361, "duration": 1, "end_time": 1340229362, "nsources": 2, "minreporting": 2, "requested_start_time": 1340229361, "requested_duration": 1, "requested_end_time": 1340229362 } 

You can see how this can be easily integrated with some like Graphite in order to keep historical information about these metrics.

You can dig deeper into a specific metric by decomposing it by different fields, such as the application name. For example, to see filesystem logical operation by application name, you would call:

# sdc-createinstrumentation -m fs -s logical_ops --decomposition execname

{

  "module": "fs",

  "stat": "logical_ops",

  "predicate": {},

  "decomposition": [

    "execname"

  ],

  "value-dimension": 2,

  "value-arity": "discrete-decomposition",

  "enabled": true,

  "retention-time": 600,

  "idle-max": 3600,

  "transformations": {},

  "nsources": 0,

  "granularity": 1,

  "persist-data": false,

  "crtime": 1340231734049,

  "value-scope": "interval",

  "id": "18",

  "uris": [

    {

      "uri": "/myjoyentusername/analytics/instrumentations/18/value/raw",

      "name": "value_raw"

    }

  ]

}

Now if you retrieve the value for this instrumentation, you see several values in the output, one value for application that performs file system logical operations:

# sdc-getinstrumentation -v 18

{

  "value": {

    "grep": 4,

    "ksh93": 5,

    "cron": 7,

    "gawk": 15,

    "svc.startd": 2,

    "mysqld": 163,

    "nscd": 27,

    "top": 159

  },

  "transformations": {},

  "start_time": 1340231762,

  "duration": 1,

  "end_time": 1340231763,

  "nsources": 2,

  "minreporting": 2,

  "requested_start_time": 1340231762,

  "requested_duration": 1,

  "requested_end_time": 1340231763

}

Another useful technique is to isolate metrics pertaining to a specific host (or 'zonename' in Joyent parlance). For this, you need to specify a predicate that will filter only the host with a specific id (you can see the id of a host when you call sdc-listmachines). Here's an example that captures the CPU wait time for a Percona SmartMachine which I provisioned earlier:

# sdc-createinstrumentation -m cpu -s waittime -p '{"eq": ["zonename","36b50e4c-88d2-4588-a974-11195fac000b"]}'

{

  "module": "cpu",

  "stat": "waittime",

  "predicate": {

    "eq": [

      "zonename",

      "36b50e4c-88d2-4588-a974-11195fac000b"

    ]

  },

  "decomposition": [],

  "value-dimension": 1,

  "value-arity": "scalar",

  "enabled": true,

  "retention-time": 600,

  "idle-max": 3600,

  "transformations": {},

  "nsources": 0,

  "granularity": 1,

  "persist-data": false,

  "crtime": 1340232271092,

  "value-scope": "interval",

  "id": "19",

  "uris": [

    {

      "uri": "/myjoyentusername/analytics/instrumentations/19/value/raw",

      "name": "value_raw"

    }

  ]

}

You can combine decomposition with predicates. For example, here's how to create an instrumentation for CPU usage time decomposed by CPU mode (user, kernel):

# sdc-createinstrumentation -m cpu -s usage -n cpumode -p '{"eq": ["zonename","36b50e4c-88d2-4588-a974-11195fac000b"]}' { "module": "cpu", "stat": "usage", "predicate": { "eq": [ "zonename", "36b50e4c-88d2-4588-a974-11195fac000b" ] }, "decomposition": [ "cpumode" ], "value-dimension": 2, "value-arity": "discrete-decomposition", "enabled": true, "retention-time": 600, "idle-max": 3600, "transformations": {}, "nsources": 0, "granularity": 1, "persist-data": false, "crtime": 1340232361944, "value-scope": "point", "id": "20", "uris": [ { "uri": "/myjoyentusername/analytics/instrumentations/20/value/raw", "name": "value_raw" } ] }

Now when you retrieve the values for this instrumentation, you can see them separated by CPU mode:

# sdc-getinstrumentation -v 20 { "value": { "kernel": 24, "user": 28 }, "transformations": {}, "start_time": 1340232390, "duration": 1, "end_time": 1340232391, "nsources": 2, "minreporting": 2, "requested_start_time": 1340232390, "requested_duration": 1, "requested_end_time": 1340232391 }

Finally, here's a MySQL-specific instrumentation that you can create on a machine running MySQL, such as a Percona SmartMachine. This one is for capturing MySQL queries:

# sdc-createinstrumentation -m mysql -s queries -p '{"eq": ["zonename","36b50e4c-88d2-4588-a974-11195fac000b"]}' { "module": "mysql", "stat": "queries", "predicate": { "eq": [ "zonename", "36b50e4c-88d2-4588-a974-11195fac000b" ] }, "decomposition": [], "value-dimension": 1, "value-arity": "scalar", "enabled": true, "retention-time": 600, "idle-max": 3600, "transformations": {}, "nsources": 0, "granularity": 1, "persist-data": false, "crtime": 1340232562361, "value-scope": "interval", "id": "22", "uris": [ { "uri": "/myjoyentusername/analytics/instrumentations/22/value/raw", "name": "value_raw" } ] }
Overall, I found the Joyent Cloud API and its associated Analytics API fairly easy to use, once I got past some nomenclature quirks. I also want to mention that the support I got from Joyent was very, very good. Replies to questions regarding some of the topics I discussed here were given promptly and knowledgeably. My next step is gauging the performance of MySQL on a SmartMachine, when compared to a similar-sized instance running in the Amazon EC2 cloud. Stay tuned.

A sweep through my Instapaper for May 2012

Here are some of the presentations/blog posts/articles I read this month, as saved in my Instapaper account. Maybe you'll find something useful in there too.

• "Blameless PostMortems and a Just Culture" -- John Allspaw describes the post mortem process at Etsy, in particular how it encourages engineers to own up to their mistakes and to help others avoid them, all in a safe and blame-free environment
• "Sprint.ly's Continuous Integration" -- good overview of what should be by now industry-standard methods and tools for rapid deployments and continuous integration
• "Big List of 20 Common Bottlenecks" -- via the super useful High Scalability blog, a list of bottlenecks that you will hit one way or another if you do any serious high-volume systems work
• "BASE: An ACID alternative" -- Dan Pritchett from EBay coined the term BASE (basically available, soft state, eventually consistent) to compare and contrast NoSQL systems with traditional ACID-based relational databases; very good overview of these types of systems
• "Calvin: Fast Distributed Transactions for Partitioned Database Systems" (PDF) -- Daniel Abadi and collaborators write about a distributed transaction mechanism that can sit on top of non-transactional storage systems, transforming them into scalable, highly-available ACID databases
• "Creating a BOSH from scratch on AWS" -- great tutorial from Dr Nic from Engine Yard on installing CloudFoundry's BOSH tool on AWS
• "Amazon's Journey to the Cloud" -- very good presentation by John Rauser on the long and winding road taken by Amazon from their beginnings running on 1 machine to the launch of AWS technologies
• "Engineering Change" -- short but very insightful presentation by Etsy's CTO Kellan Elliott-McCrea on continuous deployment strategies and metrics-driven development
• "People Make Poor Monitors for Computers" -- eye-opening article on the dangers on relying on highly automated and sophisticated monitoring systems; when they fail, human operators are expected to jump in and fix the issues, but unfortunately those rare issues are extremely hard to diagnose and fix by humans who have lost their edge by relying on the automated systems in the first place!
• "vbench - benchmarking performance through time" -- from Wes McKinney's Panda project, vbench is a lightweight Python library for measuring code performance and catching performance regressions
• "Big Data -- a little analysis" -- switching gears to Big Data, here's a good overview/taxonomy of types of problems in this space, based on data volume and algorithm complexity, courtesy of Chris Swan
• "The unsexy side of big data: 5 tools to manage your Hadoop cluster" -- some tools I had never heard of for managing Hadoop clusters, including Apache Ambari and Apache Mesos
• "Online resources for handling big data and parallel computing in R" -- from the R-bloggers blog aggregator, a useful collection of links on mostly parallel computing with R
• "Much to like about HBaseCon" -- quick overview of some talks from last week's HBaseCon 2012

I also want to give a shout-out here to Gareth Rushgrove, who publishes an email newsletter called 'Devops Weekly'. If you are working in this field, I highly recommend you subscribe to it, as it is always full of interesting links and summaries to articles and tools.

The correct way of using DynamoDB BatchWriteItem with boto

In my previous post I wrote about the advantages of using the BatchWriteItem functionality in DynamoDB. As it turns out, I was overly optimistic when I wrote my initial code: I only called the batch_write_item method of the layer2 module in boto once.

The problem with this approach is that many of the batched inserts can fail, and in practice this happens quite frequently, probably because of transient network errors. The correct approach is to inspect the response object returned by batch_write_item -- here is an example of such an object:


{'Responses': {'mytable': {'ConsumedCapacityUnits': 5.0}},
 'UnprocessedItems': {'mytable': [
{'PutRequest': {'Item': {'mykey': 'key1', 'myvalue': 'value1'}}},
{'PutRequest': {'Item': {'mykey': 'key2', 'myvalue': 'value2'}}},
{'PutRequest': {'Item': {'mykey': 'key3', 'myvalue': 'value3'}}}]}}

You need to look for the value corresponding to the 'UnprocessedItems' key. This value is a dictionary keyed by the name of the table you're inserting items in. The value corresponding to that key gives you a list of other dictionaries with keys corresponding to the operations you applied to the table ('PutRequest' in my case). Going one level deeper allows you to finally obtain the attributes (keys + values) of the items that failed, which you can then try to re-insert.

So basically you need to stay in a loop and keep calling batch_write_items until UnprocessedItems corresponds to an empty list. Here is a gist containing code that reads a log file in lzop format, looks for lines containing a key + white space + a value, then inserts items based on those key/value pairs into a DynamoDB table. I've been pretty happy with this approach.

Before I finish, I'd like to reiterate the gripe I have about the static nature of determining your Read and Write Throughput when dealing with DynamoDB. I understand that it makes life easier for AWS in terms of the capacity planning they have to do on their end to scale the table across multiple instances, but it's a black art when it comes to capacity planning you need to do as a user. You almost always end up overcommitting as a DynamoDB user, and it's hard to make sense sometimes of the capacity units you're consuming, especially when doing inserts of large volumes of data.

Using DynamoDB BatchWriteItem with boto

This is just a quick note about the advantage of using DynamoDB's newly introduced BatchWriteItem functionality, which allows you to write multiple items at the same time to a table, with the write operation parallelized behind the scenes by DynamoDB. Currently there is a limit of 25 items that can be batch-written or batch-deleted to/from a DynamoDB table.

I was glad to see that the boto library already supports this new feature -- the fact that Mitch Garnaat is now an employee of Amazon probably helps too ;-) You do have to git pull the latest boto code from GitHub, since BatchWriteItem is not available in the latest boto release 2.3.0.

I tested this feature inside a script which was parsing mail logs and uploading lines corresponding to certain regular expressions as items to a DynamoDB table. When I used the standard item-at-a-time method, it took 7 hours to write 2 million items into the table. When using BatchWriteItem, it only took 26 minutes -- so a 16x improvement.

Here's how I used this new functionality with boto:

1) I created a DynamoDB connection object and a table object:

dynamodb_conn = boto.connect_dynamodb(aws_access_key_id=MY_ACCESS_KEY_ID, aws_secret_access_key=MY_SECRET_ACCESS_KEY)

mytable = dynamodb_conn.get_table('mytable')

2) I created a batch_list object:

batch_list = dynamodb_conn.new_batch_write_list()

3) I populated this object with a list of DynamoDB items:

batch_list.add_batch(mytable, puts=items)

where items is a Python list containing item objects obtained via


mytable.new_item(attrs=item_attributes)

4) I used the batch_write_item of the layer2 module in boto to write the batch list:

dynamodb_conn.batch_write_item(batch_list)

That was about it. I definitely recommend using BatchWriteItem whenever you can, for the speedup it provides.

Initial experiences with Amazon DynamoDB

I've been experimenting a bit with Amazon DynamoDB -- the "fully managed NoSQL database service that provides fast and predictable performance with seamless scalability" according to Amazon -- in order to see how easy to use it is, and what kind of performance you can get out of it. My initial impressions are favorable, with some caveats.

Defining tables

To get started with DynamoDB, you can use the AWS Console web interface. You need to define a table by giving it a name. Then you need to define a hash key, which enables DynamoDB to build an unordered hash index for partitioning and querying purposes. You can also define a range key, in which case DynamoDB will build an unordered hash index on the hash key, and an sorted range index on the range key. For most intents and purposes, the range key will be some sort of timestamp-related attribute of your data. You can find out more details in the DynamoDB Data Model documentation.

The most confusing part when defining a table is coming up with Read Throughput and Write Throughput values for the table. This revolves around estimating the capacity units you will need when reading from and writing to that table. Here is an excerpt from the "Capacity Unit Calculations" documentation:

For each request you send, Amazon DynamoDB computes the capacity units consumed by that operation. Item size is one of the factors it uses in computing the capacity units consumed. The size of an item is sum of lengths of its attribute names and values. This section describes how Amazon DynamoDB determines the size of item(s) involved in an operation.

The get, put, and delete operations involve one item. However, batch get, query and scan operations can return multiple items.

For operations that involve only one item, Amazon DynamoDB rounds the item size up to the next 1 KB. For example, if you get, put, or delete an item of 1.6 KB in size, Amazon DynamoDB rounds the items size to 2 KB. This rounding also applies to batch get operation, which operates on several items. Amazon DynamoDB rounds the size of each individual item returned in the batch. For example, if you use the batch get operation to retrieve 2 items of 1.2 KB and 3.6 KB, Amazon DynamoDB rounds these items sizes to 2 KB and 4 KB respectively, resulting a total size for the operation of 6 KB.

A query or scan can return multiple items. By default Amazon DynamoDB returns up to 1 MB of items for a query or scan. In this case Amazon DynamoDB computes the total item size for the request by computing the sum of all items sizes and then rounding to the next KB. For example, suppose your query returns 10 items whose combined size is 10.2 KB. Amazon DynamoDB rounds the item size for the operation to 11 KB, for the purpose of computing capacity units consumed by that operation. Note that unlike for single item operations, this size is not necessarily proportional to the number of items. Instead it is the cumulative size of processed items, rounded up to the next KB increment. For example, if your query returns 1,500 items of 64 bytes each, the cumulative size is 94 KB, not 1,500 KB.

In case of a scan operation, it is not the size of items returned by scan, rather it is the size of items evaluated by Amazon DynamoDB. That is, for a scan requests, Amazon DynamoDB evaluates up to 1 MB of items and returns only the items that satisfy the scan condition.

So what this means in practice is that you need to estimate your read and write operations/second, and multiply them by N, where N is the item size in KB. Of course, you need to err on the conservative size and overestimate, at least initially, which works in Amazon's favor, since they bill you by your estimated capacity and not by your actual consumed capacity. This is one caveat I have when using DynamoDB. I find it a bit odd that the billing process is not more dynamic and usage-based, as is the case with pretty much all the other AWS offerings.

However, once you start using the service, you can see the actual usage by looking at the Monitoring tab in the AWS console. Both Consumed Read Capacity Units and Consumed Write Capacity Units are shown, which does enable you to tweak your throughput values on the fly and bring them closer to the actual usage. You can also define monitoring alerts that will notify you if your read or write throughput is greater than a certain percentage (by default 80%) of the read or write throughput you specified for a given table.

Once you define the table, the keys and the throughput values, you're ready to use the service. If you like Python, you'll be glad to know that the latest version of boto supports DynamoDB very well. I used boto-2.3.0 which I downloaded from here. I used the DynamoDB boto API reference pretty heavily, and also the DynamoDB boto tutorial. There were some slight discrepancies though in the API reference compared to the actual boto 2.3.0 usage, and I'll highlight them below.

My scenario for using DynamoDB was to write items that represent mail log entries. I have a script that tails the mail log every 10 seconds and sends new entries packaged as items to DynamoDB. For this purpose, I created a table called 'maillog', with a hash key called 'email' (which is the destination email) and a range key called 'timestamp_sent' (which represents the time when that mail message was sent to its destination; if the message is deferred and the mail server retries, there will be multiple items sent to DynamoDB, each with their respective timestamp).

Before I go on, I want to mention that boto offers two ways of interacting with DynamoDB: a 'Layer1' class which deals with the low-level format of the DynamoDB API (things such as properly formatting the requests and interpreting the replies), and a 'Layer2' class which encapsulates the Layer1 functionality, and enables you to use higher-level constructs in your code. The DynamoDB boto tutorial deals exclusively with Layer2 functionality and that's what I am going to cover in my code examples too.

Writing items into a table

The first thing you need to do when interacting with DynamoDB via boto is to open a connection:

dynamodb_conn = boto.connect_dynamodb(aws_access_key_id='YOUR_AWS_ACCESS_KEY_ID', aws_secret_access_key='YOUR_AWS_SECRET_ACCESS_KEY)

Then you create a table object via the get_table method, called with the desired table name as a parameter:

maillog_table = dynamodb_conn.get_table('maillog')

You write items into DynamoDB, so you need to construct the proper item object, which is based on a Python dictionary. In my case, I use some complicated regular expressions to parse the mail log lines and extract the elements I am interested in, such as source mail server, relay, message id, source email, destination email, received timestamp, sent timestamp, sent status. So my item dictionary looks something like this:

item_dict = {
'email': dest_email, 
'timestamp_sent': timestamp_sent,
'msgid': msgid,
'timestamp_rcvd': timestamp_rcvd,
'source_email': src_email,
'size': size,
'delay': delay, 
'xdelay': xdelay, 
'relay_sent': relay, 
'status': stat,

}

Note that the hash key name ('email') and range key name ('timestamp_sent') are part of the keys of the dictionary. 

To actually create a DynamoDB item and write it into the database, I used the new_item and put methods on the table object:

item = maillog_table.new_item(attrs=item_dict)

item.put()

I could have also passed hash_key and range_key as parameters to new_item, like this:

item = maillog_table.new_item(hash_key=dest_email, range_key=timestamp_sent, attrs=item_dict)

But as long as those 2 keys are part of the item_dict dictionary, the new_item method is happy.

Note that the item.put() method returns a dictionary of the form:

{'ConsumedCapacityUnits': 1.0}

which is very useful in determining your actual write throughput.

Reading items, querying and scanning tables

The easiest way to read an item is to use the get_item method on the table object. This requires that you pass the hash key, and also the range key if you have one defined for that table. In my example, I do it like this:

item = table.get_item(hash_key=email, range_key=timestamp)

print item['timestamp_sent'], item['email'], item['status']

For my purposes though, after I save the mail log entries in DynamoDB, I want to query the table for a specific destination email, based on a certain time range. To do that, I use the query method on the table object.

items = table.query(hash_key=email, range_key_condition=BETWEEN(timestamp1, timestamp2))

for item in items:

    print item['timestamp_sent'], item['email'], item['status']

Note that I used a class named BETWEEN, which I needed to import:


from boto.dynamodb.condition import LE, EQ, GE, BETWEEN

This is where the online documentation for boto strays from the actual code. The documentation says that range_key_condition is a dict, when in fact it needs to be a condition object such as LE, GE, EQ etc.

The query method is very fast. A slower way to retrieve items, which is not recommended, is to scan the table based on some attributes of the items saved in the table. For example, if I wanted to scan all mail log entries sent within a time range, I could do:

LIMIT = 20

items = table.scan(scan_filter={'timestamp_sent': BETWEEN(timestamp1, timestamp2)}, request_limit=LIMIT)

for item in items:

    print item['timestamp_sent'], item['email'], item['status']

I needed to specify a scan_filter parameter, which is a dictionary with the key being the attribute you want to scan on, and with the value for that key being the condition that defines your scan. 

Whenever you run a scan, it is a very good idea to specify a request_limit parameter in the scan method. This will tell DynamoDB to only retrieve at the most that many items at a time. If you don't specify a limit, the scan operation will attempt to retrieve 1 MB worth of items at a time, which means 1024 items if each item is less than 1 KB. This would normally result in 1024 Consumed Read Capacity Units, but this number gets halved if you don't specify that you want a consistent read type of scan, but instead you go with the default of 'eventually consistent'. In my case, the scan operation with no request_limit resulted in 512.5 Consumed Read Capacity Units, which was way past my specified Read Throughput of 20 for my maillog table. In consequence, I got a monitoring alert of the type:

You are receiving this email because your Amazon CloudWatch Alarm "maillog-ReadCapacityUnitsLimit-xbzrl3i5zunh" in the US - N. Virginia region has entered the ALARM state, because "Threshold Crossed: 1 datapoint (512.5) was greater than or equal to the threshold (16.0)." at "Thursday 12 April, 2012 20:06:50 UTC".

(the threshold which was crossed is 16.0, which is 80% of 20.0, which is the Read Throughput I specified for my table)

When the request_limit is specified, the scan operation will retrieve that many items at a time, and will continue until the scan filter condition is met. This continuation of item retrieval is done at the Layer2 of boto via a mechanism involving a data structure called LastEvaluatedKey, obtained from the previous item retrieval. The LastEvaluatedKey looks something like this in my example:

{'RangeKeyElement': '20120405053639', 'HashKeyElement': 'email@example.com'} 

One functionality I wish was exposed is the exact number of Capacity Units consumed during a scan or query operation. That information is available to the Layer2 query or scan method, in a response object which is not however exposed via the boto API. I raised this issue with Mitch Garnaat, the author of boto, and he kindly opened a Github issue which hopefully will make it into a new feature.

In order to glean that information myself during a scan operation, I hacked the boto/dynamodb/layer2.py module and changed the yield statement in the scan and query methods from

yield item_class(table, attrs=item)

to

yield response['ConsumedCapacityUnits'], item_class(table, attrs=item)

This enabled me to troubleshoot the mysterious (at first) boto exception of the form

boto.exception.BotoServerError: BotoServerError: 400 Bad Request

It turns out you get this exception during a table scan or query operation when your consumed read capacity units are more than your defined read throughput.

I printed the ConsumedCapacityUnits when I specified various values for request_limit in the scan operation. For request_limit = 100, I was getting 19.0 consumed read capacity units, just slightly below my specified read throughput of 20. This is with consistent_read parameter set to the default value of False. As a rule of thumb though, if you keep consistent_read = False, you should be OK with a limit set to twice your table's read throughput.

In my tests, query operations only took 0.5 read capacity units when consistent_read is False, and 1.0 read capacity units when consistent_read is True.

That's about it for now. I haven't really started to use the DynamoDB service in anger, so I don't have any more data points. From what I've seen so far, DynamoDB is good at operations such as getting a specific item, or querying by hash key and range key. It is not good at reporting operations that involve scanning a table. For that, my intention is to use DynamoDB in conjunction with Elastic MapReduce, specifically with Hive. Stay tuned for another blog post on that soon.

Dynamic DNS updates with nsupdate and BIND 9

I first saw nsupdate mentioned on the devops-toolchain mailing list as a tool for dynamically updating DNS zone files from the command line. Since this definitely beats manual editing of zone files, I'd thought I'd give it a try. My setup is BIND 9 on Ubuntu 10.04. I won't go into the details of setting up BIND 9 on Ubuntu -- see a good article about this here.

It took me a while to get nsupdate to work. There are lots of resources out there, but as usual it's hard to separate the grain from the chaff. When everything was said and done, the solution was relatively simple. Here it is.

Generate TSIG keys

# dnssec-keygen -r /dev/urandom -a HMAC-MD5 -b 512 -n HOST myzone.com

This generates 2 files of the form:

-rw-------   1 root bind   122 2012-03-21 15:47 Kmyzone.com.+157+02058.key
-rw-------   1 root bind   229 2012-03-21 15:47 Kmyzone.com.+157+02058.private

Note that I specified /dev/urandom as the source of randomness, which may not meet your security requirements. When I didn't specify the -r /dev/urandom parameter, the dnssec-keygen command appeared to hang.

Also note that the type of the key needs to be HOST (specified via -n HOST).

Add key to DNS master server configuration and allow updates

I modified /etc/bind/named.conf.local and added a 'key' section:

key "myzone.com." {
 algorithm hmac-md5;
 secret "JKlA76FvmGboEQ8R2yoc9AtpFqkIncH5yf2mXY+s8m6a/RRC0thUVGnqrJSO1QKhzlnkbxTjmArap+WuVW9iLQ==";
};

The key name can be anything you want. The secret is the actual key, which can be found in both of the files generated by dnssec-keygen.

I also added an allow-update directive to the zone that I wanted to modify via nsupdate. This is still in /etc/bind/named.conf.local:

zone "myzone.com" {

       type master;
       file "/var/lib/bind/myzone.com.db";
       allow-update { key "myzone.com."; };
};

I then restarted bind9 on the master DNS server via 'service bind9 restart'. I checked /var/log/daemon.log to make sure there were no errors during the restart.

Note that you can use a more finely grained control over which operations you allow for the updates. See the 'Allowing Updates' section in this 'Secure DDNS Howto' document.

Use nsupdate to do remote updates

On a remote trusted host of your choice, copy the private file generated by dnssec-keygen, and create a file containing the desired updates to the zone file on the master. This file is of the form:

# cat nsupdate.txt
server master.dns.server.myzone.com
debug yes
zone myzone.com.
update add testnsupdate.myzone.com. 86400 CNAME ns1
show
send

Then run nsupdate and specify the kddey and the file you just created:

# nsupdate -k Kmyzone.com.+157+02058.private -v nsupdate.txt

If everything goes well, you should see something like this in the debug output of nsupdate (because we specified 'debug yes' in the nsupdate.txt file):

;; UPDATE SECTION:

testnsupdate.myzone.com. 86400 IN CNAME ns1

;; TSIG PSEUDOSECTION:

myzone.com. 0 ANY TSIG hmac-md5.sig-alg.reg.int. 1332788750 300 16 UxiMG7+X2RejWzQ9rkuPaQ== 3305 NOERROR 0 

Reply from update query:

;; ->>HEADER<<- opcode: UPDATE, status: NOERROR, id:   3305

;; flags: qr ra ; ZONE: 0, PREREQ: 0, UPDATE: 0, ADDITIONAL: 1

;; TSIG PSEUDOSECTION:

myzone.com. 0 ANY TSIG hmac-md5.sig-alg.reg.int. 1332788857 300 16 KBubhEggwBHnPlbmlQ7iTw== 3305 NOERROR 0 

On the master DNS server, you should see something like this in /var/log/daemon.log:

Mar 26 12:07:37 dns01 named[14952]: client 10.0.10.133#58265: signer "myzone.com" approved

Mar 26 12:07:37 dns01 named[14952]: client 10.0.10.133#58265: updating zone 'myzone.com/IN': adding an RR at 'testnsupdate.myzone.com' CNAME

Mar 26 12:07:37 dns01 named[14952]: zone myzone.com/IN: sending notifies (serial 2012032104)

Mar 26 12:07:37 dns01 named[14952]: client 10.0.10.121#50790: transfer of 'myzone.com/IN': IXFR started

Mar 26 12:07:37 dns01 named[14952]: client 10.0.10.121#50790: transfer of 'myzone.com/IN': IXFR ended

One other important note: the modifications made with nsupdate take effect immediately on the DNS master server (and they also get pushed from there to slave servers), but they are not written immediately to the actual DNS zone file on disk on the master server. Instead, a journal file is used, in the same directory as the zone file. The journal entries get applied periodically to the main zone file. If you restart bind9, the journal entries also get applied.

That's about it. If everything went well, you now have an API of sorts into your Bind 9 server. Now go automate all the things!

More resources:

• "DNS Security Part 1: Dynamic Updates" by Kerry Thompson
• "DNS Security Part 2: DNSSEC" by Kerry Thompson
• "A DDNS server using BIND and nsupdate"
• "Running a secure DDNS service with BIND"

PostgreSQL dump/restore and client_encoding

I started to look into EnterpriseDB recently. Pretty pleased with it so far. At first I launched the beta version of their Postgres Plus Cloud Database product, but since this version is in the process of being decomissioned, I've had to transfer the database I had already created to a newly created cluster in their DBaaS model -- which basically means that the cluster manager is maintained by them, and the cluster member servers (1 master + N replicas) are part of your EC2 footprint.

In any case, I did a pg_dump of the database from the initial master, then I tried to load the dump via psql into a newly created database on the new master. However, the client_encoding parameter in postgresql.conf was SQL_ASCII on the first master, and UTF8 on the second. This posed a problem. The psql load operation failed with errors of the type:

ERROR:  invalid byte sequence for encoding "UTF8": 0xe92044
CONTEXT:  COPY table1, line 6606
ERROR:  invalid byte sequence for encoding "UTF8": 0xa0
CONTEXT:  COPY table2, line 978
ERROR:  invalid byte sequence for encoding "UTF8": 0xd454
CONTEXT:  COPY table3, line 3295

Obviously the encodings didn't match. I first tried to re-run the pg_dump on the first master (which had client_encode = 'SQL_ASCII') and specified "--encoding utf8" on the pg_dump command line. This didn't do the trick. I had the same exact errors when loading the dump on the second master.

One solution suggested by EnterpriseDB was to set client_encoding to SQL_ASCII on the new master, restart Postgres and retry the load. I found another solution though in a blog post very aptly titled (for my purposes) 'PostgreSQL database migration, the SQL_ASCII to UTF8 problem'. What I ended up doing, following the advice in the post, was to install the GNU recode utility (I did a yum install recode), then run the initial dump through recode, converting it from ascii to utf8. Something like this:

cat dump.sql | recode iso-8859-1..u8 > utf8_withrecode.sql

Then I modified the line

SET client_encoding = 'SQL_ASCII';

and turned it into:

SET client_encoding = 'UTF8';

after which loading the dump into the new master with psql worked like a charm.

Anyway...now for the fun part of doing some load testing against this Postgres cluster!