Wednesday, January 19, 2011

Passing user data to EC2 Ubuntu instances with libcloud

While I'm on the topic of libcloud, I've been trying to pass user data to newly created EC2 instances running Ubuntu. The libcloud EC2 driver has an extra parameter called ex_userdata for the create_node method, and that's what I've been trying to use.

However, the gotcha here is that the value of that argument needs to be the contents of the user data file, and not the path to the file.'s what worked for me:

1) Created a test user data file with following contents:

apt-get update
apt-get install -y munin-node python2.6-dev
hostname coolstuff

2) Used the following script to create the node (I also created a keypair which I passed to create_node as the ex_keypair argument):
#!/usr/bin/env python

import os, sys
from libcloud.types import Provider 
from libcloud.providers import get_driver 
from libcloud.base import NodeImage, NodeSize, NodeLocation
EC2_ACCESS_ID     = 'MyAccessID'
EC2_SECRET_KEY    = 'MySecretKey'
EC2Driver = get_driver(Provider.EC2) 

keyname = sys.argv[1]
resp = conn.ex_create_keypair(name=keyname)
key_material = resp.get('keyMaterial')
if not key_material:
private_key = '/root/.ssh/%s.pem' % keyname
f = open(private_key, 'w')
f.write(key_material + '\n')
os.chmod(private_key, 0600)

ami = "ami-88f504e1" # Ubuntu 10.04 32-bit
i = NodeImage(id=ami, name="", driver="")
s = NodeSize(id="m1.small", name="", ram=None, disk=None, bandwidth=None, price=None, driver="")
locations = conn.list_locations()
for location in locations:
    if == 'us-east-1b':

userdata_file = "/root/proj/test_libcloud/"
userdata_contents = "\n".join(open(userdata_file).readlines())

node = conn.create_node(name='tst', image=i, size=s, location=location, ex_keyname=keyname, ex_userdata=userdata_contents)
print node.__dict__

3) Waited for the newly created node to get to the Running state, then ssh-ed into the node using the key I created and verified that munin-node and python2.6-dev were installed, and also that the hostname was changed to 'coolstuf'.
# ssh -i ~/.ssh/lc1.pem ubuntu@domU-12-31-38-00-2C-3B.compute-1.internal

ubuntu@coolstuff:~$ dpkg -l | grep munin
ii  munin-common                      1.4.4-1ubuntu1                    network-wide graphing framework (common)
ii  munin-node                        1.4.4-1ubuntu1                    network-wide graphing framework (node)

ubuntu@coolstuff:~$ dpkg -l | grep python2.6-dev
ii  python2.6-dev                     2.6.5-1ubuntu6                    Header files and a static library for Python

ubuntu@coolstuff:~$ hostname

Anyway....hope this will be useful to somebody one day, even if that somebody is myself ;-)


Anonymous said...

I recommend

userdata_contents = open(userdata_file).read()

It is simpler and doesn't insert new blank lines between every two lines of text.

Also, I wonder if you need to add 'coolstuff' to /etc/hosts. Or is it a standin for a name that's actually DNS-resolvable?

Grig Gheorghiu said...

Hi Marius -- thanks for the comment. You're right, open(filename).read() is a better solution. As far as 'coolstuff' it was just a test for my user data script, I wanted to see that 'stuff' happens at boot time when I launch the new instance. In real life I would add it either to /etc/hosts, or to DNS.

Modifying EC2 security groups via AWS Lambda functions

One task that comes up again and again is adding, removing or updating source CIDR blocks in various security groups in an EC2 infrastructur...