Jul 22

Playing with S3 in Python

Programming Python, S3 Add Comment (3)

This is a brief guide to getting up and running against S3 in Python.

You will need the following things:

1. AWS Developer account (you can sign up for it on http://aws.amazon.com)
2. Make sure you sign up for S3 (select Amazon Simple Storage Service in the sidebar on the left of the AWS page and then click ‘Sign Up For This Web Service’ over on the right)
3. Python (I’m assuming you have it set up and working already)

A great library to use for access to AWS is called boto and its homepage is here: http://code.google.com/p/boto. You can download the code directly and then run its setup.py to install it, or if you already have easy_install you can make it do all of the work:

C:boto-1.3a>python setup.py install

or

C:>python easy_install.py boto

Either way, you can verify that boto is properly installed by trying to import it in python:

C:>python
Python 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) [MSC v.1310 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import boto
>>>

Once boto is set up correctly get your AWS identifiers from the “Your Web Services Account” dropdown in the upper right of the AWS pages. You’ll need to use both the Access key and the Secret key when you make the connection.

At the highest level AWS stores data in ‘buckets’. These are unique across the entire service and each account is limited to a max of 100 of them. Within the bucket you create key/data pairs that look like filenames.

The code to do this is super simple. You’ll probably need to replace ‘examplebucket’ with something else:

from boto.s3.connection import S3Connection
from boto.s3.key import Key
conn = S3Connection('', '')
bucket = conn.create_bucket('examplebucket')
k = Key(bucket)
k.key = 'foo'
k.set_contents_from_filename('foo.png')

Doing this will upload the file foo.png to the bucket ‘examplebucket’. That would mean that the file is accessible at: http://s3.amazonaws.com/examplebucket/foo. If you attempt to go to the appropriate URL for that file you’ll get an access denied error back. By default new objects in the bucket aren’t publicly readable. Making the object readable takes advantage of a shortcut built into boto. In the boto.s3.acl module there is a list of ‘ready-to-use’ modes:

CannedACLStrings = ['private', 'public-read', 'public-read-write', 'authenticated-read']

Now we just set the ‘public-read’ ACL on our key and it will become accessible through the URL:

k.set_acl('public-read')

From here, check out the official documentation and tools such as S3Fox that will make your S3 experience much easier.

You should also grab the bota documentation archive and save it locally to use as a reference.