How to deal with AWS profiles

I don't know how common it is to be a part of an organisation having many AWS (Amazon Web Services) accounts, but it's make things tricky. Amazon make it relatively easy to use many 'named profiles' (account) with AWS CLI. (If you haven't try see this documentation). Boto (python AWS interface) developers also added easy way to use the same profiles in version 2.29. (How to use the same profiles in Boto and other SDKs check this article.) But release 2.29 is not so old and what if you got stacked with older version (for example the one from the latest Ubuntu LTS)? I was in such situation and wrote this small function to use it with profiles from the ~/.boto (not ~/.aws/) file.

def set_account(environment):
    """set_credentials(environment) -
    sets credentials for given environment/account.
    """
    for i in boto.config.items(environment):
        boto.config.set('Credentials', i[0], i[1])

So if your profile is called 'prod':

import boto

set_account('prod') 
conn = boto.connect_ec2()

Another program having issues with many AWS accounts is Ansible. (However, authors claims it's a feature not a bug). My first approach was to add the above function in the ec2.py inventory script and  further extended it by adding following lines:

• to the __init__ method of the Ec2Inventory class:

   
  set_credentials(self.args.environment)
   

• and to the parse_cli function:
  

   
  parser.add_argument('-e', '--environment', type=str, required=True,
                             help="select an environment/profile to run.")
   
  

Such prepared script is not ready to use with Ansible yet - a wrapper around it is needed as well. For example the bash script called ec2-prod.sh to use Ansible in the 'prod' environment.

#!/bin/sh
cd $(dirname $0)
./ec2.py -e prod --refresh-cache

If you need to know why the wrapper is needed check the Ansible inventory code.

Such approach is not ideal if you have many account to work with, you will need a wrapper for each one. What worse, it doesn't work with unified AWS config approach and require to keep a unique version of the inventory script. Therefore, I tried to find a better resolution. I could not find anything interesting and decided to write a small shell script to read ~/.aws/credentials and exports AWS keys for selected profile. The script is a simple wrapper around a bit complicated awk command. To use it you have to source, not execute, it, because the script should execute in a current shell.

#!/bin/bash

TMP_FILE=/tmp/current_aws
awk  \
 'BEGIN{a=0};\
 /\['$1'\]/ {a=1};\
 /access_key_id/ {if (a==1){printf "export %s=%s\n", toupper($1), $3}};\
 /secret_access_key/ {if (a==1) {printf "export %s=%s\n", toupper($1), $3;a=0}}'\
 ~/.aws/credentials > $TMP_FILE

source $TMP_FILE
rm $TMP_FILE

The script ensure that:

• AWS_ACCESS_KEY_ID
• AWS_SECRET_ACCESS_KEY

are exported. This works not only with Ansible, but also can simplified AWS CLI usage, because the --profile option can be dropped. It might also help with other tools.

Links:

• http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html
• http://blogs.aws.amazon.com/security/post/Tx3D6U6WSFGOK2H/A-New-and-Standardized-Way-to-Manage-Credentials-in-the-AWS-SDKs
• http://boto.readthedocs.org/en/latest/
• http://www.ansible.com/home
• http://docs.ansible.com/intro_dynamic_inventory.html