IvyNet DevOps Projects

Before IvyNet shut down, we decided to open-source the code. There are a few DevOps bits, which might be useful for others.

I believe documentation is important, and the README is a good starting point to see what's there. Good documentation doesn't have to be long or very detailed, so you'll find links to the actual code (e.g. Ansible, GitHub Actions, pre-commit, OpenTofu/Terraform, or Packer) plus some extra explanations.

Other things I'd like to point out:

The repository with OpenTofu/Terraform modules includes pre-commit and GitHub Actions (GHA). Each module has tests, documentation, and proper versioning. Some of the test scenarios are quite complex because they require extra components to run. https://github.com/ivy-net/otofu-modules 

The OpenTofu infrastructure definition is separated from the modules. This separation makes it easy to upgrade environments independently. Each can be upgraded separately, so you can test things in staging or dev before touching production.  https://github.com/ivy-net/infra

  
The applications are written in Rust, and the GitHub Actions and pre-commit setup could be a good starting point for Rust project automation. 

GRPC in Google Cloud

 

Recently, I've worked on setting up a GRPC endpoint behind the load balancer in the GCP (Google Cloud). That was a new project and in the first iteration we decided to serve it from a VM. The same machine was also the endpoint for the standard HTTP API. I set two load balancers to redirect traffic to each port. Both were healthy, but the GRPC didn't work properly. 

To my great surprise, I learnt that GRPC traffic requires not only HTTP2 protocol, but also a TLS/SSL setup on the server end of the internal connection (LB-VM). It has to be done, despite the fact traffic to be encrypted (with so-called automatic network-level encryption) [3]. The LB documentation kind of indicates the situation [1].  But initial, I could not believe, that Google, who introduce GRCP, could set it in such a messy way, and assumed that I didn't understand documentation properly. The fact that, AI didn't give a clear answer and hallucinated a few scenarios wasn't helpful. Only after a friend of a friend, who had faced the same issue before, confirmed that GCP cannot properly handle GRPC in LB, I found more documentation [2]. 

What really shocking is that the SSL/TLS certificate don't need to be valid at all. It can be an old, self sign. Doesn't matter. Just need to be there! It certainly does not need to be the certificate used by the LB

We automate VMs setup with Ansible [4], so I prepared a small script to set the SSL certificate. It's generic enough to be useful for others with minor adjustment. The code can be found at the end of the article. The "path variables" should be changed. Changing the DNS subject might also be not a bad idea, but probably it's going to work anyway, because the certificate does not need to be a valid one.

Summary

So to have GRPC behind load balancer in GCP you have to:

• Prepare an External Application Load Balancer
• Set HTTPS frontend (e.g. with certificate provided by Google)
  
• Configure backend to be HTTP2 (with GRPC and HTTP2 health check)
• Prepare an SSL certificate and ensure it's present at the end point 
  

Links

1. https://cloud.google.com/load-balancing/docs/https
2. https://cloud.google.com/load-balancing/docs/ssl-certificates/encryption-to-the-backends
3. https://cloud.google.com/load-balancing/docs/https/http-load-balancing-best-practices
4. https://ansible.readthedocs.io/
   

Ansible code

- name: Set  directory
  ansible.builtin.file:
    path: "{{ ivynet_backend_path_secrets }}"
    state: directory
    owner: root
    group: root
    mode: "0700"
  tags:
    - ssl

- name: Check if pem file exists
  ansible.builtin.stat:
    path: "{{ ivynet_backend_path_secrets }}/self.pem"
  register: pem

- block:
  - name: Create private key (RSA, 4096 bits)
    community.crypto.openssl_privatekey:
      path: "{{ ivynet_backend_path_secrets }}/self.key"
    tags:
      - ssl

  - name: Create certificate signing request (CSR) for self-signed certificate
    community.crypto.openssl_csr_pipe:
      privatekey_path: "{{ ivynet_backend_path_secrets }}/self.key"
      common_name: self.ivynet.dev
      organization_name: IvyNet
      subject_alt_name:
        - "DNS:grpc.test.ivynet.dev"
        - "DNS:self.test.ivynet.dev"
        - "DNS:test.ivynet.dev"
    register: csr
    tags:
      - ssl

  - name: Create self-signed certificate from CSR
    community.crypto.x509_certificate:
      path: "{{ ivynet_backend_path_secrets }}/self.pem"
      csr_content: "{{ csr.csr }}"
      privatekey_path: "{{ ivynet_backend_path_secrets }}/self.key"
      provider: selfsigned
    tags:
      - ssl
  when:
    - not pem.stat.exists
 

Debugging of Ansible and Molecule

As I descrbided in this article additional Ansible options are passed at the end of molecule command. This can be use to increase verbosity of Ansible:

molecule converge -- -vvvv

Molecule own debug information can be printed to the standard output with *--debug** option, but it has be specify without molecule command. For example:

molecule --debug converge

Of course, both can be mixed togther:

molecule --debug converge -- -vvvv

Molecule and extra variables for Ansible

Imagine that you want to use Molecule to test an Ansible role with a not default value. That's just one off test, so adjusting molecule configuration in the converge.yml (or similar) file is not required. It's not very prominent in documentation, but you can pass Ansible variables after "--" (minus, minus) in the molecule commend. For example:

molecule converge -- -e "role_new_var=true"

Many AWS accounts and Zsh

That might not be a common problem, but I have to deal with many AWS accounts in the same time. For example I might to have to run an Ansible playbook for one account and a CLI commend for other one in parallel. To make my life easier i wrote this ZSH function. (It's probably going to work in other advance shell).

There are two ways to use it.

• awsenv list - returns the list of all available accounts/environments.
• awsenv - sets, based on values provided in ~/.aws/credentaials and ~/.aws/config, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_DEFAULT_REGION.

The list is create with simple AWK script (assuming that any lines with "[]" is OK.

The actual command to set environment uses two AWK scripts. First one looks for requested account name and set variable "a" to 1. When "a" equals 1 it prints shell export command for access_key_id and for secret_access_key to  standard output, which is redirected to $TMP_FILE. Then it sources, prints and deletes that file.

Please note that in current form script requires access_key_id being define before secret_access_key.  Printing the value of all variables, especially secret_aceess_key could be consider as security weakness, so you might want to modify/remove "cat $TMP_FILE line.

# vim: set filetype=sh
awsenv () {
    if [[  $1 == list ]]
        then
        print $1
        awk  '/\[.*\]/ {print $1}'  ~/.aws/credentials
    else
        TMP_FILE=/tmp/current_aws
        awk \
           'BEGIN{a=0};\
           /\['$1'\]/ {a=1};\
           /access_key_id/ {if (a==1){printf "export %s=%s\n", toupper($1), $3}};\
           /secret_access_key/ {if (a==1) {printf "export %s=%s\n", toupper($1), $3;a=0}}'\
            ~/.aws/credentials > $TMP_FILE
        awk \
            'BEGIN{a=0};\
            /\[profile '$1'\]/ {a=1};\
            /region/ {if (a==1){printf "export AWS_DEFAULT_%s=%s\n", toupper($1), $3; a=0}}'\
            ~/.aws/config >> $TMP_FILE
        source $TMP_FILE
        cat $TMP_FILE
        rm $TMP_FILE
    fi
}


Finally, initial version of this script and some discusion on profiles in older boto versions is in this post http://larryn.blogspot.co.uk/2015/03/how-to-deal-with-aws-profiles.html

A small example of advance aptitude usage

Few years ago I 'preserved' few example of advance aptitude usage I found on a Debian mailing list. Recently I had a chance to use them again. I was looking for a version of packages with distinguish string in a name (let say it was 'apache').

aptitude versions \$(aptitude search ~iapache| awk '{print \$2}')

• So first I ran aptitude search for 'apache' term, but only among installed packages - aptitude search ~iapache
• From the results I cut package name (second column) - awk '{print \$2}'. Please note I escaped dollar character, because I run this command using Ansible (see this blog entry for more details).
• The outcome of those two command allows me to query package versions - aptitude versions

Links

• http://larryn.blogspot.co.uk/2012/01/some-time-ago-on-debian-mailing-list.html
• http://larryn.blogspot.co.uk/2015/04/the-cassandra-ansible-pipe-and.html

The Cassandra, the Ansible, a pipe and a complicate command

I've been working on Cassandra ring downsizing (not a fanny task). To make things a bit easier I've been using Ansible for configuration management. In theory you don't need to change the value of 'initial_token' in Cassandra configuration, and everything else stays the same during ring resize. Therefore, Ansible is not really necessary, but I believe it's good to have consistence in your configuration.

Cassandra role

In first place I added a variable with the Cassandra ring size as a max_nodes  to its role.

  - {role: cassandra, max_nodes: 13}

This value is used to create a Ansible local fact. To learn more on local facts please read this Curtis Collicutt article, which is the best for my script. Please note that the following code is a Jinja template not ready script.

#!/bin/sh
NODE=$(hostname |grep -o -P "\d+")
TOKEN=$(echo '2^127/{{ max_nodes }}' | bc)
cat <
{
    "node_number": $NODE,
    "token": $TOKEN
}
EOF

The above script print onto standard output a json with two variables.

• node_number - is created based on hostname. The number a hostname correspond to a ring position (i.e. cassandra-4 is forth node).
• token - is 'a main part' of token value calculation. The rest is done in Cassandra configuration template:

    initial_token: {{ '%d' % ( ansible_local.cassandra.token *  (ansible_local.cassandra.node_number - 1)) }}

That's sounds complicated and it is. As often there is a historical explanation to that situation. Initially, the whole calculation was done in the template. It worked for 16 node ring, but not for 15. It didn't work, because division two integral  number (2**127/15) results in float one in jinja what lead to rounding error!

Ansible commands

As a bonus two example of using Ansible to run a bit more complicated command on many hosts.

1. Reads token position from configuration file and initialize node move in a screen session. Please note escape character in front of '$'.
   ansible \
    -i inventory/ec.py \
    -m shell \
    -a "screen -d -m \
      nodetool -h localhost move \
      \$(awk '/initial_token/ {print \$2}' \
        /etc/cassandra/default.conf/cassandra.yaml)"\
   tag_pool_cassandra
   
   
2. Much simpler. Checks that screen runs. AFAIK you have to use shell module if you want to use UNIX pipe.
   ansible \
    -i inventory/ec2.py \
    -m shell \
    -a "ps -ef| grep SCREEN"\
   tag_pool_cassandra
   

How to deal with AWS profiles

I don't know how common it is to be a part of an organisation having many AWS (Amazon Web Services) accounts, but it's make things tricky. Amazon make it relatively easy to use many 'named profiles' (account) with AWS CLI. (If you haven't try see this documentation). Boto (python AWS interface) developers also added easy way to use the same profiles in version 2.29. (How to use the same profiles in Boto and other SDKs check this article.) But release 2.29 is not so old and what if you got stacked with older version (for example the one from the latest Ubuntu LTS)? I was in such situation and wrote this small function to use it with profiles from the ~/.boto (not ~/.aws/) file.

def set_account(environment):
    """set_credentials(environment) -
    sets credentials for given environment/account.
    """
    for i in boto.config.items(environment):
        boto.config.set('Credentials', i[0], i[1])

So if your profile is called 'prod':

import boto

set_account('prod') 
conn = boto.connect_ec2()

Another program having issues with many AWS accounts is Ansible. (However, authors claims it's a feature not a bug). My first approach was to add the above function in the ec2.py inventory script and  further extended it by adding following lines:

• to the __init__ method of the Ec2Inventory class:

   
  set_credentials(self.args.environment)
   

• and to the parse_cli function:
  

   
  parser.add_argument('-e', '--environment', type=str, required=True,
                             help="select an environment/profile to run.")
   
  

Such prepared script is not ready to use with Ansible yet - a wrapper around it is needed as well. For example the bash script called ec2-prod.sh to use Ansible in the 'prod' environment.

#!/bin/sh
cd $(dirname $0)
./ec2.py -e prod --refresh-cache

If you need to know why the wrapper is needed check the Ansible inventory code.

Such approach is not ideal if you have many account to work with, you will need a wrapper for each one. What worse, it doesn't work with unified AWS config approach and require to keep a unique version of the inventory script. Therefore, I tried to find a better resolution. I could not find anything interesting and decided to write a small shell script to read ~/.aws/credentials and exports AWS keys for selected profile. The script is a simple wrapper around a bit complicated awk command. To use it you have to source, not execute, it, because the script should execute in a current shell.

#!/bin/bash

TMP_FILE=/tmp/current_aws
awk  \
 'BEGIN{a=0};\
 /\['$1'\]/ {a=1};\
 /access_key_id/ {if (a==1){printf "export %s=%s\n", toupper($1), $3}};\
 /secret_access_key/ {if (a==1) {printf "export %s=%s\n", toupper($1), $3;a=0}}'\
 ~/.aws/credentials > $TMP_FILE

source $TMP_FILE
rm $TMP_FILE

The script ensure that:

• AWS_ACCESS_KEY_ID
• AWS_SECRET_ACCESS_KEY

are exported. This works not only with Ansible, but also can simplified AWS CLI usage, because the --profile option can be dropped. It might also help with other tools.

Links:

• http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html
• http://blogs.aws.amazon.com/security/post/Tx3D6U6WSFGOK2H/A-New-and-Standardized-Way-to-Manage-Credentials-in-the-AWS-SDKs
• http://boto.readthedocs.org/en/latest/
• http://www.ansible.com/home
• http://docs.ansible.com/intro_dynamic_inventory.html