IvyNet DevOps Projects

Before IvyNet shut down, we decided to open-source the code. There are a few DevOps bits, which might be useful for others.

I believe documentation is important, and the README is a good starting point to see what's there. Good documentation doesn't have to be long or very detailed, so you'll find links to the actual code (e.g. Ansible, GitHub Actions, pre-commit, OpenTofu/Terraform, or Packer) plus some extra explanations.

Other things I'd like to point out:

The repository with OpenTofu/Terraform modules includes pre-commit and GitHub Actions (GHA). Each module has tests, documentation, and proper versioning. Some of the test scenarios are quite complex because they require extra components to run. https://github.com/ivy-net/otofu-modules 

The OpenTofu infrastructure definition is separated from the modules. This separation makes it easy to upgrade environments independently. Each can be upgraded separately, so you can test things in staging or dev before touching production.  https://github.com/ivy-net/infra

  
The applications are written in Rust, and the GitHub Actions and pre-commit setup could be a good starting point for Rust project automation. 

GRPC in Google Cloud

 

Recently, I've worked on setting up a GRPC endpoint behind the load balancer in the GCP (Google Cloud). That was a new project and in the first iteration we decided to serve it from a VM. The same machine was also the endpoint for the standard HTTP API. I set two load balancers to redirect traffic to each port. Both were healthy, but the GRPC didn't work properly. 

To my great surprise, I learnt that GRPC traffic requires not only HTTP2 protocol, but also a TLS/SSL setup on the server end of the internal connection (LB-VM). It has to be done, despite the fact traffic to be encrypted (with so-called automatic network-level encryption) [3]. The LB documentation kind of indicates the situation [1].  But initial, I could not believe, that Google, who introduce GRCP, could set it in such a messy way, and assumed that I didn't understand documentation properly. The fact that, AI didn't give a clear answer and hallucinated a few scenarios wasn't helpful. Only after a friend of a friend, who had faced the same issue before, confirmed that GCP cannot properly handle GRPC in LB, I found more documentation [2]. 

What really shocking is that the SSL/TLS certificate don't need to be valid at all. It can be an old, self sign. Doesn't matter. Just need to be there! It certainly does not need to be the certificate used by the LB

We automate VMs setup with Ansible [4], so I prepared a small script to set the SSL certificate. It's generic enough to be useful for others with minor adjustment. The code can be found at the end of the article. The "path variables" should be changed. Changing the DNS subject might also be not a bad idea, but probably it's going to work anyway, because the certificate does not need to be a valid one.

Summary

So to have GRPC behind load balancer in GCP you have to:

• Prepare an External Application Load Balancer
• Set HTTPS frontend (e.g. with certificate provided by Google)
  
• Configure backend to be HTTP2 (with GRPC and HTTP2 health check)
• Prepare an SSL certificate and ensure it's present at the end point 
  

Links

1. https://cloud.google.com/load-balancing/docs/https
2. https://cloud.google.com/load-balancing/docs/ssl-certificates/encryption-to-the-backends
3. https://cloud.google.com/load-balancing/docs/https/http-load-balancing-best-practices
4. https://ansible.readthedocs.io/
   

Ansible code

- name: Set  directory
  ansible.builtin.file:
    path: "{{ ivynet_backend_path_secrets }}"
    state: directory
    owner: root
    group: root
    mode: "0700"
  tags:
    - ssl

- name: Check if pem file exists
  ansible.builtin.stat:
    path: "{{ ivynet_backend_path_secrets }}/self.pem"
  register: pem

- block:
  - name: Create private key (RSA, 4096 bits)
    community.crypto.openssl_privatekey:
      path: "{{ ivynet_backend_path_secrets }}/self.key"
    tags:
      - ssl

  - name: Create certificate signing request (CSR) for self-signed certificate
    community.crypto.openssl_csr_pipe:
      privatekey_path: "{{ ivynet_backend_path_secrets }}/self.key"
      common_name: self.ivynet.dev
      organization_name: IvyNet
      subject_alt_name:
        - "DNS:grpc.test.ivynet.dev"
        - "DNS:self.test.ivynet.dev"
        - "DNS:test.ivynet.dev"
    register: csr
    tags:
      - ssl

  - name: Create self-signed certificate from CSR
    community.crypto.x509_certificate:
      path: "{{ ivynet_backend_path_secrets }}/self.pem"
      csr_content: "{{ csr.csr }}"
      privatekey_path: "{{ ivynet_backend_path_secrets }}/self.key"
      provider: selfsigned
    tags:
      - ssl
  when:
    - not pem.stat.exists
 

Summary of a Terraform plan output

One of the most annoying thing when working with terraform is the size of output of the terraform plan command. For more complex environments, it easily can get to many thousand lines, even for what seems to be a small change.  It makes very hard to confirm that a code change does not have a side effects.

It would be nice to have the summary option, showing only resources and modules changed. I guess one day such feature will be added. In the meantime, I thought to use the grep command on the terraform plan output. It wasn't easy, because the output contain a few control character. After quite a few attempts, I found that following regex is a substitute.

terraform plan | grep -E "^[[:cntrl:]][[:print:]]+[[:space:]]+#\ "

How to find s3 bucket in multiple accounts (with awk and multiple field separator)

Imagine you have quite a few AWS accounts. In one of them, you don't know which one, there is an S3 bucket. The AWS CLI with awk and zsh can help to find it.

In the first step, let's prepare a list of all accounts, or rather profiles from the AWS CLI config (the ~/.aws/config file).

accounts=($(awk -F "( |])" '/profile sso/ {print $2}'  ~/.aws/config))

In the example, we limit the list only to profiles with the prefix "sso". The command uses awk to find any line with the string "profile sso" and print the second field from it. However, it does no use the standard field separator. There are 3 characters working as a separator: space, "|" and "]". Please also note the awk command is two pairs of "()".

 The list is saved into the accounts variable and used in the second command. It lists all s3 buckets from each account, and grep for the selected string, which of course can be the whole bucket name.

p=sso-prod 

bucket=my-company-not-so-important-bucket

for accounts ($accounts) {echo $account; aws s3 ls --profile $p| grep $bucket}

Jenkins and Splunk (Cloud)

There is the Jenkins Plugin to set log export to Splunk. In Splunk Marketplace you can find the Splunk App for Jenkins, but the app works only for Splunk Enterprise, not the Cloud edition. Documentation describe how to connect Jenkins to Splunk. It includes values for cloud deployment as well as on-premises one. Thanks to that, the connection was easy to establish, but there were no logs from Jenkins. I had to manually created 4 indexes (I got them from this discussion) to get Jenkins logs visible in Splunk.

Get Public IP address of Azure VM from shell on VM - part II

Last year I wrote a quick note with a method of getting Azure VM public address from the shell on VM. Recently I spent a bit more time to look at https://docs.microsoft.com/en-us/azure/virtual-machines/linux/instance-metadata-service and found more precise call to get IP address from the metadata url: 

curl -H Metadata:true \
 "http://169.254.169.254/metadata/instance/network/interface/0/ipv4/ipAddress/0/publicIpAddress?api-version=2017-08-01&format=text"

BTW. The same method allows to get a private IP address.

curl -H Metadata:true \
 "http://169.254.169.254/metadata/instance/network/interface/0/ipv4/ipAddress/0/privateIpAddress?api-version=2017-08-01&format=text"

Get Public IP address of Azure VM from shell on VM

Sometimes you need to get an IP address of the VM from inside of it. You can do this relatively simple from Azure VM with curl and jq thanks to Metadata endpoint as described in this document https://azure.microsoft.com/en-us/blog/announcing-general-availability-of-azure-instance-metadata-service/

And if you need the public IP address of interface eth0 this is the command:

curl -H Metadata:true http://169.254.169.254/metadata/instance?api-version=2017-04-02| jq '.network.interface[0].ipv4.ipAddress[0].publicIpAddress'

Terraform debugging (Azure Storage long names)

I had strange problem with Terraform on Azure. Everything looked good, but ever time I got. I tried to changed few things. Didn't help. All the time:

* module.storage.azurerm_storage_container.vhds: 1 error(s) occurred:

* module.storage.azurerm_storage_container.vhds: Resource \
'azurerm_storage_account.vhds' not found for variable \
'azurerm_storage_account.vhds.name'

I couldn't find any explanation to me problems, so I started to look to increase amount of information from Terraform. I found that TF_LOG controls log level. I ran

TF_LOG=TRACE terraform plan| grep ERROR

And found following line:

2017/10/27 20:31:37 [ERROR] root.storage: eval: *terraform.EvalValidateResource, \
err: Warnings: []. Errors: [name can only consist of lowercase letters and numbers, \
and must be between 3 and 24 characters long]

Yes, the create name was a bit long, but aaaa... The original error message is not the most obvious one.

Many AWS accounts and Zsh

That might not be a common problem, but I have to deal with many AWS accounts in the same time. For example I might to have to run an Ansible playbook for one account and a CLI commend for other one in parallel. To make my life easier i wrote this ZSH function. (It's probably going to work in other advance shell).

There are two ways to use it.

• awsenv list - returns the list of all available accounts/environments.
• awsenv - sets, based on values provided in ~/.aws/credentaials and ~/.aws/config, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_DEFAULT_REGION.

The list is create with simple AWK script (assuming that any lines with "[]" is OK.

The actual command to set environment uses two AWK scripts. First one looks for requested account name and set variable "a" to 1. When "a" equals 1 it prints shell export command for access_key_id and for secret_access_key to  standard output, which is redirected to $TMP_FILE. Then it sources, prints and deletes that file.

Please note that in current form script requires access_key_id being define before secret_access_key.  Printing the value of all variables, especially secret_aceess_key could be consider as security weakness, so you might want to modify/remove "cat $TMP_FILE line.

# vim: set filetype=sh
awsenv () {
    if [[  $1 == list ]]
        then
        print $1
        awk  '/\[.*\]/ {print $1}'  ~/.aws/credentials
    else
        TMP_FILE=/tmp/current_aws
        awk \
           'BEGIN{a=0};\
           /\['$1'\]/ {a=1};\
           /access_key_id/ {if (a==1){printf "export %s=%s\n", toupper($1), $3}};\
           /secret_access_key/ {if (a==1) {printf "export %s=%s\n", toupper($1), $3;a=0}}'\
            ~/.aws/credentials > $TMP_FILE
        awk \
            'BEGIN{a=0};\
            /\[profile '$1'\]/ {a=1};\
            /region/ {if (a==1){printf "export AWS_DEFAULT_%s=%s\n", toupper($1), $3; a=0}}'\
            ~/.aws/config >> $TMP_FILE
        source $TMP_FILE
        cat $TMP_FILE
        rm $TMP_FILE
    fi
}


Finally, initial version of this script and some discusion on profiles in older boto versions is in this post http://larryn.blogspot.co.uk/2015/03/how-to-deal-with-aws-profiles.html

Real "Get All Launch Configuration" In Boto

During my recent Ansible tests I created 'some' number of launch configuration, enough to reach account limit and I wanted/had to clean it. Boto sounded like a good candidate to do this. This few lines should address my problem.

import boto
import boto.ec2
import boto.ec2.autoscale

asg = boto.ec2.autoscale.connect_to_region('us-east-1')
results = asg.get_all_launch_configurations()

But it didn't. I could even found my launch configurations in the results sets. I figured out quickly that my results set is rather big and by default get_all_lunch_configurations method paging results (AFAIR default value is 20). Using example from this post on SDB I create following function doing what above method name promises - get all lunch configuration.


def get_all_launch_configuration(connection)
    """get_all_launch_configuration(connection) -
       returns results set of all launch configuration,
       regardless it size. Function require established
       boto.ec2.autoscaling connection."""

    results = connection.get_all_launch_configurations()
    token = results.next_token
    while True:
        if token:
            r =  asg.get_all_launch_configurations(next_token=token)
            token = r.next_token
            results.extend(r)
        else:
            break
    return results
       

How to deal with AWS profiles

I don't know how common it is to be a part of an organisation having many AWS (Amazon Web Services) accounts, but it's make things tricky. Amazon make it relatively easy to use many 'named profiles' (account) with AWS CLI. (If you haven't try see this documentation). Boto (python AWS interface) developers also added easy way to use the same profiles in version 2.29. (How to use the same profiles in Boto and other SDKs check this article.) But release 2.29 is not so old and what if you got stacked with older version (for example the one from the latest Ubuntu LTS)? I was in such situation and wrote this small function to use it with profiles from the ~/.boto (not ~/.aws/) file.

def set_account(environment):
    """set_credentials(environment) -
    sets credentials for given environment/account.
    """
    for i in boto.config.items(environment):
        boto.config.set('Credentials', i[0], i[1])

So if your profile is called 'prod':

import boto

set_account('prod') 
conn = boto.connect_ec2()

Another program having issues with many AWS accounts is Ansible. (However, authors claims it's a feature not a bug). My first approach was to add the above function in the ec2.py inventory script and  further extended it by adding following lines:

• to the __init__ method of the Ec2Inventory class:

   
  set_credentials(self.args.environment)
   

• and to the parse_cli function:
  

   
  parser.add_argument('-e', '--environment', type=str, required=True,
                             help="select an environment/profile to run.")
   
  

Such prepared script is not ready to use with Ansible yet - a wrapper around it is needed as well. For example the bash script called ec2-prod.sh to use Ansible in the 'prod' environment.

#!/bin/sh
cd $(dirname $0)
./ec2.py -e prod --refresh-cache

If you need to know why the wrapper is needed check the Ansible inventory code.

Such approach is not ideal if you have many account to work with, you will need a wrapper for each one. What worse, it doesn't work with unified AWS config approach and require to keep a unique version of the inventory script. Therefore, I tried to find a better resolution. I could not find anything interesting and decided to write a small shell script to read ~/.aws/credentials and exports AWS keys for selected profile. The script is a simple wrapper around a bit complicated awk command. To use it you have to source, not execute, it, because the script should execute in a current shell.

#!/bin/bash

TMP_FILE=/tmp/current_aws
awk  \
 'BEGIN{a=0};\
 /\['$1'\]/ {a=1};\
 /access_key_id/ {if (a==1){printf "export %s=%s\n", toupper($1), $3}};\
 /secret_access_key/ {if (a==1) {printf "export %s=%s\n", toupper($1), $3;a=0}}'\
 ~/.aws/credentials > $TMP_FILE

source $TMP_FILE
rm $TMP_FILE

The script ensure that:

• AWS_ACCESS_KEY_ID
• AWS_SECRET_ACCESS_KEY

are exported. This works not only with Ansible, but also can simplified AWS CLI usage, because the --profile option can be dropped. It might also help with other tools.

Links:

• http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html
• http://blogs.aws.amazon.com/security/post/Tx3D6U6WSFGOK2H/A-New-and-Standardized-Way-to-Manage-Credentials-in-the-AWS-SDKs
• http://boto.readthedocs.org/en/latest/
• http://www.ansible.com/home
• http://docs.ansible.com/intro_dynamic_inventory.html

Datadog and many dataseries stacked together

Recently, I've started to use Datadog. It has nice features, but I have also found some annoying lacks. One of them is no easy way to prepare a graph with a stack of different series in one graph, for example nice representation of CPU time spent in different states.

Luckily, as you can see above it can be done. You just need to change some things in JSON and have something similar to what I got below. The main point is to have all dataseries in the argument of one "q".

{
  "viz": "timeseries",
  "requests": [
    {

      "q": "avg:system.cpu.system{host:host-01}, avg:system.cpu.user{,host:host-01}, avg:system.cpu.iowait{host:host-01}, avg:system.cpu.stolen{host:host-01}, avg:system.cpu.idle{host:host-01}",
    },
      "type": "area"
  ],
  "events": []
}