Search This Blog

Showing posts with label Terraform. Show all posts
Showing posts with label Terraform. Show all posts

Saturday, February 21, 2026

IvyNet DevOps Projects

Before IvyNet shut down, we decided to open-source the code. There are a few DevOps bits, which might be useful for others.

I believe documentation is important, and the README is a good starting point to see what's there. Good documentation doesn't have to be long or very detailed, so you'll find links to the actual code (e.g. Ansible, GitHub Actions, pre-commit, OpenTofu/Terraform, or Packer) plus some extra explanations.

Other things I'd like to point out:

The repository with OpenTofu/Terraform modules includes pre-commit and GitHub Actions (GHA). Each module has tests, documentation, and proper versioning. Some of the test scenarios are quite complex because they require extra components to run. https://github.com/ivy-net/otofu-modules 

The OpenTofu infrastructure definition is separated from the modules. This separation makes it easy to upgrade environments independently. Each can be upgraded separately, so you can test things in staging or dev before touching production.  https://github.com/ivy-net/infra

  
The applications are written in Rust, and the GitHub Actions and pre-commit setup could be a good starting point for Rust project automation. 

Tuesday, August 20, 2024

Pain with Names (of computing resources)

 

The good naming convention of computing resources (API, functions, ...) is an important aspect of code usability. Ideally, the names are short, but descriptive, follow some kind of conventions and allows people, who read or use code/product, intuitive usage, and easy documentation search. The Terraform (or OpenTofu) GCP (Google Cloud Platform) Provider is an example on how a small 'paper cuts' might make the experience painful.

Resources in GCP, at least some of them, can be global or regional. Each 'type' has the own API call (e.g. there is 'healthChecks', and 'regionHealthChecks'). Terraform follows the same convention. That's the first design decision to discuss. I, as the end user, would prefer GCP, or at least Terraform as a higher level language, to put them together in one resource type (e.g. healthCheck) and have an option inside in the resource to switch between them. However, Terraform covers a lot of providers, so I can guess (haven't searched the answer) there is a policy to translate underling API in the most direct way.

The real pain, is that there are no way to easy, general way to distinguish between regional and global resources. As mentioned above, default health check is global and regional has the 'region' prefix, but for the addresses the endpoint 'address' is regional and the global one has the prefix (globalAddress). Terraform follows. I don't know why API introduces the chaos, but I would much appreciate, if Terraform introduced an order. For example, everything what is regional, has a prefix region. If authors afraid of introducing problems by cross naming API endpoints and Terraform resources (e.g. a global_address become address and address: address_region), the solution could be to introduce a prefix for both types. Then we would have global_address and region_address.

Another problem is usage of prefixes rather than postfixes. As a user, I first look to create a load balancer and then decides if I want to make it global or regional. API and Terraform resources should make it easier for me, by exposing the more important resource feature first, on the left, e.g. address_global and address_region. Maybe global_address sounds better, but this is not a poetry and users don't read the code aloud. Using prefix makes searching and reading documentation harder, too. Methods and objects are usually listed in the alphabetic order, and with postfixes similar resource are next to each other, healthCheck is next to helthCheckRegion etc. It case of GCP would also make API usage a bit easier. The GCP API follows the Camel function name convention, and we have 'address', but 'globalAddress' ('a' vs 'A' in 'address'). With the postfix, it would be 'address' and 'addressGlobal'. The 'a' in 'address' would be always lowercase.

So what did go wrong for me with all of that? Recently, I've been working on a scenario to deploy a load balancer pointing onto a test VM. That took me too much time. The GCP documentation is rich. Quite a lot of APIs have Terraform resources examples, but not all of them. So I was patching them, with extra steps translating Console steps into Terraform using the GCP provider documentation. Some example were for regional resources, others for global one. The provider does not link between them. It took me too much time to figured out the right resource names. It was like this: "The error reads that the regional resource cannot point onto the global one. But why? I don't have any global resources. Oh, in this case, you have to add the word regional." I fixed them one by one. The last error was the address. On the Terraform provided docs page, I put "compute_address" in the search bar. And I read the help page over and over, looking on how to enforce the resource to be global. Finally, not sure how, maybe looking on one of the examples, I noticed that the address endpoint by default is regional, and I needed to add the 'global' prefix. Of course, in the search bar, I could type only 'address'. Unfortunately, for many resources keywords a search returns so many results, that they are not helpful, especially when you start the adventure with a new product.



Sunday, November 12, 2023

Summary of a Terraform plan output

One of the most annoying thing when working with terraform is the size of output of the terraform plan command. For more complex environments, it easily can get to many thousand lines, even for what seems to be a small change.  It makes very hard to confirm that a code change does not have a side effects.

It would be nice to have the summary option, showing only resources and modules changed. I guess one day such feature will be added. In the meantime, I thought to use the grep command on the terraform plan output. It wasn't easy, because the output contain a few control character. After quite a few attempts, I found that following regex is a substitute.

terraform plan | grep -E "^[[:cntrl:]][[:print:]]+[[:space:]]+#\ "

Friday, October 27, 2017

Terraform debugging (Azure Storage long names)

I had strange problem with Terraform on Azure. Everything looked good, but ever time I got. I tried to changed few things. Didn't help. All the time:

* module.storage.azurerm_storage_container.vhds: 1 error(s) occurred:

* module.storage.azurerm_storage_container.vhds: Resource \
'azurerm_storage_account.vhds' not found for variable \
'azurerm_storage_account.vhds.name'

I couldn't find any explanation to me problems, so I started to look to increase amount of information from Terraform. I found that TF_LOG controls log level. I ran

TF_LOG=TRACE terraform plan| grep ERROR

And found following line:

2017/10/27 20:31:37 [ERROR] root.storage: eval: *terraform.EvalValidateResource, \
err: Warnings: []. Errors: [name can only consist of lowercase letters and numbers, \
and must be between 3 and 24 characters long]


Yes, the create name was a bit long, but aaaa... The original error message is not the most obvious one.