Recently, I've worked on setting up a GRPC endpoint behind the load balancer in the GCP (Google Cloud). That was a new project and in the first iteration we decided to serve it from a VM. The same machine was also the endpoint for the standard HTTP API. I set two load balancers to redirect traffic to each port. Both were healthy, but the GRPC didn't work properly.
To my great surprise, I learnt that GRPC traffic requires not only HTTP2 protocol, but also a TLS/SSL setup on the server end of the internal connection (LB-VM). It has to be done, despite the fact traffic to be encrypted (with so-called automatic network-level encryption) [3]. The LB documentation kind of indicates the situation [1]. But initial, I could not believe, that Google, who introduce GRCP, could set it in such a messy way, and assumed that I didn't understand documentation properly. The fact that, AI didn't give a clear answer and hallucinated a few scenarios wasn't helpful. Only after a friend of a friend, who had faced the same issue before, confirmed that GCP cannot properly handle GRPC in LB, I found more documentation [2].
What really shocking is that the SSL/TLS certificate don't need to be valid at all. It can be an old, self sign. Doesn't matter. Just need to be there! It certainly does not need to be the certificate used by the LB
We automate VMs setup with Ansible [4], so I prepared a small script to set the SSL certificate. It's generic enough to be useful for others with minor adjustment. The code can be found at the end of the article. The "path variables" should be changed. Changing the DNS subject might also be not a bad idea, but probably it's going to work anyway, because the certificate does not need to be a valid one.
Summary
So to have GRPC behind load balancer in GCP you have to:
- Prepare an External Application Load Balancer
- Set HTTPS frontend (e.g. with certificate provided by Google)
- Configure backend to be HTTP2 (with GRPC and HTTP2 health check)
- Prepare an SSL certificate and ensure it's present at the end point
Links
- https://cloud.google.com/load-balancing/docs/https
- https://cloud.google.com/load-balancing/docs/ssl-certificates/encryption-to-the-backends
- https://cloud.google.com/load-balancing/docs/https/http-load-balancing-best-practices
- https://ansible.readthedocs.io/
Ansible code
- name: Set directory
ansible.builtin.file:
path: "{{ ivynet_backend_path_secrets }}"
state: directory
owner: root
group: root
mode: "0700"
tags:
- ssl
- name: Check if pem file exists
ansible.builtin.stat:
path: "{{ ivynet_backend_path_secrets }}/self.pem"
register: pem
- block:
- name: Create private key (RSA, 4096 bits)
community.crypto.openssl_privatekey:
path: "{{ ivynet_backend_path_secrets }}/self.key"
tags:
- ssl
- name: Create certificate signing request (CSR) for self-signed certificate
community.crypto.openssl_csr_pipe:
privatekey_path: "{{ ivynet_backend_path_secrets }}/self.key"
common_name: self.ivynet.dev
organization_name: IvyNet
subject_alt_name:
- "DNS:grpc.test.ivynet.dev"
- "DNS:self.test.ivynet.dev"
- "DNS:test.ivynet.dev"
register: csr
tags:
- ssl
- name: Create self-signed certificate from CSR
community.crypto.x509_certificate:
path: "{{ ivynet_backend_path_secrets }}/self.pem"
csr_content: "{{ csr.csr }}"
privatekey_path: "{{ ivynet_backend_path_secrets }}/self.key"
provider: selfsigned
tags:
- ssl
when:
- not pem.stat.exists
