GRPC in Google Cloud

 

Recently, I've worked on setting up a GRPC endpoint behind the load balancer in the GCP (Google Cloud). That was a new project and in the first iteration we decided to serve it from a VM. The same machine was also the endpoint for the standard HTTP API. I set two load balancers to redirect traffic to each port. Both were healthy, but the GRPC didn't work properly. 

To my great surprise, I learnt that GRPC traffic requires not only HTTP2 protocol, but also a TLS/SSL setup on the server end of the internal connection (LB-VM). It has to be done, despite the fact traffic to be encrypted (with so-called automatic network-level encryption) [3]. The LB documentation kind of indicates the situation [1].  But initial, I could not believe, that Google, who introduce GRCP, could set it in such a messy way, and assumed that I didn't understand documentation properly. The fact that, AI didn't give a clear answer and hallucinated a few scenarios wasn't helpful. Only after a friend of a friend, who had faced the same issue before, confirmed that GCP cannot properly handle GRPC in LB, I found more documentation [2]. 

What really shocking is that the SSL/TLS certificate don't need to be valid at all. It can be an old, self sign. Doesn't matter. Just need to be there! It certainly does not need to be the certificate used by the LB

We automate VMs setup with Ansible [4], so I prepared a small script to set the SSL certificate. It's generic enough to be useful for others with minor adjustment. The code can be found at the end of the article. The "path variables" should be changed. Changing the DNS subject might also be not a bad idea, but probably it's going to work anyway, because the certificate does not need to be a valid one.

Summary

So to have GRPC behind load balancer in GCP you have to:

• Prepare an External Application Load Balancer
• Set HTTPS frontend (e.g. with certificate provided by Google)
  
• Configure backend to be HTTP2 (with GRPC and HTTP2 health check)
• Prepare an SSL certificate and ensure it's present at the end point 
  

Links

1. https://cloud.google.com/load-balancing/docs/https
2. https://cloud.google.com/load-balancing/docs/ssl-certificates/encryption-to-the-backends
3. https://cloud.google.com/load-balancing/docs/https/http-load-balancing-best-practices
4. https://ansible.readthedocs.io/
   

Ansible code

- name: Set  directory
  ansible.builtin.file:
    path: "{{ ivynet_backend_path_secrets }}"
    state: directory
    owner: root
    group: root
    mode: "0700"
  tags:
    - ssl

- name: Check if pem file exists
  ansible.builtin.stat:
    path: "{{ ivynet_backend_path_secrets }}/self.pem"
  register: pem

- block:
  - name: Create private key (RSA, 4096 bits)
    community.crypto.openssl_privatekey:
      path: "{{ ivynet_backend_path_secrets }}/self.key"
    tags:
      - ssl

  - name: Create certificate signing request (CSR) for self-signed certificate
    community.crypto.openssl_csr_pipe:
      privatekey_path: "{{ ivynet_backend_path_secrets }}/self.key"
      common_name: self.ivynet.dev
      organization_name: IvyNet
      subject_alt_name:
        - "DNS:grpc.test.ivynet.dev"
        - "DNS:self.test.ivynet.dev"
        - "DNS:test.ivynet.dev"
    register: csr
    tags:
      - ssl

  - name: Create self-signed certificate from CSR
    community.crypto.x509_certificate:
      path: "{{ ivynet_backend_path_secrets }}/self.pem"
      csr_content: "{{ csr.csr }}"
      privatekey_path: "{{ ivynet_backend_path_secrets }}/self.key"
      provider: selfsigned
    tags:
      - ssl
  when:
    - not pem.stat.exists