My dumbass could not think of a better idea to use little free compute

I never thought I needed a personal site but it makes sense now

I never thought I would need a personal site/portfolio till now, because I never thought, I would have enough content to put in it? atleast for the early stage of my career. Then AI happened and made accessible, so everything in the tech space changed. You could easily solo ship cool ideas, ideas that made sense, ideas that were just outright dumb, ideas that had no business of existing, everything accelerated.

Fast-forward to today, I have so much content/projects that I think are cool, so I needed a space to share it. Obviously I could just post on social media and forget it, but with time it fades away. So the next and obvious option, a personal website. I could fully manage it, I could fully control what and how content gets shown in there.

So where should I host my personal site?

Now that we have established that I needed a personal site, next obvious thing was buying a domain. Was looking at a lot of options and where other people put their sites on, the obvious one was just a [name].com, and for the first time in my lifetime the exact domain I wanted was available. So I bought aakashreddy.com.

I was looking for inspiration for the site. I wanted it to be unique not only visually, but uniquely built as well.

On a day like any regular day while doomscrolling on X I found something very interesting https://github.com/nce/oci-free-cloud-k8s. This has a detailed instructions of setting up a free Kubernetes Cluster leveraging the always free tier of OCI. I was immediately interested as this is the setup I usually work with for m job, so leveraging it on a portfolio would be cool, as it reflects the tech I use for my work, and would definitely be unique, a static personal site being served from a Kubernetes Cluster. So I picked and proceeded with this idea.

MVP1

The setup to use a static website was pretty straight forward from this. Setup the domain. Point it to the static IP for the Kubernetes LB. Direct the traffic to the pod serving the site. It is pretty simple. You could find the setup here https://github.com/skyaara/oci-k8s-static-portfolio.

For the MVP1, the setup was directly serving the static Astro SSG site directly from the nginx pod. It was functional, It was unique. The issue was only I knew it was unique. A person viewing the site in their browser does not see the difference of it being hosted on Cloudflare pages, a Kubernetes Cluster or the Moon. So I was still not being satisfied of it being uniquely built.

Inspiration for the current idea

If you are on tech twitter, you would remember there was a guy who got laid off from Atlassian, and spoke about the internal infrastructure of Atlassian. He dropped a informative youtube video on his tenure there https://www.youtube.com/watch?v=55pTFVoclvE&t=1510s. A curious cat found its way. In the video he heavily spoke about how good and configurable of a L7 Proxy Envoy actually was. I had only worked with nginx based HTTP proxies till now, and Envoy has piqued my interest, and i went through the documentation and I loved it. Anything we do with it could directly fit into serving our site from our Kubernetes Cluster.

Current setup

The immediate basic setup I could come up with Envoy was using it as a router for different versions of the site. Initially I thought I could serve a full site through the Cluster, then the sites might diverge and people might not feel the similarity if they opened the portfolio on 2 different days. So I converged to a shared site, that cycles through the canvas shaders that changes the visuals on the site, keeping the core content same. This is the idea the current site is built on. The same setup can be done in multiple simpler ways for free, but the way I built it is stupid and dumb, so lets stick with it.

There are currently 4 different visual shaders for the site. Each of them is compiled into appN/current.js where N is 1,2,3 or 4. The client requests a shader with a static url from the site, and the request is routed through the Envoy filter, which cycles the shaders per day. This configuration can be extended to any number of apps and is jsut a small change to lua script to support new shaders.

Below is a simple Envoy Lua filter script, that only allows GET requests, only from a certain domain, calculate epochs in terms of days and cycles through apps. It modifies the app path and adds appN to the route such that it can pick up correct js file for the version that day. This enables us to serve different versions of javascript from the same url.

lua

function envoy_on_request(handle)
  local headers = handle:headers()
  local authority = string.gsub(headers:get(":authority") or "", ":%d+$", "")
  if authority ~= "shaders.aakashreddy.com" then return end

  if headers:get(":method") ~= "GET" then
    handle:respond({[":status"] = "405", ["allow"] = "GET"}, "Method Not Allowed\n")
    return
  end

  local path = string.match(headers:get(":path") or "/", "^[^?]*")
  if path ~= "/current.js" then
    handle:respond({[":status"] = "404"}, "Not Found\n")
    return
  end

  -- A Unix day counter continues cleanly across month and year boundaries.
  local utc_day = math.floor(os.time() / 86400)
  local app = (utc_day % 4) + 1

  -- The browser keeps requesting /current.js; this is an internal rewrite.
  headers:replace(":path", "/app" .. app .. "/current.js")
end

Kubernetes resource definitions

This could get very long for no reason, so lets intentionally just keep it short.

Our setup currently consist of 2 namespaces, istio-system and site and a native OCI loadbalancer that has a static IP, that we supply to the Cloudflare DNS.

istio-system has istiod installed and manages the istio resources and EnvoyIngress. Whenever a new deployment happens istiod reads the updated config if any config is changed it sends the updated config via xDS to the relevant resources. This is a pretty standard setup for custom Envoy setup that uses istio.

site namespace has the actual shader files in a lightweight nginx pod, that just acts as a file system in our setup.

Client to the Origin request flow

The important final block: Security

Considering this whole site is unauthenticated everyone should be able to read the site with no restriction always. There are 2 important moving parts in the request flow. The part where it reaches the LoadBalancer and from LoadBalancer to the nginx pod.

The second part of the request flow is the relatively easier one. We are able to inspect L7 in this part and identify only requests that we need to allow, and Kubernetes provides a lot of request filtering APIs that makes this the easiest part. So we will get this out of the way first.

After the request reaches LoadBalancer, the flow of the request is to IstioGateway, where the TLS termination happens and only accepts when the host is shaders.aakashreddy.com. A validated requests gets passed onto the EnvoyFilter that only allows GET methods and 405 Method Not Allowed for others, accepts only /current.js paths, 404 Not Found for any other path, remove query params before matching, rewrites /current.js to /appN/current.js. Only the verified requests and requests that are part of VirtualService and DestinationRule are allowed to pod via ClusterIP. The site namespace embodies restricted Pod security profile that restricts most of the privileged escalation processes. We run the nginx pod as a non-root linux process with a read only user permission, and allows only runtime nginx linux calls restricts any other linux system calls, remove privileged escalation and do not supply any Kubernetes serviceAccountToken to disable permissions to read and write access to any Kubernetes APIs with a final duplicated HTTP Method and path validation (This is an overkill as we only intended to use nginx as a filesystem).

The above part sounds complicated but is mostly solved via Industry wide accepted Kubernetes practices.

Now coming the part of the flow where we need to restrict most of the traffic from reaching LoadBalancer and serve via Cloudflare. We have DNS records set for shaders.aakashreddy.com and proxy via Cloudflare edge to the LoadBalancer static IP and use allowlist only public Cloudflare published IPs, and setup aakashreddy.com on the Cloudflare Pages, and allow caching for shaders.aakashreddy.com and we intend to serve most of our traffic via CDN. I added protective WAF rules on Cloudflare that blocks any requests other than GET shaders.aakashreddy.com/current.js , ignore all query params so that cache hits happen irrespective of query params, add one hour Cache-Control and ratelimiting. This like 99.99% of the time guarantees the security we need and rest of it when gets passed through are handled by the Kubernetes Security protocols.

Now that we have secured everything and allow only restricted inbound HTTP traffic, we should be done right? Nope, another question exists how do we deploy the application if we do not allow any public requests to the OKE Kubernetes API. We use a service called OCI Bastion which is an ssh-like service managed by OCI that is part of subnet and allows request forwarding to the OKE managed Kubernetes service to update Kubernetes resources. When we want to deploy, we open a Bastion session via authenticated OCI CLI, upload our public SSH key, connects to Bastion using matching private key on the machine from an allowed client CIDR. We use helm to connect to OKE Kubernetes API via Bastion ssh which applies the updated configuration.