An AWS ECR Reverse Proxy with NGINX and Lua

By: Matt Jachowski

An AWS ECR Reverse Proxy with NGINX and Lua

Conducto is a developer tool that relies heavily upon Docker. We effectively give each of our clients push and pull access to our own Docker registry. When we were building our infrastructure, we weren’t sure how to best provide this functionality.

This was going to be a central part of our infrastructure, and our users would potentially generate many images. Right away, I knew that I did not want to take on the burden of deploying and maintaining a highly available, autoscaling Docker registry myself. So, I looked at the existing registry providers to see if I could come up with a solution that allowed us to integrate our own authentication. Nothing fit the bill.

Finding a Solution

Then, I came across this awesome blog post from Kloudless. The gist of their solution is to create a reverse proxy with NGINX in front of the Elastic Container Registry (ECR) from AWS. Then, using OpenResty, inject Lua directly into the NGINX config to call out to a custom auth service to gate client requests. This is great for a few reasons. First, by using ECR as a hosted registry, we get scalability, availability, and stability for free. Second, an NGINX reverse proxy is lightweight and performant, so it doesn’t degrade the baseline performance we get with ECR. And finally, Lua is a powerful scripting language that allows us to implement whatever auth logic we need. So, we implemented essentially this solution, with a few differences and special challenges.

Containerizing the Reverse Proxy

Our first difference was that we wanted to deploy our reverse proxy as a containerized microservice in AWS Elastic Container Service. This was straightforward to deploy just like our other microservices, and just required that we write a Dockerfile that included OpenResty, NGINX, the AWS CLI, and our code.

FROM openresty/openresty:buster
RUN apt-get update && apt-get install -y awscli cron
WORKDIR /conducto
COPY ./nginx.conf ./
COPY ./start.sh ./
COPY ./conducto.lua /usr/local/lib/lua/
RUN chmod o+r /usr/local/lib/lua/conducto.lua
EXPOSE 80
CMD ["./start.sh"]

The start.sh script sets up a cron job to refresh the AWS credentials every 8 hours and starts nginx.

#!/bin/sh

# Set cron job to refresh AWS creds every 8 hours. Start cron.
echo "11 */8 * * * root export AWS_REGION=$AWS_REGION && /conducto/get_token.sh >> /conducto/cron_log 2>&1" >> /etc/crontab
cron

# Run once so that token gets populated right away.
/conducto/get_token.sh > /conducto/cron_log

# Populate env vars in nginx config.
envsubst "\$CONDUCTO_URL \$CONDUCTO_REGISTRY \$ECR_REGISTRY \$PROXY_SCHEME \
  < /conducto/nginx.conf > /etc/nginx/conf.d/default.conf
  cat /etc/nginx/conf.d/default.conf

# Start nginx.
nginx -g "daemon off;"

Supporting Docker Push

Our second difference from Kloudless, which only needed to support pulling Docker images, was that we needed to support pushing too. The Kloudless solution almost worked, successfully doing most of a docker push, but always failing at the end. After debugging my docker traffic with the excellent mitmproxy tool, I noticed that my reverse proxy was sending back requests with the Location header incorrectly set to the ECR URL. Since this was my first time trying to customize an NGINX config, it took me embarrassingly long to discover the fix: set the proxy_redirect directive. These are the proxy settings from nginx config, with the all important proxy_redirect on the last line.

    # $ecr_host looks something like {account}.dkr.ecr.{region}.amazonaws.com
    proxy_pass                          https://$ecr_host$proxy_uri;
    proxy_set_header  Host              $ecr_host;
    proxy_set_header  X-Real-IP         $remote_addr;
    proxy_set_header  X-Forwarded-For   $proxy_add_x_forwarded_for;
    proxy_set_header  X-Forwarded-User  $http_authorization;
    proxy_set_header  X-Forwarded-Proto "https";
    proxy_pass_header                   Server;
    proxy_read_timeout                  900;
    proxy_redirect                      https://$ecr_host/ https://$host/;

Rewriting the Repo and Tag

We also needed to do something a little funky -- when a user pushes an image like docker.conducto.com/abc-def:tag, we actually want to store it in ECR as an image like {account}.dkr.ecr.{region}.amazonaws.com/shared-repo:tag_abc-def. That is, we wanted our reverse proxy to rewrite the repo and tag. The reasons for this are too technical for this article, but basically it helped us avoid repo proliferation and gave us some caching benefits.

Luckily, this was really easy with OpenResty and Lua. We added this section to our nginx config.

    rewrite_by_lua_block {
      local conducto = require("conducto")
      ngx.var.proxy_uri = conducto.rewrite_uri()
    }

And defined the rewrite_uri() function in conducto.lua. This is a simplified version.

function _M.rewrite_uri()
  local proxy_uri = ngx.var.request_uri

  -- URI looks like /version/repo/stuff, extract just the repo.
  local repo = string.match(ngx.var.request_uri, "/.-/(.-)/")

  -- URI with tag looks like /version/repo/manifests/tag, extract just the tag.
  local tag = string.match(ngx.var.request_uri, "/manifests/(.-)$")

  -- Replace old_repo with new_repo.
  local old_repo = string.format("/%s/", repo)
  local new_repo = "/shared-repo/"
  proxy_uri = ngx.re.sub(proxy_uri, old_repo, new_repo)

  -- Rewrite tag to look like "{tag}_{pipeline_id}"
  if tag ~= nil then
    local old_tag = string.format("/%s", tag)
    local new_tag = string.format("/%s_%s", tag, repo)
    proxy_uri = ngx.re.sub(proxy_uri, old_tag, new_tag)
  end

  return proxy_uri
end

Unit Testing the Reverse Proxy

Finally, we wanted to be able to unit test the reverse proxy, in isolation from ECR. So, I modified the start.sh script in the container to spin up a mock ECR registry if launched with TEST_MOCK set in the environment.

if [ -n "$TEST_MOCK" ] ; then
  echo "# Start and test mock server"
  export ECR_REGISTRY=0.0.0.0:8000
  python3 test/mock_http_server.py &> mock.out &
  apt-get install -y curl > /dev/null
  curl -i -X GET http://0.0.0.0:8000/health
  cat mock.out
fi

The ECR registry is mocked by mock_http_server.py, which is super simple and hardcodes responses to a few specific GET, PUT, POST, and PATCH requests. This was just enough to let me write a full range of unit tests for the reverse proxy.

Clueless to Competent

Prior to implementing this solution, I had never written an NGINX config or used Lua in any significant way, and I had no deep understanding of what a reverse proxy even was. This just one example of a constant theme of my experience as a co-founder at Conducto -- going from 0 to 1, clueless to competent, over and over again. I wouldn’t be able to do my job without the helpful blog posts and documentation that thoughtful developers have shared all over the web. I hope that this small contribution to that ecosystem helps somebody else.