Rerouting Container Registries With Envoy
By ,
Introduction #
In this post, I will detail the discovery of Envoy’s dynamic rewriting location capabilities and the relationship to OCI registries.
What is Envoy?
open source edge and service proxy, designed for cloud-native applications
What is an OCI container registry?
a standard, and specification, for the implementation of container registries
I’ve been playing around with and learning Envoy for a number of months now. One of the concepts I’m investigating is rewriting the request’s host. Envoy is a super powerful piece of software. It is flexible and highly dynamic.
Journey #
My expectations #
The goal is to set up Envoy on a host to rewrite all requests dynamically back to a container registry hosted by a cloud-provider, such as GCP.
Initial discoveries #
One of the first things I investigated was the ability to get traffic from one site and serve it on another (proxying). I searched in the docs and in their most basic example could see that, by using envoy’s http filter in the filter_chains, a static host could be rewritten.
Example:
...
static_resources:
listeners:
- name: listener_0
address:
socket_address:
address: 0.0.0.0
port_value: 10000
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
stat_prefix: ingress_http
access_log:
- name: envoy.access_loggers.file
typed_config:
"@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
path: /dev/stdout
http_filters:
- name: envoy.filters.http.router
route_config:
name: local_route
virtual_hosts:
- name: local_service
domains: ["*"]
routes:
- match:
prefix: "/"
route:
host_rewrite_literal: www.envoyproxy.io
cluster: service_envoyproxy_io
...
This is a great start! This serves the site and its content under the host where Envoy is served. However, the host in the rewrite is static and not dynamic. It seems at this point like doing the implementation this way is not viable.
Learning about filter-chains #
Envoy has the lovely feature to set many kinds of middleware in the middle of a request. This middleware can be used to add/change/remove things from the request. Envoy is particularly good at HTTP related filtering. It also supports such features as dynamic forward proxy, JWT auth, health checks, and rate limiting.
The functionality is infinitely useful as filters can be such things as gRPC, PostgreSQL, Wasm, and even Lua.
The implementation #
Once I found the ability to write Lua as a filter, I found that it provided enough capability to perform the dynamic host rewrite.
static_resources:
listeners:
- name: main
address:
socket_address:
address: 0.0.0.0
port_value: 10000
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
stat_prefix: ingress_http
codec_type: auto
route_config:
name: local_route
virtual_hosts:
- name: local_service
domains:
- "*"
routes:
- match:
prefix: "/"
route:
cluster: web_service
http_filters:
- name: envoy.filters.http.lua
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua
inline_code: |
local reg1 = "k8s.gcr.io"
local reg2 = "registry-1.docker.io"
local reg2WithIP = "192.168.0.1"
function envoy_on_request(request_handle)
local reg = reg1
remoteAddr = request_handle:headers():get("x-real-ip")
if remoteAddr == reg2WithIP then
request_handle:logInfo("remoteAddr: "..reg2WithIP)
reg = reg2
end
if request_handle:headers():get(":method") == "GET" then
request_handle:respond(
{
[":status"] = "302",
["location"] = "https://"..reg..request_handle:headers():get(":path"),
["Content-Type"] = "text/html; charset=utf-8",
[":authority"] = "web_service"
},
'<a href="'.."https://"..reg..request_handle:headers():get(":path")..'">'.."302".."</a>.\n")
end
end
- name: envoy.filters.http.router
typed_config: {}
clusters:
- name: web_service
connect_timeout: 0.25s
type: LOGICAL_DNS
lb_policy: round_robin
load_assignment:
cluster_name: web_service
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: ii.coop
port_value: 443
With envoy running this config, the behaviour of the requests is
- rewrite all traffic hitting the web service to k8s.gcr.io
- except if the IP is 192.168.0.1 then set the location to registry-1.docker.io
Since I’m using a Pair instance, it sets the local subnet to 192.168.0.0/24 so when I try to docker pull humacs-envoy-10000.$SHARINGIO_PAIR_BASE_DNS_NAME/library/postgres:12-alpine it will go to docker.io.
On my local machine, pulling container images using docker pull humacs-envoy-10000.$SHARINGIO_PAIR_BASE_DNS_NAME/e2e-test-images/agnhost:2.26 will instead use k8s.gcr.io.
To achieve this, I research how other http libraries handle redirects - namely Golang’s net/http.Redirect. The main things that Golang’s http.Redirect does is:
- set the content-type header to text/html
- set the location to the destination
- set the status code to 302
- set the body to the same data in earlier steps, but in an a tag.
Final thoughts #
I’m learning that Envoy is highly flexible and seemly limitless in it’s capabilities.
It’s exciting to see Envoy being adopted in so many places - moreover to see the diverse usecases and implementations.
Big shout out to Zach for pairing on this with a few different aspects and attempts! (Zach is cool™️)