Installing ingress-nginx on K3S

The most commonly used ingress operator for kubernetes clusters is ingress-nginx, but by default K3S clusters deploy Traefik ingress instead. While Traefik is perfectly fine as an ingress controller, it may be desirable to use nginx for better performance and more standardized use within the community, making help troubleshooting easier.

Ensuring Traefik is not deployed

When installing a K3S cluster, simply adding the --disable traefik option to the installation script of the master (server) nodes will ensure Traefik is not automatically deployed. This approach will also ensure that Traefik does not get deployed on node failure or server restarts. Note that all master nodes need to be installed with this flag to work properly!

Installing ingress-nginx

Once cluster installation completes, it is time to install the ingress controller:

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.8.2/deploy/static/provider/cloud/deploy.yaml

Wait for the installation to finish. You can watch the state of the deployed pods with this command:

kubectl get pods -n ingress-nginx --watch

Once all pods have a Status of either Running or Completed, your ingress is ready to serve http requests.

Exposing a sample deployment

To test the newly installed ingress is working properly, let's install a sample application and expose it as a ClusterIP service:

kubectl create deployment web --image=gcr.io/google-samples/hello-app:1.0
kubectl expose deployment web --type=ClusterIP --port=8080

Next, we can expose the internal ClusterIP service through the nginx ingress, for example through a domain name. This example uses sample-web.com, you will need to change this to a domain pointing at any of your K3S cluster nodes.

kubectl create ingress web --rule="sample-web.com/=web:8080" --class=nginx

If all is working well, you should now be able to access the domain in your browser and see the sample application's output, which will look similar to this:

Hello, world!
Version: 1.0.0
Hostname: web-548f6458b5-jk4f8

Customizing ingress-nginx config

While the default configuration of the ingress-nginx operator is working for most use cases, you may want to customize it to better fit your needs. There are a lot of well-documented settings that can be altered through a simple ConfigMap.

Let's look at how to use that by enabling GZIP and Brotli compression for our ingress. First, retrieve the current state of the configmap in yaml format:

kubectl get configmap ingress-nginx-controller -n ingress-nginx -o yaml > config.yml

This will create a new file config.yml if your current directory, containing the configuration of your ingress-nginx operator. It's contents should look similar to this:

apiVersion: v1
data:
 allow-snippet-annotations: "true"
kind: ConfigMap
metadata:
 # more fields here, but those are not important for our changes

To enable the compression algorithms mentioned above, we need to add the keys enable-brotli and enable-gzip under the data key in the yaml, so it looks like this:

apiVersion: v1
data:
 allow-snippet-annotations: "true"
 enable-brotli: "true"
 enable-gzip: "true"
kind: ConfigMap
metadata:
 # more fields here, but those are not important for our changes