Træfik outside Kubernetes

• 3 min read
Tags: Kubernetes  
⬇Download as Markdown ⬇Download as Gemtext

How to use a single Traefik for Docker, Kubernetes and regular webservers

I have a single-node Kubernetes cluster, but I also run docker workloads on it, and wanted to use traefik for both.

Sure, I could run two separated instances but it would be a waste.

Traefik is a versatile reverse proxy, and has many different backend for Configuration Discovery, such as Docker labels, KubernetesIngress, Nomad…

    graph TB
Traefik-->pid1(local process)
Traefik-->pid2(local process)
subgraph Kubernetes
pod1
pod2
end
Traefik-->pod1(Pod)
Traefik-->pod2(Pod)
subgraph Docker
container1
container2
end
Traefik-->container1(container)
Traefik-->container2(container)

example deployment with multiple targets

Targets

I want my Traefik to proxy traffic and handle TLS termination for the following types of services:

  • standalone Docker containers
  • Kubernetes pods
  • regular Linux processes

Because everything runs on a single machine, a lot of the network setup can be ignored. Traefik must be able to reach the IP adresses of the Docker containers and the Kubernetes pods.

Installation

I wanted to run Traefik with systemd, but since Traefik does not provide Debian packages, you need to install the binary by hand and setup a service file.

Setup Traefik as a systemd Service (.kg-bookmark-container)

This post uses TOML configuration, but the result is the same

Setup

Traefik will need to access the Docker socket and the Kubernetes API.

And everything else will be configured in Traefik dynamic files

Instead of using the credentials in you kubeconfig, you should create a ServiceAccount so that Traefik can access the Kubernets API without root access.

Fortunately the work of identifying the correct permissions has been done by the Traefik team in their Helm chart

Because Traefik runs outside of Kubernetes, its ServiceAccount token will expire. You can issue tokens with no expiration by manually creating an empty Secret.

Configure Service Accounts for Pods

I use the following configuration in kubernetes:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: traefik-account
  namespace: kube-public
---
apiVersion: v1
kind: Secret
metadata:
  annotations:
    kubernetes.io/service-account.name: traefik-account
  name: traefik-token
  namespace: kube-public
type: kubernetes.io/service-account-token
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: traefik-role
rules:
  - apiGroups:
      - ""
    resources:
      - services
      - secrets
      - nodes
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - discovery.k8s.io
    resources:
      - endpointslices
    verbs:
      - list
      - watch
  - apiGroups:
      - extensions
      - networking.k8s.io
    resources:
      - ingresses
      - ingressclasses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
      - networking.k8s.io
    resources:
      - ingresses/status
    verbs:
      - update
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: traefik-role-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: traefik-role
subjects:
  - kind: ServiceAccount
    name: traefik-account
    namespace: kube-public

Once this is created, you need to extract some informations: get the ServiceAccount token and CA using the following commands

kubectl --namespace kube-public get secret traefik-token \
 --output jsonpath='{.data.token}' \
 | base64 --decode # → the token
kubectl --namespace kube-public get secret traefik-token \
 --output jsonpath="{['data']['ca\.crt']}"\
 | base64 --decode > your-CA.pem # → the CA

And for the endpoint and CA, get it from your kubeconfig.

You can check you got the correct endpoint with curl --cacert your-CA.pem https://your-endpoint:6443

If the api-server certificate does not match the hostname/IP of the endpoint, then you can skip certificate verification with kubectl --insecure-skip-tls-verify=false ....

Beware, this exposes you to man-in-the-middle attacks !

And now for the Traefik main config:

providers:
  kubernetesIngress:
    endpoint: https://your-endpoint:6443 # your endpoint
    token: 'your-token'
    certAuthFilePath: 'path/to/your-CA.pem'
    ingressEndpoint:
      ip: your-traefik-ip # optional
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false
  file:
    directory: "traefik/conf.d"
    watch: true

api: {}
ping:
  entryPoint: "ping"
entryPoints:
  ping:
    address: ":8082"
    reusePort: true
  web:
    address: ":80"
    reusePort: true
    http:
      redirections:
        entryPoint:
          to: websecure
          scheme: https
  websecure:
    address: ":443"
    reusePort: true
    http3: {}
    http:
      tls: {}
defaultEntryPoints: 
  - websecure

File configuration

The File provider will dynamically reload Traefik state as the files change.

Read more