Skip to content

Containers

Source directory: modules/containers/

default.nix

modules/containers/default.nix

No option declarations; see source for implementation.

docker.nix

modules/containers/docker.nix

  • Enable option: Docker container support

Options: enable, users, rootless, dataRoot

Options declaration (Nix) 
  options.modules.containers.docker = {
    enable = mkEnableOption "Docker container support";

    users = mkOption {
      type = types.listOf types.str;
      default = hostUsers; # Use host users as default
      description = "List of users to add to the docker group";
      example = [ "olafkfreund" "workuser" ];
    };

    rootless = mkOption {
      type = types.bool;
      default = false;
      description = "Enable rootless Docker";
    };

    dataRoot = mkOption {
      type = types.nullOr types.path;
      default = null;
      example = "/home/docker";
      description = ''
        Directory holding image layers, container writable layers and
        volumes. `null` keeps Docker's own default (/var/lib/docker).

        Pick the filesystem deliberately: this path absorbs every container
        write on the host, so it wants free space *and* low write latency.
        An SMR drive is the wrong answer even when it is the emptiest one —
        shingled rewrites collapse sustained random writes to single-digit
        MB/s, which starves image pulls and database fsyncs alike.
      '';
    };
  }

k3d.nix

modules/containers/k3d.nix

k3d (k3s in Docker) cluster bootstrap.

Runs a single-node k3s cluster inside the host Docker daemon. The cluster is created on first boot by a one-shot systemd unit and persisted via Docker volumes (under the host's Docker data-root). Local-path PV storage is bind-mounted to ${cfg.storageDir} so PVCs survive cluster recreation AND don't pressure /mnt/media.

After the cluster exists this unit also applies a GitOps bootstrap (kubectl apply -k /), which is expected to install ArgoCD plus the App-of-Apps root Application. ArgoCD then self-manages everything else.

Tailnet exposure model — SIDECAR pattern:

Each Pod that should be reachable on the tailnet runs an in-pod tailscale sidecar container alongside its main container(s). The sidecar registers a new tailnet node (e.g. argocd.tail833f7.ts.net) using a reusable auth key materialised by the bootstrap unit at tailscale/auth-key (key TS_AUTHKEY) inside the cluster.

We chose sidecars over the Tailscale Kubernetes Operator because the operator requires OAuth client credentials (admin → OAuth clients) and this homelab is wired with a plain auth key (admin → Keys). Sidecar trade-offs vs operator: + works with an auth key + no CRDs, no extra control-plane component - more boilerplate per service (sidecar container + 2 env refs) - auth-key rotation is manual (Tailscale max key TTL is 90 days, reusable keys can be longer if configured). When the key in agenix is replaced (manage-secrets.sh edit), restart the bootstrap unit to refresh the in-cluster Secret, then bounce any sidecar-running Pods so they re-register.

Why this won't clash with the host-level tailscale serve setup on p510: the host's tailscaled and tailscale serve config bind port 443 on the p510 tailnet node only. Sidecars register entirely separate tailnet nodes from inside the cluster — completely independent.

This module is opt-in per host — import ../../modules/containers/k3d.nix in the host's configuration.nix imports list, then flip modules.containers.k3d.enable = true.

  • Enable option: k3d (k3s in Docker) cluster bootstrap
  • Enable option: Apply the GitOps bootstrap (kubectl apply -k /) after cluster create
  • Enable option: Materialise the Tailscale auth-key Secret in the cluster (for sidecar consumption)
  • Enable option: Apply all factory-namespace agenix-encrypted Secret manifests during cluster bootstrap
  • Enable option: Periodically drop unreferenced container images and orphaned Docker volumes

Options: enable, k3sImage, clusterName, resolvers, apiPort, apiWaitSeconds, apiHostBind, storageDir, kubeconfigPath, users, gitopsRepo, gitopsRef, bootstrapPath, authKeyFile, targetNamespaces, dates

Options declaration (Nix) 
  options.modules.containers.k3d = {
    enable = mkEnableOption "k3d (k3s in Docker) cluster bootstrap";

    package = mkPackageOption pkgs "k3d" { };

    k3sImage = mkOption {
      type = types.str;
      default = "rancher/k3s:v1.31.5-k3s1";
      description = ''
        k3s node image k3d boots the cluster from (passed to
        `k3d cluster create --image`).

        Pin this explicitly rather than relying on the k3d binary's
        built-in default: that default was an ancient `v1.21.7-k3s1`
        whose bundled containerd/runc mishandles shared-memory page
        faults on modern host kernels, making every PostgreSQL pod die
        during `initdb` with `Bus error (core dumped)`. A current k3s
        ships a runc that handles this correctly.

        NOTE: this only takes effect on cluster *creation*. To adopt a
        new image on an existing cluster, delete and recreate it:
        `k3d cluster delete ${"$"}{clusterName}` then re-run the
        bootstrap unit (`systemctl start k3d-cluster-bootstrap`).
      '';
    };

    clusterName = mkOption {
      type = types.str;
      default = "factory";
      description = "Name of the k3d cluster (used as the docker container prefix and config selector).";
    };

    resolvers = mkOption {
      type = types.listOf types.str;
      default = [ "1.1.1.1" "9.9.9.9" ];
      description = ''
        Upstream DNS servers CoreDNS forwards to, written into a resolv.conf
        that is mounted into every node and passed to k3s as `--resolv-conf`.

        These MUST be reachable from a *pod* network namespace. Do not use a
        Docker bridge-gateway address (e.g. 172.18.0.1) or a loopback address:
        both resolve only inside the node container, and CoreDNS does not run
        there. See the `nodeResolvConf` comment above and #1232.

        NOTE: like `k3sImage`, this only applies on cluster *creation* — the
        mount is fixed at container-create time. To adopt a change on an
        existing cluster, recreate it or edit the nodes' resolv.conf by hand.
      '';
    };

    apiPort = mkOption {
      type = types.port;
      default = 6443;
      description = ''
        Host port for the kube API server. Use `kubectl --kubeconfig
        ${"$"}{kubeconfigPath}` from the host, or set `apiHostBind` to
        a non-loopback address to reach it from elsewhere.
      '';
    };

    apiWaitSeconds = mkOption {
      type = types.ints.positive;
      default = 1800;
      description = ''
        Roughly how long the bootstrap waits for the kube API before giving
        up on a self-heal attempt, in seconds. Approximate: the poll costs a
        `docker exec` plus a readiness probe on top of its 2s sleep, so the
        real ceiling runs somewhat longer than the number set here.

        This is a CEILING, not a fixed delay: the wait ends as soon as the API
        answers, and also ends early if the k3s server stops making progress
        (no datastore reads for ~2min). Generous by design — k3s must open the
        whole datastore before it serves, logging nothing meanwhile, and
        restarting it mid-scan discards the work. An 11GB state.db on a
        spinning disk took ~17min.
      '';
    };

    apiHostBind = mkOption {
      type = types.str;
      default = "127.0.0.1";
      example = "0.0.0.0";
      description = ''
        Host IP the kube API server binds to.

        Defaults to `127.0.0.1` — loopback only, requires SSH tunnel /
        port-forward for off-host kubectl. Set to `0.0.0.0` (or a
        specific interface IP) to expose the API to the LAN/tailnet so
        kubectl can connect directly. With a default-open Tailscale
        ACL and a disabled host firewall, `0.0.0.0` and `127.0.0.1`
# … truncated — see source link above