Containers¶
Source directory: modules/containers/
default.nix¶
modules/containers/default.nix
No option declarations; see source for implementation.
docker.nix¶
- Enable option: Docker container support
Options: enable, users, rootless
Options declaration (Nix)
options.modules.containers.docker = {
enable = mkEnableOption "Docker container support";
users = mkOption {
type = types.listOf types.str;
default = hostUsers; # Use host users as default
description = "List of users to add to the docker group";
example = [ "olafkfreund" "workuser" ];
};
rootless = mkOption {
type = types.bool;
default = false;
description = "Enable rootless Docker";
};
}
k3d.nix¶
k3d (k3s in Docker) cluster bootstrap.
Runs a single-node k3s cluster inside the host Docker daemon. The cluster is created on first boot by a one-shot systemd unit and persisted via Docker volumes (under the host's Docker data-root). Local-path PV storage is bind-mounted to ${cfg.storageDir} so PVCs survive cluster recreation AND don't pressure /mnt/media.
After the cluster exists this unit also applies a GitOps bootstrap
(kubectl apply -k
Tailnet exposure model — SIDECAR pattern:
Each Pod that should be reachable on the tailnet runs an in-pod
tailscale sidecar container alongside its main container(s). The
sidecar registers a new tailnet node (e.g. argocd.tail833f7.ts.net)
using a reusable auth key materialised by the bootstrap unit at
tailscale/auth-key (key TS_AUTHKEY) inside the cluster.
We chose sidecars over the Tailscale Kubernetes Operator because the operator requires OAuth client credentials (admin → OAuth clients) and this homelab is wired with a plain auth key (admin → Keys). Sidecar trade-offs vs operator: + works with an auth key + no CRDs, no extra control-plane component - more boilerplate per service (sidecar container + 2 env refs) - auth-key rotation is manual (Tailscale max key TTL is 90 days, reusable keys can be longer if configured). When the key in agenix is replaced (manage-secrets.sh edit), restart the bootstrap unit to refresh the in-cluster Secret, then bounce any sidecar-running Pods so they re-register.
Why this won't clash with the host-level tailscale serve setup on
p510: the host's tailscaled and tailscale serve config bind port
443 on the p510 tailnet node only. Sidecars register entirely separate
tailnet nodes from inside the cluster — completely independent.
This module is opt-in per host — import ../../modules/containers/k3d.nix in the host's configuration.nix imports list, then flip modules.containers.k3d.enable = true.
- Enable option: k3d (k3s in Docker) cluster bootstrap
- Enable option: Apply the GitOps bootstrap (kubectl apply -k
/ ) after cluster create - Enable option: Materialise the Tailscale auth-key Secret in the cluster (for sidecar consumption)
- Enable option: Apply all factory-namespace agenix-encrypted Secret manifests during cluster bootstrap
Options: enable, k3sImage, clusterName, apiPort, apiHostBind, storageDir, kubeconfigPath, users, gitopsRepo, gitopsRef, bootstrapPath, authKeyFile, targetNamespaces
Options declaration (Nix)
options.modules.containers.k3d = {
enable = mkEnableOption "k3d (k3s in Docker) cluster bootstrap";
package = mkPackageOption pkgs "k3d" { };
k3sImage = mkOption {
type = types.str;
default = "rancher/k3s:v1.31.5-k3s1";
description = ''
k3s node image k3d boots the cluster from (passed to
`k3d cluster create --image`).
Pin this explicitly rather than relying on the k3d binary's
built-in default: that default was an ancient `v1.21.7-k3s1`
whose bundled containerd/runc mishandles shared-memory page
faults on modern host kernels, making every PostgreSQL pod die
during `initdb` with `Bus error (core dumped)`. A current k3s
ships a runc that handles this correctly.
NOTE: this only takes effect on cluster *creation*. To adopt a
new image on an existing cluster, delete and recreate it:
`k3d cluster delete ${"$"}{clusterName}` then re-run the
bootstrap unit (`systemctl start k3d-cluster-bootstrap`).
'';
};
clusterName = mkOption {
type = types.str;
default = "factory";
description = "Name of the k3d cluster (used as the docker container prefix and config selector).";
};
apiPort = mkOption {
type = types.port;
default = 6443;
description = ''
Host port for the kube API server. Use `kubectl --kubeconfig
${"$"}{kubeconfigPath}` from the host, or set `apiHostBind` to
a non-loopback address to reach it from elsewhere.
'';
};
apiHostBind = mkOption {
type = types.str;
default = "127.0.0.1";
example = "0.0.0.0";
description = ''
Host IP the kube API server binds to.
Defaults to `127.0.0.1` — loopback only, requires SSH tunnel /
port-forward for off-host kubectl. Set to `0.0.0.0` (or a
specific interface IP) to expose the API to the LAN/tailnet so
kubectl can connect directly. With a default-open Tailscale
ACL and a disabled host firewall, `0.0.0.0` and `127.0.0.1`
are equivalent reachability-wise — the API is gated by the
bearer token in the kubeconfig regardless.
'';
};
storageDir = mkOption {
type = types.path;
default = "/mnt/img_pool/k3d/storage";
description = ''
Host directory bind-mounted into the k3d server container at
/var/lib/rancher/k3s/storage. Used as the backing store for the
local-path-provisioner (the default StorageClass shipped with k3s).
Keep this off /mnt/media so cluster PVCs don't compete with the
media library for IOPS.
'';
};
kubeconfigPath = mkOption {
type = types.path;
default = "/etc/k3d/kubeconfig";
description = ''
Where to write the host-side kubeconfig. Mode 0640, group wheel —
any user in the wheel group can read it. Set KUBECONFIG to this
path in your shell to drive the cluster.
'';
};
users = mkOption {
type = types.listOf types.str;
default = hostUsers;
example = [ "olafkfreund" ];
description = ''
Users whose login shell will export KUBECONFIG=${"$"}{kubeconfigPath}.
These users must already be in the wheel group to actually read
the file — this option just sets the env var.
'';
# … truncated — see source link above