The edge was running. I could have just left it alone.
On the Proxmox LXC, my edge hosting had been running stably for months: Debian as a base, managed with Ansible, in front of it OpenResty with native HTTP/3 and separate PHP 8.4 FPM pools behind it per domain. The vhosts were cleanly generated from group_vars/sites_php.yml and group_vars/sites_proxy.yml.
A working setup.
During routine cleanup, I stumbled upon a few things.
In sites-available there were still three old Nginx vhosts. Plus two PHP pools and certificate remnants from services I had shut down over a year ago. Ansible over the years had done exactly what the playbooks said. But what I had eventually deleted from the variables simply continued to exist on the host.
That's when I wondered whether Ansible was still the right tool for this particular server. Orchestration doesn't automatically delete what you no longer mention.
The Difference Between Convergence and Declaration
Ansible describes desired states and works largely idempotently. If a variable is defined, the playbooks ensure that file, user, and service exist.
The problem lies on the other side: when I remove a domain from my variables file, the target system knows nothing about it. Ansible simply stops running the tasks for that domain. The vhost remains in /etc/nginx/sites-enabled/, the system user stays in the operating system, and the PHP pool keeps running until you clean it up manually or explicitly build state: absent for it.
On a classic, persistent host, an astonishing amount of drift can accumulate over the years.
Classic Linux: System configuration = Base OS + Updates + History of all changes
NixOS: System configuration = Reproducible result of the current Flake configuration
That's why I made the cut and migrated the edge to NixOS 26.05 in August.
The Architecture on the Edge
Nothing changed in the network flow. Externally, the OPNsense firewall stands in front, forwarding TCP 80/443 and UDP 443 (for HTTP/3 QUIC) to the internal edge container.
WAN ──► OPNsense (NAT TCP/UDP 443) ──► NixOS Edge LXC
├── OpenResty (HTTP/3 + Lua)
├── PHP-FPM Pools (nerdbear, bearwave, ...)
└── Upstream Proxies (Forgejo, Vaultwarden)
The difference lies in how the host is defined. The entire operating system configuration lives in a Git repository:
hosts/edge.nix: Host definition, SSH keys, firewall rules, and global services.sites/php.nix: PHP sites as structured attributes.sites/proxy.nix: Reverse proxy targets including buffer and timeout values.sites/shared.nix: Redirects and aliases.
I deliberately didn't want more abstraction at this point. A PHP site like nerdbear.de or bearwave.app in Nix is just a compact data record:
"nerdbear.de" = {
type = "php";
user = "nerdbear";
phpVersion = "8.4";
deployRootSubdir = "index";
docrootSubdir = "index/public";
tls = {
enable = true;
mode = "external";
certName = "nerdbear.de";
};
};
From this, NixOS generates the system user nerdbear, the systemd unit for the PHP 8.4 pool with Unix socket at /run/phpfpm/nerdbear.de.sock, and the OpenResty configuration.
When I delete this block from the file and run nixos-rebuild switch --flake .#edge, the vhost, systemd unit, socket, and the declaratively generated system components are removed with the new generation. What is no longer configured is no longer dragged along.
User Isolation and File Permissions
I have no patience for global web server users. If ten different websites run on a single server, each one belongs in its own prison.
| Domain | Linux User | PHP-FPM Socket |
|---|---|---|
nerdbear.de |
nerdbear |
/run/phpfpm/nerdbear.de.sock |
palencsar.de |
nerdbear (Shared Root) |
/run/phpfpm/nerdbear.de.sock |
The separation between declarative system configuration and persistent data is strict:
- Site content: Lives under
/srv/www/<domain>/index/public/. Deployment is Git-based (push-to-deploy via bare repository and hook) as userragnar. - Group permissions: Directories under
/srv/www/belong to the domain user and thewebdeploygroup with SetGID (chmod 2775). Newly checked-out files automatically inherit the correct group permission. - Certificates: Are not generated on the edge, but centrally obtained via my PowerDNS host using ACME and stored under
/etc/ssl/hosting/<cert>/. The web server reads them via thetls-cert-readersgroup.
Where NixOS Annoys in Daily Use
NixOS is not a system for spontaneous experiments directly on the server. Coming from Debian, you stumble over quirks that at first simply require adjustment.
No standard FHS: There is no classic /usr/bin/php or /etc/nginx. The binaries and configurations sit immutably and hashed in /nix/store/....
In practice, this means:
-
No quick hotfixes in
/etc: Quickly editing something in/etcand forgetting the fix there no longer works. The permanent change belongs back in the Nix configuration, followed by the rebuild. -
Socket permissions: OpenResty runs as
webdeploy, PHP-FPM as the respective domain user. If the socket permissions in the PHP module aren't meticulously set tolisten.mode = 0660andlisten.group = webdeploy, Nginx responds to the first request with a502 Bad Gateway. -
LXC on Proxmox: In unprivileged containers, you need to cleanly align the UID/GID mapping and the required systemd permissions with the Proxmox host so that sockets and file access work without friction.
Why the Switch Was Worth It Anyway
The advantage shows itself during updates and maintenance.
When I roll out a new PHP version or OpenResty configuration today and something hiccups, nixos-rebuild switch --rollback immediately activates the previous system generation. Packages, services, and configurations jump back to the last working state in seconds. The persistent data under /srv/www remains untouched.
Should the container be destroyed, I don't need a root filesystem backup. I create an empty NixOS LXC, mount /srv/www and the certificates, let the Flake run through — and the edge is back up in a few minutes, exactly as before.
The switch wasn't a hype project. I wanted a system that can be controlled reproducibly, without having to guess after a year what's actually still active on the host.