What We Found Useful About NixOS

Curling IO used to run on Debian configured with Ansible. That setup worked. It installed packages, configured Caddy and the firewall, created systemd services, and prepared our blue-green deployment slots.

The problem was not that Ansible couldn't describe the server. The problem was that a working, mutable server let us get away with an incomplete description. Old files remained under /etc. Build tools had been installed through a different path. Provider choices about disks and RAID lived outside the playbook. A provisioning run could succeed because an earlier run had already left the right thing behind.

NixOS is much less tolerant of these gaps. This strictness is occasionally annoying, but it is also the main benefit. It forces us to decide who owns a file, a service, a secret, a package, or a recovery input.

Moving from Ansible to NixOS taught us how much validation and cleanup the old approach had been missing. It exposed the difference between a server that continues to work and one we can recreate from declared inputs.

This is what we found useful while rebuilding an OVH bare-metal server from empty disks, restoring its state, and making the process safe enough to repeat.

The Boundary We Wanted

We wanted to answer a simple question: if the server disappeared, could we rebuild Curling IO in a timely manner, without depending on undocumented state from the old host, and with basically no customer data loss?

A successful recovery had to:

• Recreate the mirrored disk and EFI boot layout.
• Activate the exact host configuration.
• Restore the host's ability to decrypt its secrets.
• Restore valid Caddy certificate state.
• Build and deploy the platform, public site, and Operations application from one frozen Git revision.
• Restore and validate SQLite from Litestream.
• Verify public and private upload storage.
• Create and verify an encrypted database snapshot in a second region.
• Pass local, public, storage, monitoring, and provider checks.

This is broader than operating-system provisioning. NixOS owns the host, but application releases and customer state have their own workflows.

Concern	Owner
Disk layout and initial installation	OVH BYOLinux and our recovery-image boot hook
Operating-system packages and services	NixOS configuration
Generated host configuration	Versioned Nix store paths
Host secret activation	SOPS recovery workflow
Platform, public-site, and Operations builds	Independent deployment workflows
Release installation and activation	Independent server-side deployment controls
Database migrations	Platform deployment workflow
Host and application monitoring	Independently deployed Operations application
Customer database and uploads	Litestream and Object Storage workflows

That separation has held up well. We can change the host without deploying an application. We can deploy the platform without deploying the public site. We can restore a database without reinstalling the server.

Application Releases Stay Separate

NixOS owns the host, not the application build. We build the platform, public site, and Operations application before deployment, package them as verified releases, and upload them over SSH. Production does not need source code, repository credentials, compiler toolchains, or build caches.

That leaves a smaller production software footprint: fewer packages and executable tools, no source checkout or build credentials, and less mutable build state. It reduces the attack surface and patching burden, though it does not replace service sandboxing or timely updates.

This keeps host changes and application releases independent. It also gives recovery a simple boundary: build from one frozen Git revision, verify each package before and after upload, then activate it using the same deployment workflow we use during normal operation.

Nix Has Two Different Host Deployment Paths

Routine provisioning and replacement-host recovery deliberately work differently.

For a normal configuration change, our bin/provision production script sends the exact committed infrastructure source to the running server. The server uses Nix to build a new immutable system generation, validates it, switches to it, and keeps the previous generation available for rollback.

Clean-host recovery cannot rely on that path. A supposedly complete recovery image should not boot and then download or build most of the real system. Our BYOLinux image therefore contains:

• A minimal, secret-free bootstrap system that can boot and accept SSH.
• The complete production NixOS closure as inactive store content.

After first boot, the recovery command activates the already embedded production system and its required secrets. If the image does not contain the required closure, recovery stops. It does not hide the incomplete image by falling back to a remote build.

This makes the recovery image a real artifact rather than a thin installer. Application-only commits can still reuse it because the image derivation depends only on infrastructure source. Application code, documentation, and task files do not force another multi-gigabyte image build.

Bare Metal Keeps the Provider in the Design

NixOS does not remove OVH from the recovery path.

Our first installer created valid software RAID metadata with a newer Linux kernel. The installed NixOS kernel and OVH's rescue kernel both refused to assemble it. Upgrading only NixOS would have made the installed system work, but it would still have left the provider's rescue environment unable to read the disks.

We changed the ownership boundary. OVH BYOLinux now creates the mirrored ext4 filesystems using the provider's supported layout. Our image supplies the NixOS filesystem and a boot hook that labels the filesystems and installs removable-path EFI GRUB on both drives.

The provider's BYOLinux hook runs inside a restricted chroot. During our first attempts, the chroot did not have the shell or PATH we expected. A working image build told us nothing about whether OVH could execute the hook. We had to:

• Use an absolute Nix store path for Bash.
• Inline the hook instead of calling a second store path that was absent from the image.
• Use shell builtins until the embedded system PATH was available.
• Test every referenced store path against the image itself.
• Replay the hook inside a provider-like chroot.

This was disappointing. QCOW2 gives us a standard image format, but not a standard contract for turning that image into a bootable bare-metal installation. We expected the provider boundary to be more predictable. Instead, OVH ran a provider-specific hook inside a sparse chroot whose shell, PATH, and available store paths had to be discovered through testing. The image format was portable; the installation contract was not.

Provider monitoring also mattered. During an early rehearsal, OVH detected the planned outage and opened a technician intervention. That intervention blocked an API request to change the server's boot mode. Recovery preflight now disables and verifies automatic intervention before allowing a reinstall, then restores the selected policy after the recovered host passes its checks.

Getting this provider boundary right consumed the lion's share of the engineering time we spent on recovery provisioning. The successful reinstall itself took under eight minutes, but reaching that point required repeated image builds, failed hooks, rescue boots, and destructive rehearsals.

Generated Configuration Needs One Owner

Caddy exposed the problem with mutable configuration quickly.

The old server had generated fragments copied into mutable directories. When we changed the layout, stale files remained and were still loaded. The new configuration was correct in the repository but wrong on the server because the server contained both old and new interpretations.

The NixOS configuration now owns generated Caddy files as links to exact Nix store paths. Obsolete paths are removed explicitly, and the assembled configuration is validated before activation.

This lesson is not specific to NixOS. An Ansible deployment can enforce the same rule, but it has to do so deliberately. Copying the latest desired file is not enough when an old file can still affect runtime behaviour.

Recovery State Must Exist Before the Wipe

The failed server is not a recovery source.

Before erasing it, preflight verifies:

• The selected infrastructure revision and recovery image.
• The required secret-recovery inputs.
• Current RAID health.
• A decryptable Caddy-state backup with valid certificates.
• The offsite Litestream replica and encrypted snapshot in a second offsite location.
• Public and private upload storage boundaries.
• Public endpoints and OVH state.

Caddy state became an explicit input after we realized that certificate issuance should not sit on the critical recovery path. NixOS checks for changed Caddy state every five minutes and stores encrypted copies offsite. Weekly retention slots provide a backstop.

During recovery, Caddy stays stopped while the platform, site, database, uploads, and encrypted snapshot are prepared. It starts only when the server can present a complete system using the restored certificate state. Normal background renewal resumes afterward.

SQLite follows a separate path. Litestream continuously replicates offsite. A daily job restores and validates that replica before creating a client-side encrypted full snapshot in a second offsite location. The restore process prevents an older disaster snapshot from replicating back to the primary replica while it is being tested.

Recovery Runs from a Frozen Revision

A long recovery cannot safely follow a moving branch.

The complete recovery command creates a temporary checkout at the selected commit. Image publication, provider installation, host activation, all three application builds, state restoration, and final acceptance run from that frozen tree. New commits in the normal checkout cannot alter a recovery already in progress.

The workflow also has durable resume points:

1. Verify the image and recovery inputs.
2. Ask OVH to install the image.
3. Activate the embedded NixOS closure.
4. Build verified application packages locally.
5. Deploy the platform and public site.
6. Restore and validate SQLite, then verify upload storage.
7. Start Caddy.
8. Deploy Operations.
9. Run final acceptance checks.

If application deployment fails after the host boots, we resume at application deployment. We do not erase the disks again. Public DNS cutover is another separate, confirmed operation after direct checks against the replacement host.

Updates Use the Same Health Check

NixOS checks for stable updates each night and keeps the previous generation available for rollback. A new generation is accepted only after it passes the same read-only health checks used by Operations monitoring.

Provisioning, automatic updates, recovery, and monitoring therefore share one definition of a healthy server. If those checks fail, the host returns to the previous generation.

Lessons Beyond NixOS

You can get pretty far by applying the same strictness to Debian and Ansible:

• provision from an exact revision;
• define explicit ownership and cleanup;
• pin the tools used during recovery;
• guard destructive commands;
• test the complete replacement-host workflow automatically.

None of this requires NixOS. The difference is that Ansible lets you skip these steps and keep going. NixOS makes disagreements with the declared system much harder to ignore. And we like having these technical guardrails at Curling IO.

Decision

We are keeping NixOS.

It does not build our applications, restore customer data, eliminate the provider, or replace deployment scripts. It gives those systems a host with a clearer definition and a safer activation boundary.

The most valuable part is not reproducibility as an abstract property. It is the pressure to make hidden assumptions explicit:

• Which system owns this file?
• Which key can recover this secret?
• Which artifact identifies this host?
• What survives loss of the server?
• What proves a new generation is healthy?
• Where can recovery resume after a failure?

These questions have improved the whole deployment system, including the parts that do not use Nix.

---

This is Part 9 of the Curling IO Foundation series. Previous: A Live Admin Panel Without Writing JavaScript.