Skip to main content

2 posts tagged with "operations"

View All Tags

Why We Built Our Own Error Tracking

· 13 min read
Dave Rapin
Dave Rapin
Founder @ Curling IO
About this post

This is a technical implementation note about error tracking in Curling IO v3. It is written for software engineers and operators, and goes deeper into Rust, SQLite, durable jobs, source maps, diagnostic safety, and LLM requests than our usual product posts.

A request returning HTTP 500 or a background job failing gives us an error message, but investigating it usually requires more: the producing commit, a source location, the failure chain, the operations that ran before the failure, and enough occurrences to see whether the inputs vary.

Repeated failures need separate handling. One defect inside a loop can produce thousands of reports. We need to retain the occurrence count without sending an alert or making an LLM request for each report.

We could have sent these errors to a hosted error-tracking service. We built a narrower system ourselves for three reasons: data sovereignty, direct integration with our application and operations pipelines, and control over which internal data leaves our infrastructure.

The implementation is split between Curling IO and our separate Operations application. Curling captures a bounded diagnostic envelope without waiting for another service. Operations imports and deduplicates it, sends the alert, asks a fast, lightweight, low-cost LLM for a structured analysis through OpenRouter when that integration is enabled, and stores the result with the issue. The redacted issue record is available through a command and a static report.

What We Found Useful About NixOS

· 10 min read
Dave Rapin
Dave Rapin
Founder @ Curling IO
About this post

This is a technical implementation note about the infrastructure and recovery architecture for Curling IO v3. It is written for software engineers and operators, and goes deeper into NixOS, bare-metal provisioning, secrets, deployment, backups, and recovery validation than our usual product posts.

Curling IO used to run on Debian configured with Ansible. That setup worked. It installed packages, configured Caddy and the firewall, created systemd services, and prepared our blue-green deployment slots.

The problem was not that Ansible couldn't describe the server. The problem was that a working, mutable server let us get away with an incomplete description. Old files remained under /etc. Build tools had been installed through a different path. Provider choices about disks and RAID lived outside the playbook. A provisioning run could succeed because an earlier run had already left the right thing behind.

NixOS is much less tolerant of these gaps. This strictness is occasionally annoying, but it is also the main benefit. It forces us to decide who owns a file, a service, a secret, a package, or a recovery input.

Moving from Ansible to NixOS taught us how much validation and cleanup the old approach had been missing. It exposed the difference between a server that continues to work and one we can recreate from declared inputs.

This is what we found useful while rebuilding an OVH bare-metal server from empty disks, restoring its state, and making the process safe enough to repeat.