Recursive Self-Development as a Lifestyle Choice
What it took to mint one LDAP service account by hand, what it took to automate that, and what it now takes to automate the things we built while automating it. A field report from a self-hosted software factory bolted to an enterprise app engine.
The ritual#
Here is what “create a service account” used to mean.
Some application — an ERP module, a CI runner, a mail relay, anything that authenticates against the directory — needs an LDAP identity. A user object, a password, maybe a group membership. Conceptually this is one row of data. Operationally, it was a liturgy:
-
Generate the password by hand. A
pwgeninvocation on whatever terminal was closest. The password now exists in exactly one place: a scrollback buffer. -
Get it into the password manager by hand. Open VaultWarden, create an item, name it according to a convention that lives in your head, paste. The convention matters enormously and is enforced by nothing.
-
Feed it to Terraform out of band. The infrastructure repo is a Terraspace project; the LDAP user is a resource in a stack. But the password is not in the repo (correctly!), so it arrives via a tfvars file you must remember not to commit, or an environment variable, or an interactive prompt at plan time. Every one of these is an operator step that no automation can see.
-
Run the apply from a workstation.
terraspace upon the one machine that has the state, the credentials, the VPN routes, and the muscle memory. The state file — which now contains the resolved password — lives wherever that machine puts it. -
Deliver the password to the consumer by hand. The application does not read LDAP passwords from Terraform. So: paste it into a Helm values file (encrypted, hopefully), or a Salt pillar, or a Kubernetes secret created with
kubectlat two in the morning. If the app is one of those enterprise engines that stores its LDAP bind credential in its own database, you paste it into an admin UI as well. -
Apply the consumers in the right order. The secret must exist before the deployment that mounts it; the LDAP object before the bind that uses it; the DNS record before the certificate that names it. The ordering diagram exists nowhere. It is re-derived, each time, from scar tissue.
Every individual step is defensible. Passwords should not live in git. State should be applied deliberately. Secrets should pass out of band. That is the trap: a chain of locally-reasonable decisions that sums to a process where creating one directory row takes an experienced operator half an evening, cannot be delegated, cannot be repeated exactly, and leaves artifacts (scrollback, tfvars, laptop state) that are each a small security incident with patience.
Pass one: automate the ritual#
The automation pass looked the way these always look, which is to say: each manual step got replaced by a system, and each system is bigger than the step it replaced.
The hand-minted password became a declarative secrets capability — ask for a secret family in code, and the mint happens inside the machinery: generated, stored, sealed to the specific cluster that may read it, delivered as a Kubernetes secret to the namespace that consumes it. Nobody sees the value. There is no scrollback. The password manager stopped being a manual convention and became a delivery target — and then, mostly, stopped being involved at all.
The workstation apply became CI-applied infrastructure. Terraform runs in a pipeline, one stack per pull request, triggered by merge. The out-of-band tfvars died because the out-of-band inputs died: everything the stack needs is either in the repo (shapes, wiring) or in the secrets machinery (values). The operator’s remaining verb is “merge.”
The laptop state file became a central state database — an actual PostgreSQL instance with locking, so that CI, workstations, and anything else that plans against the estate see one truth.
And the ordering scar tissue became explicit: apply-ordering encoded in the layering of stacks and apps, with the GitOps controller retrying until dependencies converge instead of a human sequencing the applies from memory.
Minting an LDAP service account is now: open a small PR that declares the account and its secret family, merge it, watch the train roll through. One row of data costs roughly one row of code. Victory.
The bill arrives#
Except now look at what exists.
The central state database is load-bearing for every apply in the estate — so it needs backups. Real ones: dual-target, off-machine, to object storage on two different clusters, with the encryption key escrowed somewhere that survives the death of the machine that runs the backups. Backups need verification, staleness alerting, and a restore drill, or they are a comfort object rather than a control.
The CI that applies infrastructure needs an identity to do it with — so now there are bot accounts, and bot accounts have tokens, and tokens must be scoped and sealed per cluster so that compromising one silo does not hand an attacker the other. Token sealing is itself a small system, with its own repo, its own conventions, its own failure modes.
The merge-fast PR culture that makes the whole thing pleasant created a new silent failure: push a commit to a branch whose PR already merged and the commit orphans — git reports success, the change never lands. So there is now a push wrapper that checks, by SHA against the forge API, that the thing you pushed is actually on a path to the base branch. A tool that exists purely because the process got fast enough to need it.
And the estate itself — forty-odd repos across two git hosts, charts, modules, app definitions, pillars, images, the declared configuration of the routers and switches that carry it all — grew past the point where any one person holds the dependency graph in their head. So there is now tooling that extracts the producer-consumer edges from the code and renders the whole thing as a navigable map, because understanding the factory became its own engineering problem.
None of this was in the plan when the goal was “stop pasting passwords into an admin UI.” All of it is the support process built by the act of automating — the second-order estate. And here is the punchline: we are now automating that. The backup pipeline is declared in the same infrastructure-as-code it protects. The control plane is being wrapped so that even the operator gates — hold this component, promote it, decommission it — are expressed as one-line changes in a synced tree, where merging a PR is the only human act left. The support tooling for the automation is being fed into the automation.
Recursion as lifestyle#
There is a name for a function that calls itself, and there is a name for a person who does this to their infrastructure, and the second name is not “finished.”
The honest framing: this never terminates, and that is not a defect. Each loop of the recursion replaces a ritual (manual, undelegatable, irreproducible, quietly dangerous) with a system (declared, reviewed, repeatable, observable) — and each system opens a smaller, stranger class of work one level up. You do not reach a fixed point where there is no work. You reach a state where the work is always about the layer below you, and the layer below you is always better than it was.
The toll is real. A self-hosted software factory plus a self-hosted enterprise app engine means you own the whole tower: the app, the platform under the app, the delivery machinery under the platform, the state and secrets under the delivery, the backups under the state, the tooling that watches the tooling — and, at the bottom, the metal itself. When a node dies here, the ticket does not go to a cloud vendor. It reads like a repair log: machine opened, memory sticks cleaned and reseated, fault reproduced on power-up, next isolation step queued. The network underneath is no different; the routers and switches are declared in the same code as the applications they carry. Every convenience you refuse to rent, you must instead operate, and everything you operate eventually demands its own automation, which demands its own support, which — yes.
It is worth noticing, quietly, what that refusal leaves you holding. Every floor of this tower is somebody’s entire business model, rented out one seat at a time. A working end-to-end copy of the whole thing — rituals distilled into systems, systems into code, code into a map — is not nothing.
But the compounding runs in your favor. The evening that once produced one service account now produces the capability to produce service accounts — the kind of capability that, elsewhere, ships with a per-seat price. The scar tissue that once lived in one operator’s memory now lives in code, in findings, in a map an agent can traverse. The recursion is the point: every trip around the loop, the estate gets a little more able to develop itself — and the humans get promoted from performing the ritual to deciding what the ritual should become.
Such is the toll. We keep paying it on purpose. There is no try — only do.