An autonomous ops platform
The system I built to run a portfolio of live production services by myself, safely. It senses events, plans and drafts fixes with LLMs, opens pull requests for a human to approve, and learns from every outcome.
The problem
I run a portfolio of live products by myself. There is no team to catch a bad deploy, watch the dashboards, or triage a failing build at 2am. So I built the ops layer I would otherwise need to hire: an event-driven system that does the first pass on everything and only ever asks me to approve, never to clean up.
The loop
Events (GitHub webhook / scheduled health check / alert)
├
Dispatcher ──► triage agent ──► policy engine
(confidence 0–100) (auto / batch /
├ interrupt / escalate)
▼
Specialized workflow (pr_review · ci_diagnose · health_triage · …)
├
▼
SENSE → PLAN → ACT → LEARN
sense validate task, check repo state, simulate change
plan build a structured plan (typed schema)
act execute step by step, git checkpoint per step,
per-step recovery ladder, run tests in a sandbox
learn store the outcome as a deduped lesson
├
▼
Draft PR opened ──► HUMAN APPROVES ──► merge (never automatic)
Safety by design
I came up through application security, so the controls are the point, not an afterthought. Defense in depth:
- Draft PRs only, never auto-merge. The worst case is a rejected PR. A human approves every change, and each merge or reject is a labeled outcome the system learns from.
- Blast-radius control. The executor works in git branches only, scans for dangerous patterns (rm -rf , force-push, DROP TABLE) before applying edits, and honors do-not-touch paths (
.env*, secrets, migrations, mainnet). - Kill switch. A file checked at the top of every handler halts all autonomous action immediately, no restart needed.
- Rate limits and a circuit breaker. Caps on commits per hour and PRs per day, and it stops after N consecutive failures to prevent restart storms.
- Prompt-injection defense. All untrusted text (issues, PR comments, CI logs) is fenced as untrusted data the model is told never to obey, and injection patterns are stripped.
- Secrets and integrity. Secrets pulled from a secret manager at start, mode-600, never in git; webhooks verified with HMAC-SHA256; hardened systemd sandboxing and key-only SSH.
Decisions I'd defend
Planner, executor, and learner are separate, not one ReAct loop.
A planner emits a structured plan, an executor runs it step by step with a git checkpoint per step, and a learner records the outcome. If step 3 fails, it recovers just step 3 through a ladder (retry, simplify, skip, escalate) instead of unwinding the whole task. That is the difference between a reliable agent and one that spirals into hallucination.
A cost-optimized, multi-provider LLM router with failover.
Each task goes to the cheapest tier that can handle it and falls back across providers on rate-limit or overload. 12 models across 6 providers and 7 tiers, with real per-token prices and per-call cost tracking in code. It runs on roughly $400 a month.
SQLite with WAL and a single writer thread, not Postgres or Kafka.
One database for the event queue, the lessons, and the task log. WAL mode for concurrent reads and a dedicated writer thread so there is never a locked database. When you are solo, operational simplicity is a feature; right-sizing is a skill.
By the numbers
- 17+ live repos and 61 services operated from one place
- ~3,000 automated tests; a reviewed PR merge is the only path to production
- Disaster recovery tested monthly, restoring into an ephemeral Docker container with canary checks
- Drift detection for schema, service registry, secrets, and configuration, with alerting